andygrove opened a new pull request, #2069:
URL: https://github.com/apache/datafusion-ballista/pull/2069
# Which issue does this PR close?
Closes #2068.
# Rationale for this change
`client_ttl` defaults to `0`, which disables the executor's Ballista client
pool
entirely. With no pool, the shuffle-read path opens and discards a **new TCP
connection per fetch**, so a single shuffle-heavy query exhausts the host's
ephemeral port range and fails with `AddrNotAvailable` — as does every query
after it, until those sockets leave `TIME_WAIT`.
A single executor never shows this: shuffle reads are served from local
files, so
there are no remote fetches. It appears as soon as a second executor exists,
which is every real deployment.
Measured on the SF1 TPC-DS correctness gate, 1 scheduler + 2 executors (`-c
4`
each), `--partitions 16`, everything else default:
| | before | after |
|---|---:|---:|
| peak TCP sockets during one query | 16563 | 1618 |
| peak `TIME_WAIT` | 16343 | 12 |
| q14 | FAILED (`AddrNotAvailable`) | verified OK |
| full gate | **11 OK / 77 FAILED** | **88 OK / 0 FAILED** |
The test host's ephemeral range is 16384 ports (49152–65535), so a single
query
consumed all of it. The `TIME_WAIT` sockets were overwhelmingly against the
executor **flight** ports (8315 on :50052, 8023 on :50053, versus 1 on the
scheduler's :50050), which pins it to the shuffle-read path.
The pooling machinery already exists and works, and the shuffle reader is
written
assuming connection reuse:
> Because total in-flight bytes stay <= the sized h2 window, the governor —
not
> the 64 KB transport window — is the binding backpressure, **which is what
makes
> multiplexing over few connections safe.**
It was simply off unless the operator knew to pass `--client-ttl`.
# What changes are included in this PR?
- Default `client_ttl` to 30 seconds (was 0) so the pool is on unless
explicitly
disabled. `--client-ttl 0` still opts out.
- Update the flag help and the `ExecutorProcessConfig::client_ttl` doc to say
what `0` actually costs.
- Add a test pinning the default to a pooling value, so it cannot regress to
the
disabled state unnoticed.
30s is chosen to comfortably outlive the gaps between fetches within a job
while
still releasing idle connections; the existing eviction thread handles
reclaim.
Happy to take a different number if maintainers prefer.
## Verification
With 2 executors and **no** `--client-ttl` flag, the full SF1 TPC-DS gate is
**88/88 verified** against single-process DataFusion (previously 11 OK / 77
FAILED in the same configuration). `cargo test --workspace`, `cargo clippy
--all-targets --workspace -- -D warnings`, and `cargo fmt --all` are clean.
# Are there any user-facing changes?
Yes — executors now cache idle client connections for 30s by default instead
of
disposing each one. This makes multi-executor clusters work out of the box.
Operators who want the old behaviour can pass `--client-ttl 0`. No API
changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
