andygrove opened a new issue, #2808: URL: https://github.com/apache/datafusion-ballista/issues/2808
Disclaimer: this issue was drafted by an LLM (Claude Code) at my request, against `bf90879ad`. The code references were checked against that commit, but please treat the proposed fix as a suggestion. ## Problem Executor and result-service gRPC listeners get TCP keepalive from `create_grpc_server_incoming` (`ballista/core/src/utils.rs`), which applies `GrpcServerConfig::tcp_keepalive_seconds` (default 3600) with `TcpIncoming::with_keepalive`. Its doc comment explains why: tonic ignores the builder's `tcp_nodelay` and `tcp_keepalive` when serving from a pre-bound listener, so they are applied by hand. The scheduler's gRPC service does not go through that helper. `start_grpc_service_with_listener` in `ballista/scheduler/src/scheduler_process.rs` takes a pre-bound `tokio::net::TcpListener` and serves it with `axum::serve`. After #2798 it sets `TCP_NODELAY` on every accepted connection via `ListenerExt::tap_io`, but sets no keepalive. The effect is that a scheduler connection whose peer disappears without closing it (a killed executor host, a dropped NAT mapping or load-balancer entry) is not reaped by the OS keepalive probes. It stays open until something else closes it. I have not measured how much this costs in practice; the point is that the scheduler is the one listener that does not honour a keepalive setting the other servers do. ## Suggested fix Set keepalive in the same `tap_io` hook (for example via `socket2::SockRef::from(&tcp).set_tcp_keepalive(...)`). The scheduler has no `GrpcServerConfig`, only `SchedulerConfig`, so this needs either a new scheduler option (for example `--grpc-tcp-keepalive-seconds`, default matching the 3600 s used elsewhere) or `start_grpc_service_with_listener` taking the value as a parameter. The standalone scheduler uses tonic's `TcpIncoming` and could take `with_keepalive` from the default `GrpcServerConfig` in the same change. It is also worth checking whether axum's HTTP/2 serving sets the HTTP/2 keepalive ping interval and timeout that `create_grpc_server` sets for tonic servers. I have not verified this. Related: #2798 (`TCP_NODELAY` on scheduler connections). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
