andygrove commented on PR #2657: URL: https://github.com/apache/datafusion-ballista/pull/2657#issuecomment-6451250118
Two speculative fixes for the `ConnectionReset`s, in #2670 at 3c6c5cf71 and rebased here at 667fb6719. The executor's Flight listener had a backlog of 128, inherited from `std::net::TcpListener::bind` via tonic's `serve`, so the burst of connections a wide job opens overflowed both the accept and SYN queues. Once the kernel falls back to SYN cookies, a connection whose first segment was dropped is reset as soon as it sends more, which matches the timing, the lack of logs on either side, and the counters. The Flight server now listens with a 4096 backlog (`GrpcServerConfig::listen_backlog`), and the reader retries a stream open whose connection failed before the server answered. `SyncookiesFailed` in `/proc/net/netstat` on a producer executor is the counter that confirms or rules out the mechanism. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
