andygrove commented on PR #2657: URL: https://github.com/apache/datafusion-ballista/pull/2657#issuecomment-6447370593
**Update:** confirmed in the code. The reader checked a pooled connection out for every stream and held it until the stream ended, so every producer and consumer task pair got its own connection. e573d270a in #2670 makes each reader partition share one connection per producer executor, with at most 64 streams on a connection. It stops short of one connection per executor pair. With the 64 MB connection window and 16 MB stream window, a few stalled streams (say a join's probe side while it builds) would fill the connection and starve the build side's streams on it. Within one reader partition all the streams feed the same output, so that can't happen. This should cut q2 from about 73k connections to roughly 8k. #2657 is rebased on it. Next step is rerunning q2 at d0572fc7d. If it still resets, I'll add the retry before the first batch and the backlog setting. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
