andygrove commented on PR #2670: URL: https://github.com/apache/datafusion-ballista/pull/2670#issuecomment-6474229213
Addressed in 353e9075f, thanks for the prototype notes. All six prototyped items are in: `SharedClient` is a flag now, the registry slot and broadcast replay use `watch`, `failure` is a `OnceLock`, the coalescer lives in `Output` with `all_closed()` derived, and `shared_connection_config` takes the config once. `open_remote_once` wraps the no-pool client in `SharedClient` too. One difference: I kept a sync fast path in `shared()` (plain mutex for the connection, tokio mutex only while connecting) rather than one async lock, since the first round asked for a hot path without it. The proposals are in as well: the registry comes from the session config only, `closed_jobs` and `closed_stages` are one bounded map keyed by job, and `evict_idle` drops discarded shared slots. That covers dead executors from #2673, and the part left there is idle slots that are still connected. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
