dwsmith1983 commented on PR #5613: URL: https://github.com/apache/datafusion-comet/pull/5613#issuecomment-5603764691
@andygrove the anchor is now a standalone one byte acquire at pool setup (d004f041b). `try_new` takes it before the pool exists and fails construction if Spark grants less than a byte or errors, so every live pool holds the anchor from birth to drop and no request carries it: `Requested`, the bootstrap mutex, the surplus release, and the exact-fit rejection are gone, and `try_grow` asks for exactly `additional` again with the reviewed reserve-under-lock, call-unlocked, roll-back shape. Your sibling-consumer case is a test now: the stub models another consumer of the same task freeing its last page while an acquire is parked, and the entry survives because the anchor byte is held. So is the serialization point: a second acquire no longer queues behind a parked first one. One consequence worth calling out: construction can now park inside Spark, and `acquire_task_shared_pool` used to run the create step under the process-wide registry lock, which would have stalled plan creation for every task on the executor. It now creates outside the lock and re-checks on insert; a concurrent loser drops its own pool and its anchor goes back over JNI after the lock is released, with a test through the real registry. On `grow`, the `unwrap` predates this branch and is DataFusion's contract for the infallible path; with the anchor off the request path the only panic left there is a short grant a plain request would also have hit. On #5466: it moves the admission basis from `state.used` to `reservation.size()`, which DataFusion updates only after `try_grow` returns, so two concurrent grows of one reservation would both pass the check before either lands. Whichever lands second needs a deliberate rebase: keep the reserve-under-lock step and apply the per-reservation limit to it, and rewrite the two comments in `try_grow` and `shrink` that explain the `state.used` basis. Nothing in this commit changes that shape. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
