dwsmith1983 commented on PR #5613: URL: https://github.com/apache/datafusion-comet/pull/5613#issuecomment-5834145319
Merged main, which brings in #6205. The pool keeps both of its checks: each consumer against its share, using the running total per consumer id, and the pool's total against `pool_size`. Both run under the lock in the same step that charges the bytes, and the charge now goes to the pool's total and the consumer's total together. Settling after the Spark call adjusts both. So a grant in flight, a shrink on its way to Spark, and a short grant being handed back count against the consumer's share as well as the pool until Spark has answered. `shrink` checks against the consumer's total, as main does. The anchor byte stays out of both totals. A few tests here had pinned the old comparison, where one consumer's bytes refused another consumer at the share. They now race a sibling reservation of the same consumer instead. The test for a grow in flight registers the second consumer late, so that the pool total is what refuses it. The concurrency test checks each consumer against its share and the pool against its size. Main's new tests count the anchor in what Spark holds. The memory management guide's paragraphs on the lock and the windows it leaves now describe both checks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
