dwsmith1983 commented on PR #5613: URL: https://github.com/apache/datafusion-comet/pull/5613#issuecomment-5805426774
Merged main, which brought in the change that lets the pools overcommit on `grow` instead of panicking. The fair pool now builds on the shared `SparkMemory` wrapper from that change instead of its own JNI bridge, and keeps the discipline from this PR: charge under the lock, call Spark with no lock held, settle afterwards. `grow` charges under the lock, takes the anchor best effort, and whatever Spark declines becomes overcommit, as on main. `try_grow` keeps a short grant charged until Spark has it back. With overcommit owed the request carries the debt, so a refused `try_grow` can get more bytes than it asked for, and a new test pins that the whole grant goes back with neither the pool's total nor the overcommit moving. The anchor byte stays outside the overcommit ledger. The overcommit test from main now expects 99 of 100 bytes granted toward its first grow, since the anchor takes one first. The fair pool section of the memory guide says the same for `grow`. The description is up dated to match. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
