cjolowicz commented on issue #12136: URL: https://github.com/apache/datafusion/issues/12136#issuecomment-2320600835
This is probably unrelated, but there's something that puzzles me about the fair spill pool logic. The try_grow function checks that the consumer isn't allocating more than its fair share. So it needs to determine how much memory the consumer has already allocated. But it only looks at the reservation that's passed, not at all reservations of that consumer. Consider a single spillable consumer that allocates the entire pool. It shouldn't be able to grow the resulting reservation. But if it splits the reservation into two, it can now grow it - even when the other reservation remains allocated. Here's a failing test showing this: https://github.com/apache/datafusion/pull/12170/files ``` ---- memory_pool::pool::tests::test_fair_with_split_reservation stdout ---- thread 'memory_pool::pool::tests::test_fair_with_split_reservation' panicked at datafusion/execution/src/memory_pool/pool.rs:417:34: called `Result::unwrap_err()` on an `Ok` value: () ``` Let me know if I'm misunderstanding something here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
