ywskycn opened a new pull request, #5516: URL: https://github.com/apache/datafusion-comet/pull/5516
## Which issue does this PR close? Solving sub-issue 11 in #5212. ## Rationale for this change `CometBoundedShuffleMemoryAllocator` tracks allocations through its own `allocatedMemory` counter because its allocations bypass Spark's memory manager. However, it inherited `MemoryConsumer.getUsed()`, which reports Spark's unused internal counter and always returned `0`. This caused Comet shuffle memory usage logging to report zero even when the bounded allocator held pages or pointer arrays. ## What changes are included in this PR? - Override `getUsed()` in `CometBoundedShuffleMemoryAllocator` to return the synchronized `allocatedMemory` counter. - Document that the value represents allocator-wide usage shared across tasks. - Add regression tests for page allocations, pointer arrays, repeated frees, failed allocations, exhausted budgets, and page table exhaustion. - Register the new test suite in Linux and macOS CI. ## How are these changes tested? - Added `CometBoundedShuffleMemoryAllocatorSuite`. - Verified all five new tests failed before the fix because `getUsed()` returned `0`. - Verified the allocator suite and existing `SpillSorterSuite` pass after the fix: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
