andygrove opened a new pull request, #5907: URL: https://github.com/apache/datafusion-comet/pull/5907
## Which issue does this PR close? Part of #5905 (finding J2). Does not close it. ## Rationale for this change `CometShuffleExternalSorter.growPointerArrayIfNecessary` sizes the replacement pointer array from `SpillSorter.getMemoryUsage()`, which Comet overrides to return the pointer array bytes plus all allocated data pages. Spark's `ShuffleExternalSorter` sizes it from `inMemSorter.getMemoryUsage()`, the array alone. The result is that the first growth, which happens after `initialSize / 2` records because half the array is reserved for radix sort, requests `2 x (pageBytes + arrayBytes) / 8` entries instead of `2 x arrayBytes / 8`. With the default page size that is tens of megabytes of pointer array for a couple of thousand rows, and every later growth compounds on top of the pages allocated since. Under the bounded allocator this either wastes memory that should have gone to data pages or fails the allocation and forces a spill after very few rows. ## What changes are included in this PR? - `SpillSorter` gains `getPointerArrayMemoryUsage()`, returning only the in-memory sorter's array size under the same lock as `getMemoryUsage()`. - `CometShuffleExternalSorter.growPointerArrayIfNecessary` uses it to size the new array, matching Spark's behaviour of doubling the pointer array. `getMemoryUsage()` itself is unchanged, so peak memory reporting and spill sizing still include the data pages. ## How are these changes tested? New test in `SpillSorterSuite` that inserts enough records to trigger the first pointer-array growth against a private off-heap allocator and asserts that memory in use afterwards equals one data page plus twice the initial array. Before this change the assertion fails because the array grows to more than the page size. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
