andygrove opened a new issue, #7075: URL: https://github.com/apache/datafusion-comet/issues/7075
### What is the problem the feature request solves? The sort-based JVM columnar shuffle sorts each spill's packed pointers in native code with a radix sort that always runs six passes, and the JVM reserves scratch memory the native sort never uses. - `native/core/src/execution/sort.rs` (`RdxSort`) sorts on the 24-bit partition id with 16 buckets and `cfg_nrounds() = 6` (two rounds per byte). It always does all six rounds, and it moves every element through two sets of 16 `Vec` buckets, so it needs about 8x the input in temporary memory. Spark's `ShuffleInMemorySorter.getSortedIterator()` skips the passes for bytes that are constant across the input (common when the partition count is below 256) and sorts in the reserved half of the pointer array. - `SpillSorter` constructs `new ShuffleInMemorySorter(allocator, 1, true)` with radix sort enabled, so the pointer array reserves a second half for Spark's radix sort. Comet sorts through `Native.sortRowPartitionsNative` instead, so that half is never used. ### Describe the potential solution Either use Spark's `ShuffleInMemorySorter.getSortedIterator()` (removes a JNI call and the 8x temporary allocation, skips constant-byte passes, uses the reserved half), or port its scheme to Rust (sort in place against a same-size scratch array, skip passes whose bucket histogram has one entry). If the native sort stays, construct the in-memory sorter with `useRadixSort=false` so the unused half is not reserved. Benchmark with a native `sorted_spill`-style bench at 100, 1000 and 10000 partitions and 100K rows per spill, and check peak memory. ### Additional context Finding J7 in #5905. Related: #7065 (one native call per spill, which also touches `SpillSorter.writeSortedFileNative`). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
