comphead opened a new pull request, #26953: URL: https://github.com/apache/datafusion/pull/26953
## Which issue does this PR close? - Closes #26940. ## Rationale for this change A multi-column sort under a memory limit can fail with `ResourcesExhausted` for `ExternalSorterMerge` instead of spilling. Its merge holds the row-format keys of every sorted run at once. When the keys are most of each row, that needs more than the 2x the sort reserves for each buffered batch. The issue has the details and a reproducer. ## What changes are included in this PR? With a memory limit, `ExternalSorter` sorts the runs of a multi-column merge into batches of an eighth of `batch_size`, but at least 1024 rows, so the merge holds the keys of one such batch per run. The merge output keeps `batch_size`, and sorts without a memory limit are unchanged. This adds a `batch_size` parameter to `sort_run`, which #26930 also changes. ## What is the testing strategy for this PR? `test_merge_keys_larger_than_batches` (current-thread and multi-thread) sorts two `Int32` keys in a pool that holds 16 buffered batches. It fails on `main` with `ResourcesExhausted` and passes here. The reproducer from the issue, with `datafusion-cli`: | | `main` | This PR | |---|---|---| | 30M rows, 1 partition, `-m 512M` | fails | 2 spills | | 30M rows, default partitions, `-m 1G` | fails | 16 spills | | 28M rows, 1 partition, `-m 1G` | fails | no spill | The criterion `sort` benchmark has no memory limit and is unchanged. With the split forced on at `batch_size = 8192`, its multi-column in-memory cases took 2 to 12% longer, which is the cost under a memory limit. ## Are there any user-facing changes? No. Known limitations: runs of at most 1024 rows are not split. That covers sorts with a `batch_size` of 1024 or less, and small input batches, which #26931 would coalesce into larger runs. Each run's existing one-batch read-ahead now holds one of the smaller batches with its reservation already released. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
