parthchandra opened a new issue, #7272:
URL: https://github.com/apache/datafusion-comet/issues/7272
Part of #7267.
## Problem
`SortExec` with a fetch (ORDER BY ... LIMIT K) uses DataFusion's TopK. When
it
emits, it gathers all K rows into one batch with `interleave_record_batch`
(DataFusion 55.2.0, `topk/mod.rs:1015`), and only then slices the result into
batch-size pieces (`topk/mod.rs:816`). Compaction takes the same path:
`maybe_compact` (`:1022`) calls `emit_with_state` (`:1043`), and it runs
while input
is still arriving (`insert_batch`, `:467`) once more than 2K rows are
stored. So it
can fail before the final emit.
With a large K and large string/binary values, the single interleave passes
2 GiB
and fails with `OffsetOverflowError`. The batch size doesn't help because the
slicing happens after the interleave.
## Proposed fix (DataFusion)
- Interleave in chunks of `batch_size` rows (or under a byte budget) instead
of all
K at once.
- Or wrap the interleave in the same `retry_interleave` helper the sort
merge uses
(`sorts/builder.rs:251`). It is a private function today, so it would need
to be
shared.
- Apply the same change to compaction.
## Comet side
- No Comet change once upstream is fixed, beyond the DataFusion bump.
- Interim option: fall back to Spark for TopK when K times the estimated row
size
could pass 2 GiB. This is probably not worth doing unless users hit it.
## Tests
- DataFusion unit test with an injected overflow, or with chunking asserted
on
output batch sizes.
- Comet SQL-file test: ORDER BY ... LIMIT with a large K over a binary
column,
compared with Spark.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
