shinzoxD opened a new pull request, #24496: URL: https://github.com/apache/datafusion/pull/24496
## Which issue does this PR close? - Closes #24468 ## Rationale for this change When `SortExec` runs in TopK mode (`ORDER BY ... LIMIT k` on unsorted input) and the k result rows span more than one output batch, `output_batches` reported `1` regardless of how many batches the operator actually emitted. For example, a top-25 query at `batch_size = 10` emits `[10, 10, 5]` but reported `output_batches=1`. This is the SortExec TopK counterpart of #24470 / #24495 (`PartitionedTopKExec` metrics). That sibling work is out of scope here. ## What changes are included in this PR? `TopK::emit` now records baseline output metrics (`output_rows`, `output_batches`, `output_bytes`) on each batch sent to the consumer after splitting to `batch_size`, instead of once on the pre-split heap batch. ## Are these changes tested? Added the issue's reproduction as `topk_output_batches_metric_matches_emitted_batches`: it asserts `output_batches` equals the number of batches actually emitted (`[10, 10, 5]` → 3). Also ran: - `cargo test -p datafusion-physical-plan --lib topk` (82 passed) - `cargo test -p datafusion --test core_integration topk_output_batches_metric_matches_emitted_batches` (passed) - `cargo fmt --all` - `cargo clippy -p datafusion-physical-plan --all-targets --all-features -- -D warnings` - `cargo clippy -p datafusion --test core_integration -- -D warnings` ## Are there any user-facing changes? `EXPLAIN ANALYZE` now reports the correct `output_batches` for SortExec in TopK mode when the result spans multiple batches. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
