nat-openai opened a new pull request, #7461:
URL: https://github.com/apache/datafusion-comet/pull/7461
## Which issue does this PR close?
<!--
We generally require a GitHub issue to be filed for all bug fixes and
enhancements and this helps us generate change logs for our releases. You can
link an issue to this PR using the GitHub syntax. For example `Closes #123`
indicates that this PR will close issue #123.
-->
Partially addresses #7092.
## Rationale for this change
<!--
Why are you proposing this change? If this is already explained clearly in
the issue then this section is not needed.
Explaining clearly why changes are proposed helps reviewers understand your
changes and offer better suggestions for fixes.
-->
`ROW_NUMBER` currently uses the generic bounded window executor, which
performs per-partition state management and batch slicing even though this
function only needs a counter and partition boundary detection. That overhead
is particularly expensive for many small window partitions.
## What changes are included in this PR?
<!--
There is no need to duplicate the description in the issue here but it is
sometimes worth providing a summary of the individual changes in this PR.
-->
- Add `RowNumberExec` to generate row numbers directly within each input
batch, carrying the final partition key and counter across batches.
- Select it when every window expression is built-in `ROW_NUMBER` with
matching partition and ordering keys.
- Reuse input column buffers and share the generated array between repeated
`ROW_NUMBER` expressions.
- Account for retained partition-key buffers through DataFusion’s memory
pool, including cleanup on completion, errors, and cancellation.
- Preserve existing ordering, distribution, statistics, and result-cast
behaviour, and document the new execution path.
## How are these changes tested?
<!--
We typically require tests for all PRs in order to:
1. Prevent the code from being accidentally broken by subsequent changes
2. Serve as another way to document the expected behavior of the code
If tests are not included in your PR, please explain why (for example, are
they covered by existing tests)?
-->
- Regression tests compare results against the generic executor across
partition boundaries, sliced and empty batches, global windows, repeated
expressions, and nullable, dictionary, and nested keys.
- Memory tests cover retained-buffer accounting, reservation failures,
cancellation, and avoiding retention of unrelated dictionary or view buffers.
- Spark SQL regressions cover global and partitioned windows, multiple batch
sizes, ANSI modes, repeated expressions, and mixed-function windows. A planner
regression checks executor selection alongside result casts and timing metrics.
- Benchmarks compare both executors on identical inputs and verify every row
number before timing. The benchmark shows improvements across all tested
partition sizes, with the largest gains for small partitions.
The 12 operator tests pass in an isolated harness. The Spark test additions
compile but have not been executed locally.
### Benchmark results
Median runtime for 262,144 rows in 32 batches of 8,192 rows, using three
string partition keys. Measurements pool two runs of 10 samples each on an
Apple M3 Max. The repeat measures streaming first.
The unchanged benchmark source ran in an isolated harness using Comet’s
release settings, DataFusion 55.2.0, and Arrow 59.3.2. These measurements cover
window execution over already-sorted input.
Rows per window partition | Generic (BoundedWindowAggExec) | Streaming
(RowNumberExec) | Speedup
-- | -- | -- | --
1 | 243.08 ms | 3.38 ms | 71.8×
4 | 66.53 ms | 2.75 ms | 24.2×
64 | 9.82 ms | 2.34 ms | 4.2×
16,384 | 6.39 ms | 2.32 ms | 2.8×
262,144 | 7.22 ms | 2.36 ms | 3.1×
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
