Kristofferjoha opened a new pull request, #26483:
URL: https://github.com/apache/datafusion/pull/26483
## Which issue does this PR close?
- Part of #26478
## Rationale for this change
None of the existing ASOF join benchmarks has a large right input that
reaches the join
unfiltered: q08–q10 filter the right side down to a few groups before the
broadcast.
This adds that workload as a baseline for #26478.
On main, the ASOF join keeps the whole right input in memory that can't be
spilled, so the
query needs at least that much memory to finish. With `--partitions 4`, Q11
completes with a
600 MB memory pool (peak 543 MB) and fails at 500 MB, where the join's
`AsOfJoinInput`
reservation holds 457.8 MB. The entire right input (20M rows x 3 Int64
columns).
## What changes are included in this PR?
- `benchmarks/sql_benchmarks/asof_join/benchmarks/q11.benchmark`: 1M left
rows, 20M right rows,
10K balanced equality groups, integer payloads, both sides covering the
same time range,
inputs unsorted so the required sorts are included.
- An example entry for Q11 in `asof_join.suite`.
## What is the testing strategy for this PR?
This PR only adds a benchmark, so there are no new tests. The benchmark
checks that the plan
contains `AsOfJoinExec`, and I ran it locally; it returns 1,000,000 rows
(one per left row):
cargo run -p datafusion-benchmarks --release --bin benchmark_runner --
asof_join --query 11
Local timings (i7-13700KF, WSL2, rough): about 4.0 s with the default 24
partitions,
2.9 s with `--partitions 4`, and 5.6 s with `--partitions 1`.
## Are there any user-facing changes?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
