getChan commented on PR #24779:
URL: https://github.com/apache/datafusion/pull/24779#issuecomment-5467102714
I ran the benchmark locally using the PR's default configuration. The
results are
consistent with the baseline reported in the PR description. The
success/failure
outcomes, allocation sizes, and spill metrics matched the recorded baseline;
only
wall-clock times varied by environment.
Environment for the numbers below: Apple M5 (10 cores), macOS 26.5.2,
`--release`,
Rust 1.97.0, DataFusion PR #24779 at `984945b`.
Command:
```bash
cd benchmarks
./bench.sh run join_mem
```
The benchmark was run with the default configuration: a 300MB fair memory
pool, 4
partitions, 20M rows, and 3 iterations.
```
# configuration outcome mean spilled
━━━━━ ━━━━━━━━━━━━━━━━━━ ━━━━━━━━━━━━━━━━━━━━ ━━━━━━━━
━━━━━━━━━━━━━━━━━━━━━━━
1 default settings ❌ Resources - —
— HashJoinExec exhausted at the
hash-table build
───── ────────────────── ──────────────────── ────────
───────────────────────
2 workaround — ✅ completes 0.437s SortExec 310.6 MB
in
prefer_hash_join 28 events
=false
───── ────────────────── ──────────────────── ────────
───────────────────────
3a build side ✅ completes 0.090s nothing
filtered to N =
10M
───── ────────────────── ──────────────────── ────────
───────────────────────
3b build side ❌ Resources - —
filtered to N = exhausted at the
12M hash-table build
───── ────────────────── ──────────────────── ────────
───────────────────────
4 row 3a forced ✅ completes 0.334s SortExec 233.0 MB
in
through 24 events
SortMergeJoinExe
c
───── ────────────────── ──────────────────── ────────
───────────────────────
5 hash aggregation ✅ completes 1.037s AggregateExec
1079.4
through the same MB in 84 events
budget
```
The recorded allocation failures were:
- Row 1: additional allocation of 95.4 MB with 38.2 MB already allocated
- Row 3b: additional allocation of 57.2 MB with 22.9 MB already allocated
SMJ tax (row 4 / row 3a): 3.7x on a join that would have fit in memory.
Every row matched the recorded baseline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
