lxc512157407 commented on issue #24822: URL: https://github.com/apache/datafusion/issues/24822#issuecomment-5535402223
Re-measured on current `main` (`9fc7a4d`, 2026-09-04) with a self-contained repro — the shape of the tax has changed since the original report, and one part of it is already fixed upstream: **Method**: identical in-memory tables where only the group-key encoding differs (`Utf8View` vs `Dictionary(Int32, Utf8View)`, produced by `cast`-ing the exact same batches); same `GROUP BY` + `COUNT(*)` + `SUM` on both; single target partition; 5 timed rounds after 1 warmup, median. Repro (buildable example, run with `cargo run --release -p datafusion-examples --example dict_group_tax`): <https://github.com/lxc512157407/datafusion/tree/bench/dict-group-tax> | case (key shape) | Utf8View ms | Dictionary ms | slowdown | |---|---|---|---| | q2-like: 2 keys, string (150 distinct, short) + int, 6M rows | 38.7 | 39.8 | **+3%** | | q19-like: 1 key, string (7 distinct), 6M rows | 25.0 | 29.3 | **+17%** | | q20-like: 1 key, long string (100k distinct, >12 bytes), 2M rows | 32.4 | 57.2 | **+77%** | Observations: 1. **The Q2-style multi-key tax from the original report is gone on main** (+3%, within noise). For context, main now contains a dictionary hashing path (introduced via #24233, see `hash_dictionary_scatter` in `datafusion/common/src/hash_utils.rs`) that hashes each dictionary value once and scatters via const-generic monomorphic loops — same shape as the fix we prototyped in the branch linked earlier in this thread. That work looks absorbed. 2. **The remaining tax concentrates on high-cardinality long-string dictionary keys** (q20-like, +77%): when the dictionary has ~100k distinct values the once-per-value hashing no longer amortizes, and the group-by additionally pays for materializing dictionary-encoded group keys. Note the view variant here has long strings heap-escaped (>12 bytes) — so this is not an inline-view advantage; the dictionary column carries strictly less data and is still slower. 3. Low-cardinality single-key (q19-like) retains a modest +17%. So the residual gap is narrower than the original 27–37% report but is still real for high-cardinality string group keys. Happy to help dig into the aggregation-side kernel if there's interest — the repro branch above can be the baseline. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
