lxc512157407 opened a new issue, #24822:
URL: https://github.com/apache/datafusion/issues/24822
## Summary
Group-by aggregation over `Dictionary(Int32, Utf8View)` group keys is
measurably slower than the identical query over plain `Utf8View` columns, on
both DataFusion 54.0.1 and 55.0.0 (rev-checked). On TPC-H SF=10 in-memory:
- Q2 (p_type/p_size keys, 2 passes): dictionary **-27%** vs Utf8View
- Q19 (ps_shipmode key): dictionary **-37%**
- Q20 (s_name key): dictionary **-33%**
Reproduced with a self-contained bench: same RecordBatches, only the
group-key column encoding differs (via Arrow cast before planning). No disk, no
confounding.
## Repro sketch
```rust
// two sessions over the same batches; key column cast:
// a) Utf8View (baseline)
// b) Dictionary(Int32, Utf8View) (taxed)
let agg = ctx.sql("select l_returnflag, count(*) from t group by
l_returnflag").await;
```
Timings (5-round median, single machine, Win11, DF 55.0.0):
| query | Utf8View | Dictionary | delta |
|---|---|---|---|
| Q2-like | 21.1ms | 28.9ms | **-27%** |
| Q19-like | 8.6ms | 13.6ms | **-37%** |
| Q20-like | 12.4ms | 18.5ms | **-33%** |
## Why this matters
Dictionary encoding is the natural choice for low-cardinality string columns
(exact-dictionary scans, smaller memory). But as soon as such a column is a
group key, users pay a ~1.3-1.6x aggregation penalty — the opposite of what
encoding choice should do to query time. In our columnar engine (custom
TableProvider over PAX storage) this forces a lose-lose choice between scan
speed and aggregation speed on every low-cardinality string column.
## Suspect area
Group-aggregation kernels appear to upcast/normalize dictionary keys per
batch rather than grouping on the codes directly (for single-batch
aggregations the dictionary *is* already a perfect group id in the common
no-duplicate case). Happy to share the standalone bench if useful.
DF 54.0.1 and 55.0.0 both affected; can test main on request.
## Environment
DataFusion 55.0.0 / 54.0.1, Arrow 57/56, Windows 11, rustc 1.93-nightly,
in-memory RecordBatches (8192-row batches), single-threaded execution to
isolate kernel cost.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
