adriangb commented on PR #10780:
URL: https://github.com/apache/arrow-rs/pull/10780#issuecomment-5376474232
Some read-side evidence for this policy, measured on the branch as pushed
(all public datasets; same methodology as the PR body, ZSTD-1, identical writer
settings, only the fallback policy differing between file sets).
## ClickBench query A/B (all 100 `hits_partitioned` files)
Both arms are rewrites of the standard dataset — one with this branch's
default policy, one with `Adaptive { max_dictionary_page_size: 64 MiB }` —
queried by the **same** DataFusion binary (built from datafusion main @
c429919c7c), 43 ClickBench queries, 15 interleaved iterations per query per arm
(5 rounds × 3, stock/adaptive/original alternating within each round to cancel
load drift), medians.
| | stock rewrite | adaptive rewrite | Δ |
|---|---:|---:|---:|
| Total bytes | 9,982,078,788 | 9,055,616,486 | **−9.28%** |
| Query geomean (43 queries) | 153.4 ms | 145.5 ms | **−5.19%** |
As a harness sanity check, the untouched original dataset's geomean lands
within ~2% of the stock rewrite.
Consistent movers (≥5% and same sign in ≥4 of 5 rounds), which line up
exactly with the columns whose dictionary retention changed (`Title` −40%
bytes, `URL` −17%, `Referer` −12%):
- **Faster**: Q22 −29.9% (`Title` LIKE), Q27 −17.5% (URL full scan), Q20
−17.3% (`URL LIKE '%google%'`), Q21 −17.3%, Q5 −16.6% (`COUNT(DISTINCT
SearchPhrase)`), Q31 −12.0%, Q23 −10.3% (`SELECT *` with URL filter), Q30 −8.7%.
- **Slower**: Q38 +32.7% and Q39 +6.9% — highly selective filters that read
`URL`/`Referer` for a few thousand matching rows, so the reader decompresses
and materializes a much larger dictionary page to serve very few values. This
is the policy's read-side trade stated plainly: full scans of
retained-dictionary columns get faster, highly selective point-ish reads of
them get slower. (Q32 also moved −13.9% but touches only byte-identical
columns; we attribute it to environment residue, not the policy.)
Row counts were identical across arms for all 43 queries in every iteration;
two apparent value diffs (Q32/Q38) reproduce between runs on the *same* files
and are LIMIT-over-tied-counts order nondeterminism — order-insensitive
aggregation checksums are exactly equal.
## Decode microbenchmark (the repetitive-blob class)
Full-column scans (single leaf projected, single-threaded, release, medians
of warmed runs) of the PR's runs-reproducer `val` column (16 KiB blobs,
1000-value pool):
| Variant | Input bytes stock → adaptive | Scan speedup (adaptive) |
|---|---|---|
| ZSTD-1 | 50.7 MB → 31.0 MB | **1.7–2.0×** |
| Uncompressed | 1,074 MB → 41.4 MB | **2.4–2.8×** |
The uncompressed variant isolates the mechanism: the speedup *grows* with no
decompression in the picture, so the win is dominated by having ~26× fewer
input bytes to move and parse — the fallen-back file re-reads every 16 KiB
repeat in full, the dictionary file expands from a hot ~10 MB dictionary. On
general string columns (ClickBench `Title`/`URL`) the same microbenchmark shows
roughly read-neutral results for a one-shot local full-column scan (the
retained dictionary must be materialized per chunk, offsetting the 16–18% byte
savings), consistent with the query-level picture above where parallel query
execution does cash the bytes in.
Caveats: warm-cache medians on a busy 12-core laptop (interleaving cancels
drift, but treat individual sub-10% per-query deltas as noise); local NVMe, so
the −9.3% bytes understates the benefit for cold/object-store reads, while the
Q38-style regression would likely shrink there (I/O-dominated).
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
