adriangb opened a new pull request, #25625: URL: https://github.com/apache/datafusion/pull/25625
## Which issue does this PR close? - Refers to https://github.com/apache/datafusion/issues/23564. - Precursor for https://github.com/apache/datafusion/pull/23565. This PR must merge first, so the benchmark bot can compare `main` and that PR with `run benchmark spill_views`. ## Rationale for this change https://github.com/apache/datafusion/pull/23565 changes how spilled `StringView` and `BinaryView` columns are compacted. No current benchmark shows the effect: - The `sort_tpch` string columns are either high-cardinality (`l_comment`) or have a dictionary buffer that is too small to be compacted (`l_shipinstruct`). - Most suites spill only when the environment sets a low memory limit. ## What changes are included in this PR? A new SQL benchmark suite `spill_views` (in `benchmarks/sql_benchmarks/spill_views/`) and a `bench.sh` entry for it. There is no change to the spill code. Each query reads 1M rows from a Parquet file that the suite writes with `COPY`. The data must come from Parquet, because then the views of a dictionary-encoded column point into one shared buffer. Each query sets its own memory limit and `target_partitions = 4`, so it spills in any environment. | Query | Subgroup | What it does | Memory limit | |---|---|---|---| | `q01_sort_string_1_distinct` | `repeated` | `ORDER BY` a shuffled key, `StringView` payload with 1 distinct value (64 bytes) | 40M | | `q02_sort_string_1000_distinct` | `repeated` | Same, with 1000 distinct values | 40M | | `q03_sort_binary_1000_distinct` | `repeated` | Same as q02, with a `BinaryView` payload | 40M | | `q04_sort_string_all_distinct` | `distinct` | Same, with all-distinct values (compaction cannot remove data) | 96M | | `q05_group_by_string_all_distinct` | `distinct` | `GROUP BY` an all-distinct string key (compaction cannot remove data) | 96M | The asserts check the row count, the number of distinct values, that the column is read as a view type, and that the memory limit and `target_partitions` settings are applied. ## What is the testing strategy for this PR? I ran the suite with `benchmark_runner` and with `./bench.sh run spill_views` (the `cargo bench --bench sql` path that the bot uses). All asserts pass and each query takes less than 150 ms per iteration. I also ran the same queries with `EXPLAIN ANALYZE`, on `main` and with this commit on top of https://github.com/apache/datafusion/pull/23565. All queries spill on both sides. Spilled bytes are from the sort or final aggregation operator. Times are the median of 40 iterations of `benchmark_runner` on a laptop (Apple M4 Pro, local SSD), with runs of both builds alternated: | Query | Spilled (`main`) | Spilled (PR) | Time (`main`) | Time (PR) | |---|---|---|---|---| | q01 sort, 1 distinct | 84.2 MB | 23.2 MB | 63.5 ms | 57.0 ms | | q02 sort, 1000 distinct | 84.2 MB | 30.6 MB | 57.0 ms | 65.3 ms | | q03 sort, 1000 distinct binary | 84.2 MB | 30.6 MB | 54.0 ms | 65.9 ms | | q04 sort, all distinct | 84.2 MB | 84.2 MB | 71.0 ms | 95.7 ms | | q05 group by, all distinct | 84.2 MB | 84.2 MB | 75.2 ms | 77.7 ms | The machine was under load, so the times are only approximate. On a local SSD, spill I/O is cheap. The benchmark bot will give better numbers. ## Are there any user-facing changes? No. This PR only adds a benchmark. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
