comphead opened a new pull request, #26960: URL: https://github.com/apache/datafusion/pull/26960
## Which issue does this PR close? - Closes #26627. ## Rationale for this change Compressed spill files are mostly string offsets, which zstd and lz4 barely compress, see #26627. Spilling is also slower than it needs to be: a batch of the table in #26627 takes about 35 `write` calls, one per IPC buffer. This replaces #26698, which needed spill reads to skip validation (#26688). ## What changes are included in this PR? - With `lz4_frame` or `zstd`, each top level `Utf8`, `LargeUtf8`, `Binary` and `LargeBinary` column is spilled as a struct of its value lengths and of its value bytes as one run-end encoded value. Both are valid arrays, so the IPC reader decodes them as usual before the spill reader rebuilds the offsets. Uncompressed spills are unchanged. - The rebuilt arrays are not validated again, as spill reads already intend with `with_skip_validation(true)`. - The encoded buffers of a batch that are under 64 KB are copied into one write. ## Benchmarks Apple M5 Max, interleaved runs against `main`. The reproducer from #26627 (`datafusion-cli -m 100m`, median of 7): | codec | main | this PR | |---|---:|---:| | zstd | 313 ms, 32.0 MB spilled | 274 ms, 8.9 MB spilled | | lz4_frame | 273 ms, 54.0 MB spilled | 252 ms, 17.1 MB spilled | `sort_tpch` queries at SF1 on `Utf8` columns (one partition, `-m 200m`, median of 5), spilled bytes and time against `main`: | query | lz4_frame | zstd | |---|---:|---:| | Q3 | 92.4 → 78.1 MB, -1.4% | 61.4 → 50.5 MB, +1.5% | | Q7 | 306.3 → 230.2 MB, -0.5% | 171.3 → 122.6 MB, -5.1% | | Q10 | 398.9 → 309.7 MB, -1.9% | 228.2 → 169.2 MB, -2.7% | In the `spill_io` benchmark, coalescing makes writing and reading spill files 2-13% faster, and the string datasets spill 27-41% smaller. With lz4, the lengths cost 8-11% more CPU than coalescing alone. <details><summary><code>spill_io</code> results and setup</summary> `cargo bench -p datafusion-physical-plan --bench spill_io`, mean of two alternating runs per build. "Coalescing only" is the first commit. Differences under about 5% are noise: the uncompressed rows of the last two columns run the same code. | benchmark | main | coalescing only | this PR | spilled, main → this PR | |---|---:|---:|---:|---:| | q2/uncompressed | 4.6 ms | 4.0 ms (-12.7%) | 4.1 ms (-10.4%) | unchanged | | q2/zstd | 17.1 ms | 16.4 ms (-3.9%) | 16.5 ms (-3.4%) | unchanged | | q2/lz4_frame | 11.4 ms | 10.5 ms (-7.4%) | 10.7 ms (-6.4%) | unchanged | | q16/uncompressed | 10.2 ms | 9.1 ms (-10.5%) | 9.3 ms (-8.3%) | unchanged | | q16/zstd | 40.2 ms | 39.2 ms (-2.4%) | 37.2 ms (-7.3%) | 4.4 MB → 2.6 MB | | q16/lz4_frame | 22.8 ms | 21.1 ms (-7.4%) | 23.5 ms (+3.0%) | 7.8 MB → 5.2 MB | | q20/uncompressed | 5.9 ms | 5.6 ms (-4.0%) | 5.9 ms (+0.3%) | unchanged | | q20/zstd | 26.3 ms | 25.4 ms (-3.3%) | 25.8 ms (-1.6%) | unchanged | | q20/lz4_frame | 16.3 ms | 15.5 ms (-4.9%) | 15.7 ms (-3.7%) | unchanged | | wide/uncompressed | 24.6 ms | 21.4 ms (-13.0%) | 23.6 ms (-3.9%) | unchanged | | wide/zstd | 100.5 ms | 97.3 ms (-3.2%) | 99.1 ms (-1.4%) | 11.6 MB → 8.2 MB | | wide/lz4_frame | 56.6 ms | 52.2 ms (-7.9%) | 56.6 ms (-0.1%) | 19.9 MB → 14.6 MB | The `sort_tpch` runs set: ```sql SET datafusion.execution.target_partitions = 1; SET datafusion.execution.parquet.schema_force_view_types = false; SET datafusion.execution.spill_compression = 'zstd'; -- or 'lz4_frame' ``` </details> ## What is the testing strategy for this PR? - `test_spill_lengths_round_trip` spills every encoded type, with nulls, sliced and empty, next to columns that are spilled as they are, with all three codecs. - `test_spill_lengths_compress` checks that a zstd spill of similar strings is under a quarter of plain zstd IPC. - `test_async_ipc_stream_writer` now checks that each batch takes one write. ## Are there any user-facing changes? Smaller compressed spill files and faster spilling. No configuration or API changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
