xudong963 opened a new pull request, #26787: URL: https://github.com/apache/datafusion/pull/26787
## Which issue does this PR close? - Closes #26785. ## Rationale for this change `COPY ... STORED AS ARROW` writes files many times larger than the data, and takes much longer, when `Utf8View` / `BinaryView` columns come out of a sort, merge, filter or slice. Each batch then references much more view data than it holds, and the Arrow IPC writer writes every referenced data buffer in full, so the same string data is written once per batch. The issue has a self-contained reproducer. Measured on an Apple M4 Pro (12 cores, `release-nonlto` `datafusion-cli`, `main` vs this PR, 6 alternating runs each): | Write | `main` | This PR | |---|---|---| | Issue reproducer: 200k 64-character `Utf8View` strings, `ORDER BY random()` | 193.5 MB, 0.21–0.28 s | 14.6 MB, 0.018–0.024 s | | TPC-H SF1 `lineitem`, `l_orderkey <= 300000`, 4 columns, `ORDER BY l_partkey` | 592.5 MB, 2.40–3.54 s | 9.1 MB, 0.06–0.12 s | | Same sort as `Utf8`, and the unsorted `Utf8View` write | 13.6 MB / 9.1 MB | unchanged | The TPC-H output files contain the same rows (`EXCEPT ALL` both ways is empty). ## What changes are included in this PR? - `ArrowFileSink` compacts each batch with `gc_view_arrays` before writing it. The spill writer already uses this function for the same reason. It only copies an array when its data buffers hold more than its views reference (and more than 10 KB), so batches without waste are written as they are. - `datafusion_physical_plan::spill::gc_view_arrays` changes from `pub(crate)` to `pub` so that `datafusion-datasource-arrow` can use it. ## What is the testing strategy for this PR? Two new tests in `datafusion/core/src/datasource/file_format/arrow.rs`, `test_write_arrow_compacts_view_buffers` and `test_write_partitioned_arrow_compacts_view_buffers`. Both write a table whose batches are slices of one batch, so that every batch references all the string data, once to a single file and once with `PARTITIONED BY`. They write uncompressed and check that the output is smaller than twice the string data. Without the change they fail, writing 13.3 MB and 50.6 MB for 1.56 MB of strings. These are Rust tests rather than `.slt` tests because sqllogictest cannot observe file sizes. Also ran the full sqllogictest suite, the `spill` and `datafusion-datasource-arrow` tests, `ci/scripts/rust_clippy.sh` and `typos`. ## Are there any user-facing changes? Arrow IPC files written from view types after a sort, merge, filter or slice are smaller and faster to write. `datafusion_physical_plan::spill::gc_view_arrays` is now public. This is an addition, not a breaking change. An open question for review: `gc_view_arrays` now lives in the public API of the `spill` module, although it is no longer used only for spilling. I kept the move minimal; it could be moved elsewhere if preferred. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
