Rich-T-kid opened a new pull request, #12106: URL: https://github.com/apache/arrow-rs/pull/12106
# Which issue does this PR close? - N/A # Rationale for this change IPC dictionary batches do not contain their value type, so readers recover it from the stream schema. A stream with `D` dictionary IDs and `F` schema fields currently walks the schema for every dictionary batch, making dictionary lookup `O(D * F)`. For example, a 1,000-column stream with one dictionary per column performs roughly 500,000 field visits just to recover the value types. This overhead is independent of how many rows or distinct values each dictionary contains. # What changes are included in this PR? Stateful IPC readers now cache dictionary value types: - The first dictionary ID still uses the existing early-exit schema lookup. - If a second unique dictionary ID is encountered, the reader builds the complete lookup once. - Subsequent dictionary batches use the cached value type. This applies to `StreamReader`, `StreamDecoder`, and `FileDecoder`. Streams with no dictionaries do no extra work, streams with a single dictionary ID avoid collecting the full schema, and the public low-level reader API is unchanged. The optimization kicks in when reading IPC data with repeated dictionary batches or more than one dictionary ID. It is most useful for wide schemas with many dictionary-encoded columns. One possible consumer is DataFusion spill reading: DataFusion uses `StreamDecoder`, so spill files containing multiple dictionary columns can reuse this lookup. Other examples include dictionary-heavy IPC network streams and cached IPC data. # Are these changes tested? Yes. The existing `ipc_reader` dictionary benchmarks were used; no new benchmarks were added. Local results on an Apple M4 Max, comparing `bb47f8091e` with this PR: | existing benchmark | main | this PR | improvement | | --- | ---: | ---: | ---: | | 1,000 columns / 4 distinct / flat | 3.65 ms | 2.56 ms | 29.8% | | 1,000 columns / 4 distinct / list | 5.29 ms | 4.31 ms | 18.6% | | 1,000 columns / 4,096 distinct / flat | 11.28 ms | 10.18 ms | 9.8% | | 1,000 columns / 4,096 distinct / list | 12.83 ms | 11.53 ms | 10.2% | The comparison used separately built baseline and patched binaries in alternating order. Hosted benchmark results are still pending, which is why this PR is a draft. Validation: - `cargo test -p arrow-ipc` - `cargo clippy -p arrow-ipc --all-targets -- -D warnings` - `cargo fmt --all -- --check` # Are there any user-facing changes? No API or IPC wire-format changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
