asubiotto opened a new issue, #12069: URL: https://github.com/apache/arrow-rs/issues/12069
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** Reading an IPC stream with many dictionary-encoded columns spends noticeable time finding each dictionary's field. For every dictionary batch, `get_dictionary_values` in `arrow-ipc/src/reader.rs` calls `Schema::fields_with_dict_id`, which collects every (nested) field in the schema into a `Vec`, then uses only the first element: ```rust let fields_using_this_dictionary = schema.fields_with_dict_id(id); let first_field = fields_using_this_dictionary.first().ok_or_else(|| ...)?; ``` With N dictionary columns, each of the N dictionary batches walks and allocates over the whole schema. In a local run, reading one batch with 1000 `Dictionary<Int32, Utf8>` columns took about 18.6 ms flat and 70.5 ms with each dictionary inside a `List`. With an early-exit lookup it took about 2.5 ms and 4.8 ms. **Describe the solution you'd like** Walk the schema in the same pre-order as `fields_with_dict_id` and stop at the first match, without allocating. This keeps the current behavior when several fields share an ID, and changes no public API. I'll add a benchmark in its own PR first, then the fix. **Describe alternatives you've considered** Building an ID → field map once per reader would remove the per-batch walk entirely. It's a larger change, though, and dictionary IDs are due to move out of `Field` (#5981), so the smaller fix seems better for now. **Additional context** Relates to #5981. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
