codewarnab opened a new pull request, #12147: URL: https://github.com/apache/arrow-rs/pull/12147
# Which issue does this PR close? - Part of #5981 (step 1 of the plan in https://github.com/apache/arrow-rs/issues/5981#issuecomment-6759164667). Does not close it. # Rationale for this change Readers find a dictionary's field through the deprecated `Field::dict_id`. Dictionary ids are an IPC encoding detail, so they should come from the IPC schema message, not from the logical `Field`. This PR adds a place to keep them so `Field::dict_id` can be deprecated and later removed without breaking anyone right away. # What changes are included in this PR? - New `arrow_ipc::IpcSchema`: a `Schema` plus a tree of dictionary ids with the same shape as the schema (nested dictionaries in structs, lists, unions, maps, run-end encoded values and dictionary values each get their own node; several fields may share one id). - `IpcSchema::try_from_fb` fills it from the schema message, so ids always come from the file. - `FileReader`, `StreamReader` and `StreamDecoder` use it internally. No API change. - New non-breaking variants: - `reader::read_record_batch_with_ipc_schema` - `reader::read_dictionary_with_ipc_schema` - `FileDecoder::new_with_ipc_schema` - `arrow_flight::utils::flight_data_to_arrow_batch_with_ipc_schema` (now used by `flight_data_to_batches`) - The old functions keep working and still use `Field::dict_id`. Not in this PR (open for discussion): - `FlightDataDecoder` and the Flight SQL client still use the old path, because the public `RecordBatchDecoder` has no way to accept an `IpcSchema` yet. I'd like feedback on whether to add a public `RecordBatchDecoder::with_ipc_schema` here or in a follow-up. - Deprecating the old variants (step 2) and removing `Field::dict_id` (step 3). # Are these changes tested? - New `test_ipc_schema_reads_dict_ids_from_message`: writes a file with dictionaries at the top level, inside a struct and inside a list, checks the id tree, then shows `FileDecoder::new_with_ipc_schema` decodes correctly even when the `Schema` has no `Field::dict_id` values (where `FileDecoder::new` cannot), including with a projection. - The existing first-match test now checks `IpcSchema` against `Schema::fields_with_dict_id`. - Existing `arrow-ipc` and `arrow-flight` tests pass. I have not yet run the cross-language integration tests; I'll do that before marking ready. # Are there any user-facing changes? Yes, additive only: `IpcSchema` and the four `*_with_ipc_schema` functions. No breaking changes. Decoded data is unchanged. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
