codewarnab commented on issue #5981: URL: https://github.com/apache/arrow-rs/issues/5981#issuecomment-6731252885
Thanks @alamb. I looked at the code to see what uses `dict_id`. Writing: the writer does not use `Field::dict_id`. It makes its own IDs ([convert.rs#L637](https://github.com/apache/arrow-rs/blob/b231cf15efa5dbbbe4ae8503849aa6ab3cfc4e0c/arrow-ipc/src/convert.rs#L637), [writer.rs#L819-L822](https://github.com/apache/arrow-rs/blob/b231cf15efa5dbbbe4ae8503849aa6ab3cfc4e0c/arrow-ipc/src/writer.rs#L819-L822)). So no change is needed there. Reading: the reader uses `Field::dict_id` in two places ([reader.rs#L177](https://github.com/apache/arrow-rs/blob/b231cf15efa5dbbbe4ae8503849aa6ab3cfc4e0c/arrow-ipc/src/reader.rs#L177), [reader.rs#L932](https://github.com/apache/arrow-rs/blob/b231cf15efa5dbbbe4ae8503849aa6ab3cfc4e0c/arrow-ipc/src/reader.rs#L932)). Some public functions get the IDs only from the `Schema` that the user passes in: - [`read_record_batch`](https://github.com/apache/arrow-rs/blob/b231cf15efa5dbbbe4ae8503849aa6ab3cfc4e0c/arrow-ipc/src/reader.rs#L820), [`read_dictionary`](https://github.com/apache/arrow-rs/blob/b231cf15efa5dbbbe4ae8503849aa6ab3cfc4e0c/arrow-ipc/src/reader.rs#L836), [`FileDecoder::new`](https://github.com/apache/arrow-rs/blob/b231cf15efa5dbbbe4ae8503849aa6ab3cfc4e0c/arrow-ipc/src/reader.rs#L1107) - Flight [`flight_data_to_arrow_batch`](https://github.com/apache/arrow-rs/blob/b231cf15efa5dbbbe4ae8503849aa6ab3cfc4e0c/arrow-flight/src/utils.rs#L55) and the Flight SQL client ([client.rs#L651](https://github.com/apache/arrow-rs/blob/b231cf15efa5dbbbe4ae8503849aa6ab3cfc4e0c/arrow-flight/src/sql/client.rs#L651)) If we remove `dict_id`, these functions need the IDs from somewhere else. One idea is a small type that has the `Schema` and the field to ID mapping. I also checked C++ and Java. Rust often reads files written by C++ or Java, so we must handle their IDs too: - C++ gives IDs to nested dictionaries parent first ([dictionary.cc#L102-L122](https://github.com/apache/arrow/blob/eb083203776c91e56a8df05765ed7ba37efa11ee/cpp/src/arrow/ipc/dictionary.cc#L102-L122)). Rust does children first. In Java the user can pick any ID ([DictionaryUtility.java#L56-L64](https://github.com/apache/arrow-java/blob/22f9750bf0de0e42bd8550fb5f01bc6ab422d350/vector/src/main/java/org/apache/arrow/vector/util/DictionaryUtility.java#L56-L64)). So the reader must read the IDs from the file. It should not guess them from the field order. - In Java, many fields can use the same dictionary ID. The C++ reader allows this if the types are the same ([dictionary.cc#L238-L245](https://github.com/apache/arrow/blob/eb083203776c91e56a8df05765ed7ba37efa11ee/cpp/src/arrow/ipc/dictionary.cc#L238-L245)). We need to keep this working. I will use the cross-language integration tests to check this. Is this direction ok? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
