Jefffrey commented on issue #5981: URL: https://github.com/apache/arrow-rs/issues/5981#issuecomment-5117462933
I was taking a look at this, and it seems theres still some usages of `dict_id` within the IPC reader that needs to be refactored before `dict_id` itself (and related methods) can be completely removed. https://github.com/apache/arrow-rs/blob/005a2fd43f1b243ed43f4cfa356ebe8f0757939b/arrow-ipc/src/reader.rs#L877-L887 Here, we need a way to find the datatype of the values type of the dictionary type, from the dictionary ID. A naive approach could be to have another data structure `HashMap<i64, DataType>` that we pass around with appropriate plumbing, one that is created when we initially parse the schema. I think this is somewhat doable. https://github.com/apache/arrow-rs/blob/005a2fd43f1b243ed43f4cfa356ebe8f0757939b/arrow-ipc/src/reader.rs#L173-L183 This one is a bit more tricky; we need to get the `dict_id` from a field. This can get gnarly when considering types like `Struct`s which can have multiple children, so we can't really plumb in `(Field, i64)` wherever we pass `Field`. We could try another map, like `HashMap<Arc<Field>>` where we hash based on arc pointer value? But that feels off. Not sure how to approach this one. It feels like we're a bit constrained in our current approach since we use `Schema` and `Field` from `arrow-schema` to pass around data, but we're trying to remove `dict_id` from them so we need to figure a different way to pass them around. I think once those two issues are solved then `dict_id` can be removed; the other usages I see in the codebase aren't significant. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
