alamb commented on issue #5981: URL: https://github.com/apache/arrow-rs/issues/5981#issuecomment-2613987682
Copy/pasting from @andygrove on https://github.com/apache/arrow-rs/issues/7017 to get all the context in this ticket: > I started upgrading Comet to use the latest DataFusion/Arrow and saw that an effort is being made to remove dict_id from arrow-rs `Field`. > > It seems odd that arrow-java has dict_id as part of its field representation, but we plan to remove it from arrow-rs. > > I am not 100% sure but this may be problematic for Comet since we seem to rely on the dictionary id when passing arrays between Java and Rust. I did not write this code and am not yet familiar with it, but in one example, in Rust we have: > > let mut dict_id = 0; > let fields = arrays > .iter() > .enumerate() > .map(|(i, array)| match array.data_type() { > DataType::Dictionary(_, _) => { > let field = Field::new_dict( > format!("c{}", i), > array.data_type().clone(), > true, > dict_id, > false, > ); > dict_id += 1; > field > } > _ => Field::new(format!("c{}", i), array.data_type().clone(), true), > }) > and then in Java: > > DictionaryEncoding dictionaryEncoding = vector.getField().getDictionary(); > Dictionary dictionary = dictionaryProvider.lookup(dictionaryEncoding.getId()); > It will take me some time to get up to speed with how we are using this in Comet, and to determine whether it is needed or not. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
