alamb commented on issue #5861: URL: https://github.com/apache/arrow-rs/issues/5861#issuecomment-2161369649
> A better approach is probably to directly insert items in StringViewArray to dictionary array. Yes I agree this is likely -- I suggest using https://docs.rs/arrow/latest/arrow/array/builder/struct.GenericByteDictionaryBuilder.html directly That will also deduplicate the values and result in the smallest dictionary possible. The downside is that inserting each value will require hashing on a string I could imagine a faster implementation that keeps a map of `u128` (aka the view) to the in-process dictionary values that would be faster (hashing u128 rather than `&[u8]`). The downside is that this wouldn't catch values where the views pointed to the different bytes that happend to have the same value (e.g. `"foofoo"` if there were two views that pointed to `"foo"` and the second `"foo"`) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
