ninglinLiu commented on PR #11925: URL: https://github.com/apache/arrow-rs/pull/11925#issuecomment-6434271277
Thanks for taking a look, @Rich-T-kid! On benchmarks: the common path does the same work as before. `MutableArrayData::try_new` is the fallible form of `new` (`new` just unwraps it), so non-dictionary inputs and dictionaries that don't overflow go through exactly the same code. The existing `arrow/benches/merge_kernels.rs` covers that path. The `interleave` fallback only runs where the kernels used to panic with `DictionaryKeyOverflowError`, so there is no previous number to compare it against. Agreed that the fallback could avoid `interleave` later, e.g. by merging the dictionaries once and remapping keys directly in the merge loop. If you think it's useful, I can add a dictionary case to `merge_kernels.rs` in a follow-up so that optimization has a baseline. I'd rather keep this PR focused on the panic. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
