ninglinLiu opened a new pull request, #11925: URL: https://github.com/apache/arrow-rs/pull/11925
# Which issue does this PR close? - Closes #11834 # Rationale for this change `merge` and `merge_n` build their output with the infallible `MutableArrayData::new`, which concatenates the input dictionaries and panics with `DictionaryKeyOverflowError` when the combined values do not fit the key type, even when every input has the same values and a merged dictionary would fit. `concat` and `interleave` already merge dictionaries in that case. # What changes are included in this PR? - Both kernels now call `MutableArrayData::try_new`. On `DictionaryKeyOverflowError` they fall back to `interleave`, which merges the dictionaries (or returns an error if the merged dictionary still does not fit). Other errors are returned as before, and the non-dictionary fast path is unchanged. - `merge` builds the `(array, row)` pairs from the mask, treating null mask entries as false and reading row 0 for a scalar side, the same way the slice-based path does. - `merge_n` builds the pairs from the indices. `interleave` has no null index, so `None` rows read from an extra one-row null array of the same type. - `merge` with two scalars delegates to `zip`, which is handled by #11663, so it is not touched here. # Are these changes tested? Yes. Two new tests use two `Int8`-keyed dictionaries with the same 100 values, so concatenation overflows but the merged dictionary fits: - `test_merge_dictionaries_whose_combined_values_overflow_the_key_type`: array/array, array/scalar and scalar/array, with a null in the mask and a null key in the input. - `test_merge_n_dictionaries_whose_combined_values_overflow_the_key_type`: `Option<usize>` indices with `None` rows and a null key. Both panic on `main` and pass with this change. `cargo test -p arrow-select` and `cargo clippy -p arrow-select --all-targets -- -D warnings` pass. # Are there any user-facing changes? `merge` and `merge_n` return a merged dictionary instead of panicking. No API changes. This PR was prepared with AI assistance and reviewed/tested by the author. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
