asubiotto commented on code in PR #8978:
URL: https://github.com/apache/arrow-rs/pull/8978#discussion_r2608038050
##########
arrow-select/src/interleave.rs:
##########
@@ -346,6 +347,57 @@ fn interleave_fallback(
Ok(make_array(array_data.freeze()))
}
+/// interleave_fallback_dictionary is a fallback implementation for
interleaving
+/// dictionaries when it was determined that the dictionary values should not
+/// be merged. This implementation concatenates the value slices and recomputes
+/// the resulting dictionary keys.
+fn interleave_fallback_dictionary<K: ArrowDictionaryKeyType>(
+ dictionaries: &[&DictionaryArray<K>],
+ indices: &[(usize, usize)],
+) -> Result<ArrayRef, ArrowError> {
+ let relative_offsets: Vec<usize> = dictionaries
+ .iter()
+ .scan(0usize, |offset, dict| {
+ let current = *offset;
+ *offset += dict.values().len();
+ Some(current)
+ })
+ .collect();
+ let all_values: Vec<&dyn Array> = dictionaries.iter().map(|d|
d.values().as_ref()).collect();
+ let concatenated_values = concat(&all_values)?;
+
+ let mut has_nulls = false;
+ let new_keys: Vec<K::Native> = indices
+ .iter()
+ .map(|(a, b)| {
+ let dict = dictionaries[*a];
+ let old_keys = dict.keys();
+ if old_keys.is_valid(*b) {
Review Comment:
Yes, I considered even computing offsets in one pass and nulls in another.
Will play around with this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
