alamb commented on code in PR #12059:
URL: https://github.com/apache/arrow-rs/pull/12059#discussion_r4652313893
##########
arrow-select/src/interleave.rs:
##########
@@ -265,32 +265,62 @@ fn interleave_dictionaries<K: ArrowDictionaryKeyType>(
let (should_merge, has_overflow) =
should_merge_dictionary_values::<K>(&dictionaries, indices.len());
if !should_merge {
+ let first_values = dictionaries[0].values();
+ let shared_values = if dictionaries
+ .iter()
+ .skip(1)
+ .all(|dictionary| Arc::ptr_eq(first_values, dictionary.values()))
+ {
+ Some(first_values)
+ } else if dictionaries.len() > 2 {
+ // For larger interleaves, distinct mappings may belong only to
+ // unused or all-null inputs and can be ignored.
+ referenced_shared_dictionary_values(&dictionaries, indices)
+ } else {
+ None
+ };
+ if let Some(shared_values) = shared_values {
+ return interleave_shared_dictionary(&dictionaries, indices,
shared_values);
+ }
+
return if has_overflow {
Review Comment:
given the control paths here, it might make sense to have some sort of enum
rather than 2 flags to make it clear how the strategy is being picked.
Something like
```rust
enum InterleaveStrategy {
SharedDictionary,
Fallback,
FallBackDictionary
}
```
##########
arrow-select/src/interleave.rs:
##########
@@ -312,6 +342,78 @@ fn interleave_dictionaries<K: ArrowDictionaryKeyType>(
Ok(Arc::new(array))
}
+/// Returns the values array shared by every referenced, non-null dictionary
key.
+///
+/// Inputs with no selected rows or only selected null keys are ignored
because their
+/// values cannot affect the output. Returns `None` if two referenced sources
use
+/// different values arrays. If every selected key is null, returns the values
array
+/// for the first selected input.
+fn referenced_shared_dictionary_values<'a, K: ArrowDictionaryKeyType>(
Review Comment:
is this really a common case? I realize it may show up in benchmarks, but it
seems to me like this could be overfitting to the benchmarks.
In real work loads I would expect that any such dictionaries should be
removed as part of a filter or evaluation 🤔
Do we have any sense of how often we end up interleaving dictionary arrays
that happen to be empty but still appear in the interleave call?
I am asking about this because it looks somewhat expensive to check (as it
walks all values)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]