Rich-T-kid commented on code in PR #12059:
URL: https://github.com/apache/arrow-rs/pull/12059#discussion_r4659322524
##########
arrow-select/src/interleave.rs:
##########
@@ -312,6 +342,78 @@ fn interleave_dictionaries<K: ArrowDictionaryKeyType>(
Ok(Arc::new(array))
}
+/// Returns the values array shared by every referenced, non-null dictionary
key.
+///
+/// Inputs with no selected rows or only selected null keys are ignored
because their
+/// values cannot affect the output. Returns `None` if two referenced sources
use
+/// different values arrays. If every selected key is null, returns the values
array
+/// for the first selected input.
+fn referenced_shared_dictionary_values<'a, K: ArrowDictionaryKeyType>(
Review Comment:
>In real work loads I would expect that any such dictionaries should be
removed as part of a filter or evaluation 🤔
filter kernals re-use the values array so they are actaully a good target
for this optimization.
> I am asking about this because it looks somewhat expensive to check (as it
walks all values)
I've updated this PR to address your and jay's concerns, the new optimized
branch only runs if every values array shares the same arc pointer which is
very cheap. just to validate that this doesnt cause any regressions[ I've added
benchmarks](https://github.com/apache/arrow-rs/pull/12059/changes/4661ae77c184d470f5c6fe65ecf5a373f73f8c98)
to target the distinct case and the results show no regression :
https://github.com/apache/arrow-rs/pull/12059#issuecomment-7083134313.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]