neilconway commented on code in PR #25754:
URL: https://github.com/apache/datafusion/pull/25754#discussion_r4290032980
##########
datafusion/functions-nested/src/utils.rs:
##########
@@ -50,6 +50,103 @@ pub(crate) fn empty_list_values<O: OffsetSizeTrait>(
))
}
+/// Build result row offsets and exclude elements stored under NULL list rows.
+///
+/// `keep` is a bitmap identifying which elements in the list's visible span
+/// should be preserved. The returned bitmap uses the same indexing, with bits
+/// for NULL list rows cleared. Selected elements may themselves be NULL; their
+/// validity is the caller's responsibility when copying values.
+pub(crate) fn prepare_list_filter<O: OffsetSizeTrait>(
+ list: &GenericListArray<O>,
+ keep: &BooleanBuffer,
+) -> (OffsetBuffer<O>, BooleanBuffer) {
+ let list_offsets = list.offsets();
+ let (first_offset, span_len) = offset_span(list_offsets);
+ debug_assert_eq!(keep.len(), span_len);
+ let mut offsets = Vec::with_capacity(list.len() + 1);
+ offsets.push(O::zero());
+ let mut kept_count = 0;
+ // Count the set bits before each row end in one pass over `keep`, which
+ // stays cheap for short rows.
+ let mut set_bits = SetBitCounter::new(keep.bit_chunks().iter_padded());
+ // Set bits stored under NULL rows, which the result does not keep.
+ let mut hidden_count = 0;
+ // NULL list rows are usually empty, so reuse `keep` until a NULL row
stores
+ // elements. Copy the bits of the preceding rows when that first happens.
+ let mut keep_mask: Option<BooleanBufferBuilder> = None;
+ for (row, window) in list_offsets.windows(2).enumerate() {
Review Comment:
Thanks for the suggestion! Splitting out the no-NULL-row case gives some
nice wins: narrow_2 −40%, keep_all −24% and keep10 −20%.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
