jhorstmann commented on a change in pull request #1499:
URL: https://github.com/apache/arrow-rs/pull/1499#discussion_r838940888
##########
File path: arrow/src/compute/kernels/boolean.rs
##########
@@ -575,10 +577,242 @@ where
Ok(PrimitiveArray::<T>::from(data))
}
+/// Creates a (mostly) zero-copy slice of the given buffers so that they can
be combined
+/// in the same array with other buffers that start at offset 0.
+/// The only buffers that need an actual copy are booleans (if they are not
byte-aligned)
+/// and list/binary/string offsets because the arrow implementation requires
them to start at 0.
+/// This is useful when a kernel calculates a new validity bitmap but wants to
reuse other buffers.
+fn slice_buffers(
Review comment:
Ideally we would be able to do exactly that. The problem in the current
code is that the `offset` in ArrayData applies to both the data and validity
buffers. If the offset in the previous ArrayData is larger than zero then we
would either have to pad the start of the validity bitmap by the same number of
bits (which was the approach tried in #510) or also slice the data buffers so
we can access them with an offset of zero, same as the newly created validity.
I don't fully like either of these approaches. A better alternative would
require quite some refactoring and api changes by removing `offset` from
`ArrayData` and instead pushing it into `Buffer` (for primitive types) and
`Bitmap` (for boolean and validity). I want to look into how much effort such a
refactoring would be, but I'm not sure when I'll find time for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
