Rich-T-kid commented on PR #10438:
URL: https://github.com/apache/arrow-rs/pull/10438#issuecomment-5335555146
> how about something like this
>
> ```rust
> let mut_buffer_result = self.values.into_inner().into_mutable();
> match mut_buffer_result {
> Ok(mut mutable_buffer) => {
> let raw_bytes = mutable_buffer.as_slice_mut();
> let byte_idx_of_end = end / 8;
> let bits_to_preserve = end % 8;
> // end on a byte boundary, so just easily rewrite at byte
level
> if bits_to_preserve == 0 {
> // TODO: this technically can modify bits beyond what
the
> // boolean buffer actually points to, but given
we
> // have unique ownership it should be fine?
there could
> // be pathological case where if this buffer was
sliced
> // there could be unused bytes that we still
process,
> // if we wanna bother with that edge case
> raw_bytes
> .iter_mut()
> .skip(byte_idx_of_end)
> .for_each(|b| *b = 0);
> } else {
> // if end in middle of a byte, need to unset only
higher bits
> raw_bytes[byte_idx_of_end] &= (1_u8 <<
bits_to_preserve) - 1;
> raw_bytes
> .iter_mut()
> // +1 since we account for one byte above
> .skip(byte_idx_of_end + 1)
> .for_each(|b| *b = 0);
> }
> // TODO: this offset is wrong?
> let boolean_buf =
BooleanBuffer::new(mutable_buffer.into(), 0, len);
> BooleanArray::new(boolean_buf, self.nulls)
> }
> Err(buf) => {
> let mut builder = BooleanBufferBuilder::new(len);
> builder.append_buffer(&BooleanBuffer::new(buf, 0, end));
> builder.append_n(len - end, false);
> BooleanArray::new(builder.finish(), self.nulls)
> }
> }
> ```
>
> essentially rewrite at the byte level, except for if the `end` was inside
a byte so we need to do some bit ops there
>
> i did a single benchmark run and it seems promising, though i havent
carefully checked for edge cases yet 🤔
@Jefffrey
https://github.com/apache/arrow-rs/pull/10438/commits/aade7413c6fdc2ad383b2003d2808b4b0b914ad0
should address this. benchmarks show pretty good results locally
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
