jorgecarleitao commented on a change in pull request #9759:
URL: https://github.com/apache/arrow/pull/9759#discussion_r598168645
##########
File path: rust/arrow/src/buffer/mutable.rs
##########
@@ -415,6 +415,61 @@ impl MutableBuffer {
buffer
}
+ /// Creates a [`MutableBuffer`] from a boolean [`Iterator`] with a trusted
(upper) length.
+ /// # use arrow::buffer::MutableBuffer;
+ /// # Example
+ /// ```
+ /// # use arrow::buffer::MutableBuffer;
+ /// let v = vec![false, true, false];
+ /// let iter = v.iter().map(|x| *x || true);
+ /// let buffer = unsafe { MutableBuffer::from_trusted_len_iter_bool(iter)
};
+ /// assert_eq!(buffer.len(), 1) // 3 booleans have 1 byte
+ /// ```
+ /// # Safety
+ /// This method assumes that the iterator's size is correct and is
undefined behavior
+ /// to use it on an iterator that reports an incorrect length.
+ // This implementation is required for two reasons:
+ // 1. there is no trait `TrustedLen` in stable rust and therefore
+ // we can't specialize `extend` for `TrustedLen` like `Vec` does.
+ // 2. `from_trusted_len_iter_bool` is faster.
+ pub unsafe fn from_trusted_len_iter_bool<I: Iterator<Item = bool>>(
+ mut iterator: I,
+ ) -> Self {
+ let (_, upper) = iterator.size_hint();
+ let upper = upper.expect("from_trusted_len_iter requires an upper
limit");
+
+ let mut result = {
+ let byte_capacity: usize = upper.saturating_add(7) / 8;
+ MutableBuffer::new(byte_capacity)
+ };
+
+ 'a: loop {
+ let mut byte_accum: u8 = 0;
+ let mut mask: u8 = 1;
+
+ //collect (up to) 8 bits into a byte
+ while mask != 0 {
+ if let Some(value) = iterator.next() {
+ byte_accum |= match value {
Review comment:
I agree with you all ❤️
I admit I have spent an immoral amount of time trying to optimize bitmaps,
but I have unfortunately not yet concluded what is the best way to handle them.
I think that we may not being able to express to the compiler what we want it
to do (some kind of operation over a single byte). @yordan-pavlov suggestion is
a great one in that direction, though.
FWIW, on my computer (a VM on azure), arrow master (not this PR) is giving
```
eq Float32 time: [113.40 us 114.81 us 116.28 us]
eq scalar Float32 time: [96.824 us 98.638 us 101.34 us]
```
and `arrow2` is giving
```
eq Float32 time: [84.519 us 86.065 us 87.772 us]
eq scalar Float32 time: [57.682 us 58.315 us 59.014 us]
```
This PR's idea on arrow2 (with corresponding changes) is giving me `-14%` on
`eq Float32` and `+35%` on `eq scalar Float32`. I pushed these benches to
master there.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
