yordan-pavlov commented on a change in pull request #9759:
URL: https://github.com/apache/arrow/pull/9759#discussion_r598307603
##########
File path: rust/arrow/src/buffer/mutable.rs
##########
@@ -415,6 +415,61 @@ impl MutableBuffer {
buffer
}
+ /// Creates a [`MutableBuffer`] from a boolean [`Iterator`] with a trusted
(upper) length.
+ /// # use arrow::buffer::MutableBuffer;
+ /// # Example
+ /// ```
+ /// # use arrow::buffer::MutableBuffer;
+ /// let v = vec![false, true, false];
+ /// let iter = v.iter().map(|x| *x || true);
+ /// let buffer = unsafe { MutableBuffer::from_trusted_len_iter_bool(iter)
};
+ /// assert_eq!(buffer.len(), 1) // 3 booleans have 1 byte
+ /// ```
+ /// # Safety
+ /// This method assumes that the iterator's size is correct and is
undefined behavior
+ /// to use it on an iterator that reports an incorrect length.
+ // This implementation is required for two reasons:
+ // 1. there is no trait `TrustedLen` in stable rust and therefore
+ // we can't specialize `extend` for `TrustedLen` like `Vec` does.
+ // 2. `from_trusted_len_iter_bool` is faster.
+ pub unsafe fn from_trusted_len_iter_bool<I: Iterator<Item = bool>>(
+ mut iterator: I,
+ ) -> Self {
+ let (_, upper) = iterator.size_hint();
+ let upper = upper.expect("from_trusted_len_iter requires an upper
limit");
+
+ let mut result = {
+ let byte_capacity: usize = upper.saturating_add(7) / 8;
+ MutableBuffer::new(byte_capacity)
+ };
+
+ 'a: loop {
+ let mut byte_accum: u8 = 0;
+ let mut mask: u8 = 1;
+
+ //collect (up to) 8 bits into a byte
+ while mask != 0 {
+ if let Some(value) = iterator.next() {
+ byte_accum |= match value {
Review comment:
yes, I tried to benchmark doing the comparison separately, but it's not
faster; on my machine the fastest version is:
```
(0..8).for_each(|i| {
if chunk[i] == rhs {
*byte = set(*byte, i)
}
});
```
and that's even faster than:
```
chunk.iter().enumerate().for_each(|(i, &c_i)| {
*byte |= unsafe { mem::transmute::<bool, u8>(c_i == rhs) <<
i };
});
```
this is with the test data configured as:
```
let vec = (0..20049).map(|x| (x * x + x) % 2).collect::<Vec<_>>();
```
I think 2000 items (the old length of the test data) is much too small for
realistic benchmarking, and it would make more sense to benchmark with test
data with length same as the default batch size in DataFusion (I think this was
recently increased).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
