Dandandan commented on a change in pull request #9759:
URL: https://github.com/apache/arrow/pull/9759#discussion_r598256420
##########
File path: rust/arrow/src/buffer/mutable.rs
##########
@@ -415,6 +415,61 @@ impl MutableBuffer {
buffer
}
+ /// Creates a [`MutableBuffer`] from a boolean [`Iterator`] with a trusted
(upper) length.
+ /// # use arrow::buffer::MutableBuffer;
+ /// # Example
+ /// ```
+ /// # use arrow::buffer::MutableBuffer;
+ /// let v = vec![false, true, false];
+ /// let iter = v.iter().map(|x| *x || true);
+ /// let buffer = unsafe { MutableBuffer::from_trusted_len_iter_bool(iter)
};
+ /// assert_eq!(buffer.len(), 1) // 3 booleans have 1 byte
+ /// ```
+ /// # Safety
+ /// This method assumes that the iterator's size is correct and is
undefined behavior
+ /// to use it on an iterator that reports an incorrect length.
+ // This implementation is required for two reasons:
+ // 1. there is no trait `TrustedLen` in stable rust and therefore
+ // we can't specialize `extend` for `TrustedLen` like `Vec` does.
+ // 2. `from_trusted_len_iter_bool` is faster.
+ pub unsafe fn from_trusted_len_iter_bool<I: Iterator<Item = bool>>(
+ mut iterator: I,
+ ) -> Self {
+ let (_, upper) = iterator.size_hint();
+ let upper = upper.expect("from_trusted_len_iter requires an upper
limit");
+
+ let mut result = {
+ let byte_capacity: usize = upper.saturating_add(7) / 8;
+ MutableBuffer::new(byte_capacity)
+ };
+
+ 'a: loop {
+ let mut byte_accum: u8 = 0;
+ let mut mask: u8 = 1;
+
+ //collect (up to) 8 bits into a byte
+ while mask != 0 {
+ if let Some(value) = iterator.next() {
+ byte_accum |= match value {
Review comment:
Ah I have an explanation @jorgecarleitao
The benchmark in your test repo only matches for 10% of the values.
When 50% of the values match (changing the benchmark to use `%2`), this
version becomes faster.
So I think it is important to see what how want to benchmark / where we want
to optimize for,
```
scalar_eq_bitmap time: [1.3688 us 1.3720 us 1.3751 us]
change: [+63.468% +64.269% +65.058%] (p = 0.00 <
0.05)
Performance has regressed.
Found 5 outliers among 100 measurements (5.00%)
5 (5.00%) high mild
scalar_eq_bitmap1 time: [1.3381 us 1.3410 us 1.3441 us]
change: [+78.511% +79.377% +80.215%] (p = 0.00 <
0.05)
Performance has regressed.
Found 3 outliers among 100 measurements (3.00%)
1 (1.00%) low mild
2 (2.00%) high mild
scalar_eq_bitmap2 time: [945.68 ns 947.16 ns 948.87 ns]
change: [+4.5904% +5.0361% +5.4662%] (p = 0.00 <
0.05)
Performance has regressed.
Found 13 outliers among 100 measurements (13.00%)
3 (3.00%) low mild
7 (7.00%) high mild
3 (3.00%) high severe
```
scalar_eq_bitmap2:
```rust
buffer[..chunks]
.iter_mut()
.zip(chunks_iter)
.for_each(|(byte, chunk)| {
(0..8).for_each(|i| {
*byte |= if chunk[i] == rhs {
BIT_MASK[i]
} else {
0
};
});
});
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
