Rich-T-kid commented on code in PR #10984:
URL: https://github.com/apache/arrow-rs/pull/10984#discussion_r3937322373
##########
arrow-buffer/src/buffer/mutable.rs:
##########
@@ -737,9 +737,32 @@ impl MutableBuffer {
///
/// This is similar to `from_trusted_len_iter_bool`, however, can be
significantly faster
/// as it eliminates the conditional `Iterator::next`
+ ///
+ /// # Panics
+ ///
+ /// Panics if the backing storage for `len` bits cannot be allocated. Use
+ /// [`MutableBuffer::try_collect_bool`] when `len` comes from untrusted
input.
#[inline]
- pub fn collect_bool<F: FnMut(usize) -> bool>(len: usize, mut f: F) -> Self
{
- let mut buffer: Vec<u64> = Vec::with_capacity(bit_util::ceil(len, 64));
+ pub fn collect_bool<F: FnMut(usize) -> bool>(len: usize, f: F) -> Self {
+ Self::try_collect_bool(len, f).unwrap_or_else(|e| panic!("{e}"))
+ }
+
+ /// Fallible version of [`MutableBuffer::collect_bool`].
+ ///
+ /// `len` is a bit count, so the reservation is `ceil(len / 64)` words.
Reserving that
Review Comment:
nit: This is true if “words” means u64 storage words, but it’s a bit
ambiguous on 32-bit targets where “word” usually means 32 bits. Maybe say
`ceil(len / 64) u64 slots` instead.
##########
arrow-buffer/src/buffer/mutable.rs:
##########
@@ -1761,4 +1784,29 @@ mod tests {
buf.push(1u8);
buf.reserve(usize::MAX);
}
+ #[test]
+ fn try_collect_bool_reports_a_len_it_cannot_reserve() {
Review Comment:
I'd prefer to not skip the miri check here.
Im not sure that we need either of these test 🤔. I think the fix here is
self explanatory enough to not warrant them.
##########
arrow-buffer/src/buffer/mutable.rs:
##########
@@ -1761,4 +1784,30 @@ mod tests {
buf.push(1u8);
buf.reserve(usize::MAX);
}
+ #[test]
+ #[cfg_attr(miri, ignore)] // miri interprets the allocation and runs out
of memory itself
+ fn try_collect_bool_reports_a_len_it_cannot_reserve() {
+ // 2^60 bits is 2^54 u64 words, which is 128 PiB of backing storage.
+ // collect_bool reserves that before calling f, so it aborts the
process.
+ let mut calls = 0usize;
+ let result = MutableBuffer::try_collect_bool(1usize << 60, |_| {
+ calls += 1;
+ true
+ });
+ assert!(matches!(
+ result,
+ Err(MutableBufferError::AllocationError(_))
+ ));
+ assert_eq!(calls, 0, "f must not run when the reservation fails");
+ }
+
+ #[test]
+ fn try_collect_bool_matches_collect_bool_for_sizes_that_fit() {
Review Comment:
nit: I don't think `try_collect_bool` does enough thats different from
`collect_bool` to warrant adding another test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
