mbutrovich commented on code in PR #12228:
URL: https://github.com/apache/arrow-rs/pull/12228#discussion_r4589508471


##########
arrow-array/src/builder/generic_bytes_builder.rs:
##########
@@ -201,33 +202,44 @@ impl<T: ByteArrayType> GenericByteBuilder<T> {
 
     /// Builds the [`GenericByteArray`] and reset this builder.
     pub fn finish(&mut self) -> GenericByteArray<T> {
-        let array_type = T::DATA_TYPE;
-        let array_builder = ArrayDataBuilder::new(array_type)
-            .len(self.len())
-            .add_buffer(std::mem::take(&mut self.offsets_builder).into())
-            .add_buffer(std::mem::take(&mut self.value_builder).into())
-            .nulls(self.null_buffer_builder.finish());
+        // SAFETY: offsets start at 0 and every append pushes the current end 
of
+        // `value_builder`, so they are non-empty and monotonically increasing.

Review Comment:
   Could we add a `debug_assert!` for this `SAFETY` claim, here and at lines 
226-227 in `finish_cloned`?
   
   The old `finish` went through `ArrayDataBuilder::build_unchecked`, which 
runs full validation under `force_validate`, including the check that offsets 
never decrease 
([data.rs](https://github.com/apache/arrow-rs/blob/d65329d3e71f4276ce41c8675446d178cecfc1fa/arrow-data/src/data.rs#L1776-L1790)).
 The new path loses that check. `OffsetBuffer::new_unchecked` has no 
`force_validate` branch 
([offset.rs](https://github.com/apache/arrow-rs/blob/d65329d3e71f4276ce41c8675446d178cecfc1fa/arrow-buffer/src/buffer/offset.rs#L68-L76)),
 and `GenericByteArray::try_new` assumes the offsets are already monotonic 
([types.rs](https://github.com/apache/arrow-rs/blob/d65329d3e71f4276ce41c8675446d178cecfc1fa/arrow-array/src/types.rs#L1732-L1741)).
 With `--features force_validate`, `BinaryArray::new_unchecked` and 
`StringArray::new_unchecked` both accept the offsets `[0, 2, 1, 3]`.
   
   Could you also update this sentence in the PR description? It holds for the 
other three builders, but not for offset ordering here. Every other caller of 
`OffsetBuffer::new_unchecked` has the same gap, which is #12284, so the assert 
covers only this builder until that's fixed.
   
   > The existing `force_validate` feature still routes these through 
validation.
   
   ```suggestion
           debug_assert!(self.offsets_builder.is_sorted());
           // SAFETY: offsets start at 0 and every append pushes the current 
end of
           // `value_builder`, so they are non-empty and monotonically 
increasing.
   ```



##########
arrow-array/src/builder/generic_bytes_builder.rs:
##########
@@ -201,33 +202,44 @@ impl<T: ByteArrayType> GenericByteBuilder<T> {
 
     /// Builds the [`GenericByteArray`] and reset this builder.
     pub fn finish(&mut self) -> GenericByteArray<T> {
-        let array_type = T::DATA_TYPE;
-        let array_builder = ArrayDataBuilder::new(array_type)
-            .len(self.len())
-            .add_buffer(std::mem::take(&mut self.offsets_builder).into())
-            .add_buffer(std::mem::take(&mut self.value_builder).into())
-            .nulls(self.null_buffer_builder.finish());
+        // SAFETY: offsets start at 0 and every append pushes the current end 
of
+        // `value_builder`, so they are non-empty and monotonically increasing.
+        let offsets = unsafe {
+            OffsetBuffer::new_unchecked(std::mem::take(&mut 
self.offsets_builder).into())
+        };
+        let values = std::mem::take(&mut self.value_builder).into();
+        let nulls = self
+            .null_buffer_builder
+            .finish()
+            .filter(|n| n.null_count() > 0);
 
-        self.offsets_builder.push(self.next_offset());
-        // SAFETY: builder is constructed from valid offset and value buffers 
maintained by the builder
-        let array_data = unsafe { array_builder.build_unchecked() };
-        GenericByteArray::from(array_data)
+        self.offsets_builder.push(T::Offset::usize_as(0));
+
+        // SAFETY: `try_new` would not error. The last offset is 
`values.len()`,
+        // `nulls` has one entry per value, and for strings every value was
+        // appended from a `&str`, so each offset pair bounds valid UTF-8.

Review Comment:
   I got part of the text I suggested for this comment wrong. The last offset 
can be less than `values.len()`, because `write!` adds bytes to `value_builder` 
without pushing an offset 
([`write_str`](https://github.com/apache/arrow-rs/blob/c3370f73078ca0d18167554504784ef2d0ed97e2/arrow-array/src/builder/generic_bytes_builder.rs#L390-L395),
 and `std::io::Write::write` on `GenericBinaryBuilder` does the same). After 
`append_value("a")` and then `write!(builder, "foo")`, `finish` returns an 
array whose last offset is 1 and whose values are 4 bytes long. `try_new` only 
requires the last offset to be at most `values.len()` 
([types.rs](https://github.com/apache/arrow-rs/blob/d65329d3e71f4276ce41c8675446d178cecfc1fa/arrow-array/src/types.rs#L1683-L1691)),
 so the code is sound, but the comment should say that. Could we use this text 
here and at lines 239-241 in `finish_cloned`?
   
   ```suggestion
           // SAFETY: `try_new` would not error. No append pushes an offset past
           // the end of `value_builder`, which only grows, so the last offset 
is
           // at most `values.len()` (`write!` can leave bytes after it). 
`nulls`
           // has one entry per value. For strings, every append, `write!`
           // included, copies whole UTF-8 sequences, so each offset pair bounds
           // valid UTF-8.
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to