neilconway commented on code in PR #11323:
URL: https://github.com/apache/arrow-rs/pull/11323#discussion_r4170260152
##########
arrow-array/src/array/string_array.rs:
##########
@@ -504,6 +504,29 @@ mod tests {
let _ = StringArray::from(list);
}
+ #[test]
+ fn test_string_array_invalid_bytes_outside_offsets() {
+ // Only the bytes that the offsets span need to be valid UTF-8, as in a
+ // slice of a larger array
+ let values = Buffer::from_slice_ref(b"\xFFa\xC3\xA9\xFF");
+ let offsets = OffsetBuffer::new(vec![1, 2, 4].into());
+ let string = StringArray::try_new(offsets, values.clone(),
None).unwrap();
+ assert_eq!(string, StringArray::from(vec!["a", "é"]));
+
+ let offsets = OffsetBuffer::new(vec![1, 3, 4].into());
+ let err = StringArray::try_new(offsets, values, None).unwrap_err();
+ assert_eq!(
+ err.to_string(),
+ "Invalid argument error: Split UTF-8 codepoint at offset 3"
+ );
+
+ // An empty value whose offset is inside a character, in a values
buffer
+ // that is valid UTF-8 as a whole. ArrayData validation must agree.
+ let offsets = OffsetBuffer::new(vec![1, 1].into());
+ let string = StringArray::try_new(offsets,
Buffer::from_slice_ref("é"), None).unwrap();
Review Comment:
That should only be possible with unsafe code (i.e., where the caller can
somehow guarantee that bytes outside the visible span are well-defined and
valid UTF-8). That was true before this PR (e.g., ArrayData did not validate
bytes outside the visible span) and is consistent with the general Arrow
philosophy of allowing arbitrary data to appear outside the visible span
(similarly to how null slots can contain arbitrary data).
I had Claude look for a way to do that with safe code today and it didn't
find a way to do it -- but if such a way does exist, I think it would clearly
be buggy code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
