cetra3 commented on code in PR #10352:
URL: https://github.com/apache/arrow-rs/pull/10352#discussion_r3599937225
##########
parquet-variant/src/variant/metadata.rs:
##########
@@ -309,15 +309,19 @@ impl<'m> VariantMetadata<'m> {
current_offset = next_offset;
}
} else {
- // Validate offsets are in-bounds and monotonically increasing
- //
- // Since shallow validation ensures the first and last offsets
are in bounds,
- // we can also verify all offsets are in-bounds by checking if
- // offsets are monotonically increasing
- if !offsets.is_sorted_by(|a, b| a < b) {
- return Err(ArrowError::InvalidArgumentError(
- "offsets not monotonically increasing".to_string(),
- ));
+ // Slicing each dictionary value validates that offsets are
in-bounds, non-decreasing,
+ // and land on UTF-8 character boundaries. Equal offsets are
legal: they encode an
+ // empty dictionary entry.
+ let mut current_offset = offsets.next().unwrap_or(0);
+ for next_offset in offsets {
+ value_buffer
+ .get(current_offset..next_offset)
Review Comment:
Yeah this is to guard against corrupt data. I don't think there is a way to
construct this shape of data with `VariantBuilder` but you can easily craft
corrupted bytes that will hit a panic.
This is the same check as the other branch makes a handful of lines above,
so nothing new here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
