klion26 commented on code in PR #10352: URL: https://github.com/apache/arrow-rs/pull/10352#discussion_r3596311259
########## parquet-variant/tests/proptest.rs: ########## @@ -0,0 +1,237 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information +// regarding copyright ownership. The ASF licenses this file +// to you under the Apache License, Version 2.0 (the +// "License"); you may not use this file except in compliance +// with the License. You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, +// software distributed under the License is distributed on an +// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +// KIND, either express or implied. See the License for the +// specific language governing permissions and limitations +// under the License. + +//! Property tests for variant decoding and validation. +//! +//! Two families of invariant are checked here: +//! +//! 1. **Untrusted input.** [`Variant::try_new`] and [`VariantMetadata::try_new`] are fallible APIs +//! over bytes read from a file. They must return `Err` on malformed input rather than panic, and +//! anything they accept must then be panic-free to access, as [`VariantMetadata`] documents. +//! 2. **Writer/reader agreement.** Anything [`VariantBuilder`] emits must validate and read back +//! unchanged, whatever order the dictionary happens to be in. +//! +//! The generators deliberately produce degenerate shapes: empty field names, multi-byte UTF-8, and +//! interior NULs. This matters more than the properties themselves -- narrowing `arb_key` to +//! `[a-z]+` makes these tests pass against code with known bugs. +//! +//! Objects are generated structurally rather than as random bytes: random bytes are rejected by +//! header and offset validation long before reaching the object logic, so they never exercise it. + +use parquet_variant::{Variant, VariantBuilder, VariantMetadata}; +use proptest::prelude::*; +use std::collections::HashSet; + +/// The empty metadata dictionary. +const EMPTY_METADATA: &[u8] = &[1, 0, 0]; Review Comment: we can reuse the `EMPTY_VARIANT_METADATA_BYTES` in `variant/metdata.rs` ########## parquet-variant/src/variant/object.rs: ########## Review Comment: Do we need to update the doc here to respect the uniqueness? ########## parquet-variant/src/variant/metadata.rs: ########## @@ -309,15 +309,19 @@ impl<'m> VariantMetadata<'m> { current_offset = next_offset; } } else { - // Validate offsets are in-bounds and monotonically increasing - // - // Since shallow validation ensures the first and last offsets are in bounds, - // we can also verify all offsets are in-bounds by checking if - // offsets are monotonically increasing - if !offsets.is_sorted_by(|a, b| a < b) { - return Err(ArrowError::InvalidArgumentError( - "offsets not monotonically increasing".to_string(), - )); + // Slicing each dictionary value validates that offsets are in-bounds, non-decreasing, + // and land on UTF-8 character boundaries. Equal offsets are legal: they encode an + // empty dictionary entry. + let mut current_offset = offsets.next().unwrap_or(0); + for next_offset in offsets { + value_buffer + .get(current_offset..next_offset) Review Comment: I'm curious about the cause of the offset not land on UTF-8 character boundaries. is the data corrupt or the writer did not write the right data? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
