sdf-jkl opened a new issue, #116: URL: https://github.com/apache/parquet-testing/issues/116
### Summary `shredded_variant` cases **41, 131, 132 and 138** omit the `value` column from a variant group, but are labeled as *valid* cases (plain filenames, no `error_message`, with expected `*.variant.bin` outputs). Per the spec and the discussion in apache/parquet-format#591, such files are out of spec, and the labeling has already misled implementations. They should be relabeled following the existing `-INVALID` convention (as suggested by @alamb in https://github.com/apache/parquet-format/pull/591#issuecomment-4918610771). ### Details [LogicalTypes.md] requires: > The group must contain a field named `metadata` and a field named `value`. and in apache/parquet-format#591 it was confirmed this is intentional and will not be relaxed: writers must always emit the `value` column (all-null is fine), for the top-level group and for shredded field groups alike. The affected cases: | case | test | omission | |---|---|---| | 41 | `testArrayMissingValueColumn` | top-level `value` (typed_value is a LIST) | | 131 | `testMissingValueColumn` | top-level `value` (typed_value is INT32) | | 132 | `testShreddedObjectMissingFieldValueColumn` | `value` inside the shredded object field groups | | 138 | `testShreddedObjectMissingValueColumn` | top-level `value` (typed_value is an object) | For example, `case-131.parquet`: ``` required group var (VARIANT(1)) { required binary metadata; optional int32 typed_value; // no `value` column } ``` and `case-132.parquet` (top level is compliant; the field groups are not): ``` optional group var (VARIANT(1)) { required binary metadata; optional binary value; optional group typed_value { required group a { optional int32 typed_value; } // no `value` required group b { optional binary typed_value (STRING); } // no `value` } } ``` ### Why it matters This corpus is the de facto conformance suite for shredded variant readers, and the mislabeling has already propagated: - **arrow-rs** runs the corpus in CI; these cases passing helped mask both reader leniency and a writer that omitted all-null `value` columns (apache/arrow-rs#10306, reported in apache/parquet-format#591). arrow-rs now rejects cases 41/131/138 and marks them expected-error. - **arrow-go** also runs the corpus (skipping only `-INVALID` filenames) and asserts these cases read successfully; its variant type validation currently requires only "at least one of `value` or `typed_value`", matching the relaxation proposed (and declined) in apache/parquet-format#591. - **DuckDB** does not run this corpus, but it illustrates the conflict: its reader rejects such files outright ("The Variant column must have 'metadata' and 'value' as the first two columns"), so a reader that enforces the spec cannot produce the expected outputs these cases assert. ### Proposed fix Rename the four cases to `case-NNN-INVALID.parquet` (and their `.variant.bin` files accordingly) and add `error_message` entries in `cases.json`, following the existing convention used by cases 43, 84 and 125. If the community instead wants to preserve them as reader-leniency tests (readers *may* read out-of-spec data, per the discussion in apache/parquet-format#591), that would need a new explicit category in `cases.json` — the current binary taxonomy cannot express "may read". [LogicalTypes.md]: https://github.com/apache/parquet-format/blob/master/LogicalTypes.md#variant -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
