raghuvanshraj opened a new pull request, #11961: URL: https://github.com/apache/arrow-rs/pull/11961
# Which issue does this PR close? - Closes #11917 # Rationale for this change The Parquet writer annotates Variant columns with `VARIANT` but leaves `specification_version` unset. The Parquet spec requires the version in the declaration ([`LogicalTypes.md` L602-L603](https://github.com/apache/parquet-format/blob/e365c6f7c05eea69c5aa11f1ead6c3db261ca9f9/LogicalTypes.md#L602-L603)), and Spark 4.1 only reads `VARIANT(1)` ([`ParquetSchemaConverter.scala#L395`](https://github.com/apache/spark/blob/branch-4.1/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetSchemaConverter.scala#L395)), so every Variant file written by arrow-rs today fails in Spark with `PARQUET_TYPE_NOT_RECOGNIZED`. # What changes are included in this PR? - `logical_type_for_struct` now writes `LogicalType::variant(Some(VARIANT_SPEC_VERSION as i8))`. The constant is the existing header-version constant in `parquet-variant` (`CORRECT_VERSION_VALUE`), renamed to `VARIANT_SPEC_VERSION` and made `pub`, so the annotation and the metadata header share one source of truth. - Existing `write_logical_type` test in `parquet/src/variant.rs` updated to assert the version. - New `arrow_variant_to_parquet_variant` schema-conversion test in `parquet/src/arrow/schema/mod.rs`, next to the UUID and JSON ones. The reader is unchanged: `try_add_extension_type` already matches `LogicalType::Variant(_)`, so files written before this change still read. # Are these changes tested? Yes. Both tests fail on `main` and pass with the change. `cargo test -p parquet --features variant_experimental,arrow,arrow_canonical_extension_types,serde --lib`, `--test variant_integration` and `--test arrow_writer` all pass. # Are there any significant user-facing changes? Variant files written with `variant_experimental` now carry `VARIANT(1)` and are readable by Spark. No API change. # AI Use Prepared with AI assistance; code, tests and links reviewed and run by the author. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
