kevinjqliu opened a new issue, #599: URL: https://github.com/apache/parquet-format/issues/599
## Background Parquet classifies new logical types as **forward compatible**: > "New logical types are considered forward compatible despite the loss of semantic meaning." -- https://parquet.apache.org/docs/file-format/versions/#compatibility-categories **Readers that do not recognize a logical type should still read the underlying physical representation rather than reject the file.** This is increasingly important for `VARIANT`, `GEOMETRY`, `GEOGRAPHY`, and future logical types. Related discussion: * https://lists.apache.org/thread/flfzz94ftdrdop9d5b0o1hkqkprzj3l5 * An older `parquet-java` reader (1.15.2) failed to read a Parquet file containing the newer `VARIANT` logical type. The reader rejected the file instead of ignoring the unknown logical annotation and exposing the underlying physical representation. ## Shared test files | File | Purpose | |------|---------| | [`case-001.parquet`](https://github.com/apache/parquet-testing/blob/master/shredded_variant/case-001.parquet) | File containing the `VARIANT` logical type | | [`unknown-logical-type.parquet`](https://github.com/apache/parquet-testing/blob/master/data/unknown-logical-type.parquet) | File containing an unknown logical type | Expected behavior: - File metadata and schema can be parsed. - Underlying physical data can be read. - Unsupported logical semantics are ignored. - The reader does not fail solely because the logical type is unknown. ## Compatibility status (NOTE: WIP, still filling this in) Implementation list from https://parquet.apache.org/docs/file-format/implementationstatus/ | Implementation | Version | Unknown logical type | `VARIANT` fallback | Issue / PR | |---------------|---------|----------------------|--------------------|------------| | parquet-java | | | | | | Arrow C++ | | | | | | arrow-rs | | | | | | Arrow Go | | | | | | DuckDB | | | | | | Polars | | | | | | cuDF | | | | | | Hyparquet | | | | | ## Completion criteria - Record results and tested versions for each implementation. - Open upstream issues for failures. - Add regression tests using the shared `parquet-testing` files. - Backport fixes where practical. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
