Hi all, I wanted to ask for guidance on the expected behavior for older parquet-java readers when reading a file that contains a newer logical type.
In PARQUET-3633 [1], parquet-java 1.15.x fails while opening a file with a VARIANT column, even when the read projection only requests a separate id column. The failure happens during footer schema conversion, before projection is applied. I put up a small repro PR here [2]. VARIANT was added in parquet-format 2.12.0 and picked up by parquet-java starting in the 1.16 line. parquet-java 1.15.x uses parquet-format 2.10.0, so this is an older-reader/newer-logical-type compatibility case. I’m looking at the parquet-format compatibility guidance [3], which calls out adding new logical types as a forward-compatible change. In this case, I’d expect parquet-java 1.15.x not to understand VARIANT semantics, but still to tolerate the annotation in the footer, preserve the physical schema, and allow reads that only project non-VARIANT columns. The behavior I’m wondering about is: if an older reader sees a logical type it does not know how to decode, should it treat that logical annotation as absent, preserve the physical schema, and continue as long as the unsupported column is not interpreted semantically? That would allow projected reads of unrelated columns to succeed without implying that the older reader understands VARIANT. I also sketched one possible fix/regression test here [4]. It treats unknown LogicalType union fields as absent during footer conversion, keeps the physical schema usable, and verifies that reading only the projected id column succeeds for the shredded_variant/case-001.parquet fixture from apache/parquet-testing. Does that match the intended forward-compatibility behavior for new logical types? Thanks, Kevin [1] https://github.com/apache/parquet-java/issues/3633 [2] https://github.com/kevinjqliu/parquet-java/pull/1 [3] https://github.com/apache/parquet-format/blob/master/CONTRIBUTING.md#compatibility-and-feature-enablement [4] https://github.com/kevinjqliu/parquet-java/pull/2
