Thanks Micah! I wanted to confirm my understanding. I would love to fix older readers so they can still open parquet files with the variant type. Do you think its worth backporting a fix to older readers as a patch release?
I'll also do some research for other parquet clients based on when they added support for the variant type and their corresponding behavior for older readers. Best, Kevin Liu On 2026/06/30 19:13:45 Micah Kornfield wrote: > Hi Kevin, > > This sounds like a bug. > > if an older reader sees a logical type > > it does not know how to decode, should it treat that logical annotation as > > absent, preserve the physical schema, and continue as long as the > > unsupported column is not interpreted semantically? > > > I think most implementations that support it, go even further and still > allow the column to be projected. The main other thing that unknown > logical types should do is not surface statistics (as that is a form of > semantic interpretation). > > Cheers, > Micah > > > > On Sat, Jun 27, 2026 at 11:01 AM Kevin Liu <[email protected]> wrote: > > > Hi all, > > > > I wanted to ask for guidance on the expected behavior for older > > parquet-java readers when reading a file that contains a newer logical > > type. > > > > In PARQUET-3633 [1], parquet-java 1.15.x fails while opening a file with a > > VARIANT column, even when the read projection only requests a separate id > > column. The failure happens during footer schema conversion, before > > projection is applied. I put up a small repro PR here [2]. > > > > VARIANT was added in parquet-format 2.12.0 and picked up by parquet-java > > starting in the 1.16 line. parquet-java 1.15.x uses parquet-format 2.10.0, > > so this is an older-reader/newer-logical-type compatibility case. I’m > > looking at the parquet-format compatibility guidance [3], which calls out > > adding new logical types as a forward-compatible change. In this case, I’d > > expect parquet-java 1.15.x not to understand VARIANT semantics, but still > > to tolerate the annotation in the footer, preserve the physical schema, and > > allow reads that only project non-VARIANT columns. > > > > The behavior I’m wondering about is: if an older reader sees a logical type > > it does not know how to decode, should it treat that logical annotation as > > absent, preserve the physical schema, and continue as long as the > > unsupported column is not interpreted semantically? That would allow > > projected reads of unrelated columns to succeed without implying that the > > older reader understands VARIANT. > > > > I also sketched one possible fix/regression test here [4]. It treats > > unknown LogicalType union fields as absent during footer conversion, keeps > > the physical schema usable, and verifies that reading only the projected id > > column succeeds for the shredded_variant/case-001.parquet fixture from > > apache/parquet-testing. > > > > Does that match the intended forward-compatibility behavior for new logical > > types? > > > > Thanks, > > > > Kevin > > > > [1] https://github.com/apache/parquet-java/issues/3633 > > [2] https://github.com/kevinjqliu/parquet-java/pull/1 > > [3] > > > > https://github.com/apache/parquet-format/blob/master/CONTRIBUTING.md#compatibility-and-feature-enablement > > [4] https://github.com/kevinjqliu/parquet-java/pull/2 > > >
