We've discussed this issue during the parquet sync today. To summarize: - We agreed that this is a bug and should be fixed. I'll put up a PR for it. - I proposed to backport the fix and create patch releases so older parquet-java versions can be fixed without being forced to upgrade to newer versions. - I’ll also do some follow-up testing across other popular implementations, since this behavior may not be limited to parquet-java. Will mentioned that Go and FastParquet will crash with new logical types. - There was a suggestion to add the “unknown logical type” case to to the parquet testing suite.
Best, Kevin Liu On 2026/07/01 04:28:14 Kevin Liu wrote: > Thanks Micah! I wanted to confirm my understanding. > I would love to fix older readers so they can still open parquet files with > the variant type. Do you think its worth backporting a fix to older readers > as a patch release? > > I'll also do some research for other parquet clients based on when they added > support for the variant type and their corresponding behavior for older > readers. > > Best, > Kevin Liu > > On 2026/06/30 19:13:45 Micah Kornfield wrote: > > Hi Kevin, > > > > This sounds like a bug. > > > > if an older reader sees a logical type > > > it does not know how to decode, should it treat that logical annotation as > > > absent, preserve the physical schema, and continue as long as the > > > unsupported column is not interpreted semantically? > > > > > > I think most implementations that support it, go even further and still > > allow the column to be projected. The main other thing that unknown > > logical types should do is not surface statistics (as that is a form of > > semantic interpretation). > > > > Cheers, > > Micah > > > > > > > > On Sat, Jun 27, 2026 at 11:01 AM Kevin Liu <[email protected]> wrote: > > > > > Hi all, > > > > > > I wanted to ask for guidance on the expected behavior for older > > > parquet-java readers when reading a file that contains a newer logical > > > type. > > > > > > In PARQUET-3633 [1], parquet-java 1.15.x fails while opening a file with a > > > VARIANT column, even when the read projection only requests a separate id > > > column. The failure happens during footer schema conversion, before > > > projection is applied. I put up a small repro PR here [2]. > > > > > > VARIANT was added in parquet-format 2.12.0 and picked up by parquet-java > > > starting in the 1.16 line. parquet-java 1.15.x uses parquet-format 2.10.0, > > > so this is an older-reader/newer-logical-type compatibility case. I’m > > > looking at the parquet-format compatibility guidance [3], which calls out > > > adding new logical types as a forward-compatible change. In this case, I’d > > > expect parquet-java 1.15.x not to understand VARIANT semantics, but still > > > to tolerate the annotation in the footer, preserve the physical schema, > > > and > > > allow reads that only project non-VARIANT columns. > > > > > > The behavior I’m wondering about is: if an older reader sees a logical > > > type > > > it does not know how to decode, should it treat that logical annotation as > > > absent, preserve the physical schema, and continue as long as the > > > unsupported column is not interpreted semantically? That would allow > > > projected reads of unrelated columns to succeed without implying that the > > > older reader understands VARIANT. > > > > > > I also sketched one possible fix/regression test here [4]. It treats > > > unknown LogicalType union fields as absent during footer conversion, keeps > > > the physical schema usable, and verifies that reading only the projected > > > id > > > column succeeds for the shredded_variant/case-001.parquet fixture from > > > apache/parquet-testing. > > > > > > Does that match the intended forward-compatibility behavior for new > > > logical > > > types? > > > > > > Thanks, > > > > > > Kevin > > > > > > [1] https://github.com/apache/parquet-java/issues/3633 > > > [2] https://github.com/kevinjqliu/parquet-java/pull/1 > > > [3] > > > > > > https://github.com/apache/parquet-format/blob/master/CONTRIBUTING.md#compatibility-and-feature-enablement > > > [4] https://github.com/kevinjqliu/parquet-java/pull/2 > > > > > >
