kevinjqliu opened a new issue, #3633: URL: https://github.com/apache/parquet-java/issues/3633
### Summary `parquet-java` 1.15.x fails when opening a Parquet file that contains a `VARIANT` column, even if the application only requests other columns and does not request the `VARIANT` column. The failure happens while converting the footer schema, before projection is applied. The parquet-format compatibility guidance says that new logical types are intended to be forward compatible: Based on that, I think older readers should be able to tolerate an unknown logical type when that field is not part of the requested projection. ### Repro This can be reproduced with the `VARIANT` fixture in `apache/parquet-testing`: ```bash git clone https://github.com/apache/parquet-testing.git PARQUET_FILE="$PWD/parquet-testing/shredded_variant/case-001.parquet" ``` The file contains an `id` column and a `var` column annotated with `VARIANT`. Using `parquet-java` 1.15.x, try to read only `id`: ```java Configuration conf = new Configuration(false); conf.set( ReadSupport.PARQUET_READ_SCHEMA, "message table { required int32 id; }"); Path input = new Path(args[0]); // Fails before the projected read can happen ParquetMetadata footer = ParquetFileReader.readFooter(conf, input); System.out.println(footer.getFileMetaData().getSchema()); ``` ### Actual behavior The read fails during footer schema conversion: ```text NullPointerException: Cannot invoke "org.apache.parquet.format.LogicalType$_Fields.ordinal()" because the return value of "org.apache.parquet.format.LogicalType.getSetField()" is null ``` The stack points to `ParquetMetadataConverter.getLogicalTypeAnnotation`. ### Expected behavior Older readers should tolerate an unknown logical type in the footer when the field is not projected, preserving the physical schema or treating the field as having no known logical annotation. They should only fail if the unsupported field is actually read or interpreted semantically. ### Notes This is relevant for files written by newer or external writers that contain `VARIANT`, where older readers only need unrelated columns. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
