RussellSpitzer opened a new issue #1635: URL: https://github.com/apache/iceberg/issues/1635
When older files are read with Iceberg the metrics are improperly evaluated. We incorrectly assumes that there are 0 nonNull values in every row group which leads to the reader skipping all row groups. https://github.com/RussellSpitzer/iceberg/blob/4d1fc91d6528db88548c506a0b110b50121d67ff/parquet/src/main/java/org/apache/iceberg/parquet/ParquetMetricsRowGroupFilter.java#L299-L302 In a Parquet 1.5.0 file we were working with the stats are reported as [num_nulls: 0, min/max not defined] This causes us to pass `colStats.isEmpty` because these are valid settings but `hasNonNullValues` is still false. This leads us to returned `ROWS_CANNOT_MATCH` and skip the row group. These failures make any filter pushdown to the files return 0 records regardless of whether or not the filter actually matches anything. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
