rdblue commented on issue #684: Support Decimal in Parquet predicates URL: https://github.com/apache/incubator-iceberg/issues/684#issuecomment-575329875 @chenjunjiedada, I think I was wrong about this issue. I was referring to the note here, where we are converting Iceberg filters to Parquet filters: https://github.com/apache/incubator-iceberg/blob/master/parquet/src/main/java/org/apache/iceberg/parquet/ParquetFilters.java#L207 But that code path is only used when reading Parquet files with a Parquet `ReadSupport`. We don't do that for Spark, we use [the Iceberg reader](https://github.com/apache/incubator-iceberg/blob/master/parquet/src/main/java/org/apache/iceberg/parquet/Parquet.java#L231-L233) now to avoid the overhead of `ReadSupport` and to implement our own row group filters (that support, for example, Decimal and IN). The new read path already has support for decimal because it [converts Parquet min/max stats](https://github.com/apache/incubator-iceberg/blob/master/parquet/src/main/java/org/apache/iceberg/parquet/ParquetMetricsRowGroupFilter.java#L104) to the objects that Iceberg uses. That [conversion supports decimal](https://github.com/apache/incubator-iceberg/blob/master/parquet/src/main/java/org/apache/iceberg/parquet/ParquetConversions.java#L71). So the new read path that we actually use already supports decimal predicates. We could go back and fix the old read path, but I don't think that we want to spend the effort since the problem is knowing what physical type to pass to Parquet (int, long, or binary). I think we should mark this issue completed. What do you think?
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
