amogh-jahagirdar commented on code in PR #16692:
URL: https://github.com/apache/iceberg/pull/16692#discussion_r3597095738
##########
spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/sql/TestFilterPushDown.java:
##########
@@ -647,6 +648,47 @@ public void testVariantExtractFiltering() {
});
}
+ @TestTemplate
+ public void testFilterPushdownOnInitialDefaultColumnAbsentFromFile() {
Review Comment:
https://github.com/cbb330/iceberg-apache/pull/4/changes published a PR to
@cbb330 branch, I think we have an additional issue in the parquet reader. The
default reader in BaseParquetReaders reads a missing nested field via
ConstantReader whose definition level is a fixed number. So in the case there's
a nested field with a non-null default but the ancestor is missing, _and_ the
filter _only_ contains a predicate on this specific nested field we end up
applying the default value unconditionally rather than using null as we should.
If there's an additional nested filter we end up being OK because we detect the
discrepancy in the definition level and use null as expected.
I think we should probably fix this issue separately in a follow on (also
qualifies for 1.10.3 imo); I think it should be separate because it's a more
fundamental fix in the parquet reader code and the change is already a good
focused fix as it is.
cc @stevenzwu @cbb330 @rdblue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
