stevenzwu commented on code in PR #16692:
URL: https://github.com/apache/iceberg/pull/16692#discussion_r3591691003
##########
spark/v4.1/spark/src/test/java/org/apache/iceberg/spark/sql/TestFilterPushDown.java:
##########
@@ -647,6 +648,47 @@ public void testVariantExtractFiltering() {
});
}
+ @TestTemplate
+ public void testFilterPushdownOnInitialDefaultColumnAbsentFromFile() {
Review Comment:
The nested-field logic added to `testNestedColumnMayReadAsDefaultOrNull` in
`TestMetricsRowGroupFilter` isn't exercised end-to-end here. A Spark scenario
would catch a class of regressions the unit test can't — the full path from
Spark filter push-down through the row-group filter for a nested leaf under an
initial-default backfill (both the `= default` match and the `IS NULL` no-prune
case).
Sketch:
```java
sql("CREATE TABLE %s (id BIGINT, loc STRUCT<city: STRING>) USING iceberg "
+ "TBLPROPERTIES ('format-version'='3')", tableName);
sql("INSERT INTO %s VALUES (1, named_struct('city', 'San Francisco'))",
tableName);
Table table = validationCatalog.loadTable(tableIdent);
table.updateSchema()
.addColumn("loc", "country", Types.StringType.get(),
Expressions.lit("US"))
.commit();
sql("REFRESH TABLE %s", tableName);
sql("INSERT INTO %s VALUES (2, named_struct('city', 'Toronto', 'country',
'CA'))", tableName);
// pre-add file backfills loc.country to 'US'; post-add row 2 has 'CA'
assertThat(sql("SELECT id FROM %s WHERE loc.country = 'US'", tableName))
.containsExactly(row(1L));
// nested leaf could read as null via a null ancestor struct, so IS NULL
must not prune row 1's row group
assertThat(sql("SELECT id FROM %s WHERE loc.country IS NULL",
tableName)).isEmpty();
```
`checkFilters(...)` can wrap the `= 'US'` case if push-down verification is
desired alongside the result set. Mirror the same addition to `spark/v3.5` and
`spark/v4.0`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
