cbb330 commented on code in PR #16692:
URL: https://github.com/apache/iceberg/pull/16692#discussion_r3563968288
##########
parquet/src/main/java/org/apache/iceberg/parquet/ParquetMetricsRowGroupFilter.java:
##########
@@ -128,6 +130,22 @@ public Boolean or(Boolean leftResult, Boolean rightResult)
{
return leftResult || rightResult;
}
+ @Override
+ @SuppressWarnings("unchecked")
+ public <T> Boolean predicate(BoundPredicate<T> pred) {
+ // a column absent from the file reads as its initial-default, so
evaluate the predicate
+ // against that default
+ if (pred.term() instanceof BoundReference) {
+ int id = ((BoundReference<T>) pred.term()).fieldId();
+ Types.NestedField field = schema.findField(id);
+ if (field != null && field.initialDefault() != null &&
!valueCounts.containsKey(id)) {
+ return pred.test((T) field.initialDefault()) ? ROWS_MIGHT_MATCH :
ROWS_CANNOT_MATCH;
+ }
Review Comment:
@pvary @stevenzwu thanks for highlighting this. I've handled the struct case
+ added coverage.
The current impl only supports a null default at struct/list/map nodes.
initialDefault is a Literal<?> type, so the spec's non-null empty-struct ({})
marker isn't supported yet. Sub-field struct values come from each child's own
default (that's what the spec link Steven shared describes as well). So this
patch is correct for structs with that premise, and whole-container
({}/list/map) defaults would need separate work when they land in the future.
While looking into this, I did find and fix an issue:
- a nested leaf takes its default only when its parent struct is present, so
if an ancestor is null, the leaf is null regardless of default.
- So, for nested leaves, I now evaluate **default OR null** (`matchesDefault
|| matchesNull(pred)`), not just default. Otherwise we'd incorrectly skip
groups with real nulls that may match the predicate seeking nulls (e.g.
`location.country IS NULL`). Reworked the nested test into
`testNestedColumnMayReadAsDefaultOrNull` to assert default-or-null.
- The updated test checks the cases that matter for nested fields: `IS NULL`
reads, `= 'CA'` skips, `= 'US'` and `notNull` read. Other predicate types
(startsWith, ranges, isNaN, non-string defaults) are already tested on the
top-level default column, so I didn't repeat them here.
##########
parquet/src/main/java/org/apache/iceberg/parquet/ParquetMetricsRowGroupFilter.java:
##########
@@ -128,6 +130,22 @@ public Boolean or(Boolean leftResult, Boolean rightResult)
{
return leftResult || rightResult;
}
+ @Override
+ @SuppressWarnings("unchecked")
+ public <T> Boolean predicate(BoundPredicate<T> pred) {
+ // a column absent from the file reads as its initial-default, so
evaluate the predicate
+ // against that default
+ if (pred.term() instanceof BoundReference) {
+ int id = ((BoundReference<T>) pred.term()).fieldId();
+ Types.NestedField field = schema.findField(id);
+ if (field != null && field.initialDefault() != null &&
!valueCounts.containsKey(id)) {
+ return pred.test((T) field.initialDefault()) ? ROWS_MIGHT_MATCH :
ROWS_CANNOT_MATCH;
+ }
Review Comment:
@pvary @stevenzwu thanks for highlighting this. I've handled the struct case
+ added coverage.
The current impl only supports a null default at struct/list/map nodes.
initialDefault is a Literal<?> type, so the spec's non-null empty-struct ({})
marker isn't supported yet. Sub-field struct values come from each child's own
default (that's what the spec link Steven shared describes as well). So this
patch is correct for structs with that premise, and whole-container
({}/list/map) defaults would need separate work when they land in the future.
While looking into this, I did find and fix an issue:
- a nested leaf takes its default only when its parent struct is present, so
if an ancestor is null, the leaf is null regardless of default.
- So, for nested leaves, I now evaluate **default OR null** (`matchesDefault
|| matchesNull(pred)`), not just default. Otherwise we'd incorrectly skip
groups with real nulls that may match the predicate seeking nulls (e.g.
`location.country IS NULL`). Reworked the nested test into
`testNestedColumnMayReadAsDefaultOrNull` to assert default-or-null.
- The updated test checks the cases that matter for nested fields: `IS NULL`
reads, `= 'CA'` skips, `= 'US'` and `notNull` read. Other predicate types
(startsWith, ranges, isNaN, non-string defaults) are already tested on the
top-level default column, so I didn't repeat them here.
cc: @amogh-jahagirdar
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
