JingsongLi commented on code in PR #9423:
URL: https://github.com/apache/paimon/pull/9423#discussion_r3930362467
##########
paimon-format/src/main/java/org/apache/parquet/filter2/predicate/ParquetFilters.java:
##########
@@ -273,9 +278,31 @@ public FilterPredicate visitNotIn(FieldRef fieldRef,
List<Object> literals) {
throw new UnsupportedOperationException();
}
+ /**
+ * A nested field carries no index into the file, only a path, so it
is re-dispatched under
+ * a {@link FieldRef} naming that path. Every other transform - casts,
string functions -
+ * has no column of its own to filter on and is given up here.
+ */
@Override
public FilterPredicate visitNonFieldLeaf(LeafPredicate predicate) {
- throw new UnsupportedOperationException();
+ if (!(predicate.transform() instanceof NestedFieldTransform)) {
+ throw new UnsupportedOperationException();
+ }
+ NestedFieldTransform nested = (NestedFieldTransform)
predicate.transform();
+ // The path reaches parquet-mr as a dot-joined string, which it
splits back into
+ // components. A component that itself contains a dot does not
survive that round trip:
+ // the filter would address a column the file does not hold, and a
missing column reads
+ // as all-null, pruning row groups that actually match. Give up
the pruning instead.
+ if (nested.fieldRef().name().indexOf('.') >= 0) {
+ throw new UnsupportedOperationException();
+ }
+ for (String component : nested.path()) {
+ if (component.indexOf('.') >= 0) {
+ throw new UnsupportedOperationException();
+ }
+ }
+ FieldRef pathRef = new FieldRef(UNUSED_INDEX, nested.fieldName(),
nested.outputType());
Review Comment:
[P1] Reject collisions between a nested path and a dotted top-level name
Joining the components here is still ambiguous even when none of the
individual components contains a dot. A valid schema can contain both `s ROW<a
INT>` and a top-level column named `s.a` (the existing flat dotted-name
coverage confirms the latter is supported). A predicate on the nested `s.a` is
re-dispatched as `FieldRef("s.a")`; `findFileColumn` checks an exact top-level
name before walking the split components, so it binds this predicate to the
top-level `s.a` column instead of `s -> a`.
If the top-level column's row-group stats do not match while the nested
column does, parquet-mr prunes the row group and Spark's residual filter never
sees the matching row. Please retain the component path through column
resolution/build a `ColumnPath`, or at minimum reject nested pushdown when the
file schema has a top-level field equal to the dot-joined path. An end-to-end
row-group test with both columns and opposing values should expose the false
negative.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
