zhuxiangyi commented on code in PR #9423:
URL: https://github.com/apache/paimon/pull/9423#discussion_r3940713148
##########
paimon-format/src/main/java/org/apache/parquet/filter2/predicate/ParquetFilters.java:
##########
@@ -273,9 +278,31 @@ public FilterPredicate visitNotIn(FieldRef fieldRef,
List<Object> literals) {
throw new UnsupportedOperationException();
}
+ /**
+ * A nested field carries no index into the file, only a path, so it
is re-dispatched under
+ * a {@link FieldRef} naming that path. Every other transform - casts,
string functions -
+ * has no column of its own to filter on and is given up here.
+ */
@Override
public FilterPredicate visitNonFieldLeaf(LeafPredicate predicate) {
- throw new UnsupportedOperationException();
+ if (!(predicate.transform() instanceof NestedFieldTransform)) {
+ throw new UnsupportedOperationException();
+ }
+ NestedFieldTransform nested = (NestedFieldTransform)
predicate.transform();
+ // The path reaches parquet-mr as a dot-joined string, which it
splits back into
+ // components. A component that itself contains a dot does not
survive that round trip:
+ // the filter would address a column the file does not hold, and a
missing column reads
+ // as all-null, pruning row groups that actually match. Give up
the pruning instead.
+ if (nested.fieldRef().name().indexOf('.') >= 0) {
+ throw new UnsupportedOperationException();
+ }
+ for (String component : nested.path()) {
+ if (component.indexOf('.') >= 0) {
+ throw new UnsupportedOperationException();
+ }
+ }
+ FieldRef pathRef = new FieldRef(UNUSED_INDEX, nested.fieldName(),
nested.outputType());
Review Comment:
Confirmed, and it turned out worse than a false negative — it's a hard
failure. findFileColumn does bind to the wrong (top-level) column as you
describe, but parquet-mr re-splits whatever dot-joined name it's handed, so the
FilterPredicate actually ends up addressing the right two-segment s -> a column
chunk — just tagged with the wrong column's physical type.
SchemaCompatibilityValidator catches that mismatch when the file is opened and
throws IllegalArgumentException, so today the query fails outright rather than
silently dropping the row.
This PR takes the "at minimum" option: visitNonFieldLeaf now also rejects
the pushdown when fileSchema actually has a top-level field literally equal to
the joined path, alongside the existing dot-in-component guards.
Tests:
-
ParquetFormatReadWriteTest.testNestedPathCollidingWithADottedTopLevelNameKeepsMatchingRows
— end-to-end, reproduces the crash before the fix.
-
ParquetFiltersTest.testNestedFieldCollidingWithADottedTopLevelSiblingIsNotPushedDown
— filter-level, asserts the pushdown is declined.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
