JingsongLi commented on code in PR #9423:
URL: https://github.com/apache/paimon/pull/9423#discussion_r4056186134
##########
paimon-format/src/main/java/org/apache/parquet/filter2/predicate/ParquetFilters.java:
##########
@@ -534,35 +573,100 @@ private static PrimitiveType timestampPrimitiveType(
|| timestampType.isAdjustedToUTC() != expectedAdjustedToUtc) {
throw new UnsupportedOperationException();
}
- return primitiveType;
+ return column;
}
- private static PrimitiveType primitiveType(
+ /**
+ * The column the file holds for {@code fieldRef}, with the file's own
spelling of the path.
+ * Callers that build a parquet column must use {@link FileColumn#path}: a
{@link PrimitiveType}
+ * only knows its own leaf name, so rebuilding the column from it drops
the enclosing path and
+ * addresses a column the file does not have.
+ */
+ private static FileColumn fileColumn(
FieldRef fieldRef, MessageType fileSchema, boolean caseSensitive) {
- PrimitiveType matched = findPrimitiveType(fieldRef, fileSchema,
caseSensitive);
+ FileColumn matched = findFileColumn(fieldRef, fileSchema,
caseSensitive);
if (matched == null) {
throw new UnsupportedOperationException();
}
return matched;
}
+ /** A column the file actually holds: its own spelling of the path, and
its physical type. */
+ private static class FileColumn {
+
+ private final String path;
+ private final PrimitiveType type;
+
+ private FileColumn(String path, PrimitiveType type) {
+ this.path = path;
+ this.type = type;
+ }
+ }
+
/**
* The file's column for {@code fieldRef}, or null when the file has no
such column. A column
* that exists but is not primitive cannot carry a predicate at all, so it
is rejected outright.
+ *
+ * <p>{@code fieldRef} names a nested field with dots ({@code addr.city}),
which is resolved by
+ * descending the file's groups. A top-level column matching the whole
name wins over that walk,
+ * keeping flat columns spelled with dots resolving as they always did.
parquet-mr identifies
+ * columns by dot-joined path too, so it cannot tell the two apart either
way.
*/
@Nullable
- private static PrimitiveType findPrimitiveType(
+ private static FileColumn findFileColumn(
FieldRef fieldRef, MessageType fileSchema, boolean caseSensitive) {
- // Paimon predicates currently reference top-level fields only. Nested
field
- // predicates are rejected before reaching the format reader.
- for (Type field : fileSchema.getFields()) {
+ Type matched = findChild(fileSchema, fieldRef.name(), caseSensitive);
+ if (matched != null) {
+ return toFileColumn(matched.getName(), matched);
+ }
+
+ String[] parts = fieldRef.name().split("\\.");
Review Comment:
[P2] Do not reinterpret a missing dotted top-level field as a nested path
This fallback runs for every FieldRef, not only refs synthesized from
NestedFieldTransform. A format table can declare a top-level BIGINT field named
s.a while a file lacks that exact field but contains s -> a as INT32. The
reader treats the top-level field as missing/null, so s.a IS NULL must keep
every row, but this walk resolves the nested column and emits eq([s,a], null);
in an end-to-end reproduction it pruned both rows (actual [] vs expected [1,
2]). Please retain explicit path origin/components: only nested transforms
should descend groups, while an ordinary dotted FieldRef without an exact
top-level match should conservatively skip pushdown (or use a literal
one-component ColumnPath).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
