JingsongLi commented on code in PR #8399:
URL: https://github.com/apache/paimon/pull/8399#discussion_r3503011617
##########
paimon-spark/paimon-spark-common/src/main/java/org/apache/paimon/spark/SparkFilterConverter.java:
##########
@@ -198,26 +203,74 @@ private static boolean isNaN(Object value) {
}
public Object convertLiteral(String field, Object value) {
- return convertLiteral(fieldIndex(field), value);
+ FieldInfo fieldInfo = resolveField(field);
+ return convertLiteral(fieldInfo.type(), value);
}
public String convertString(String field, Object value) {
Object literal = convertLiteral(field, value);
return literal == null ? null : literal.toString();
}
- private int fieldIndex(String field) {
- int index = rowType.getFieldIndex(field);
- // TODO: support nested field
- if (index == -1) {
+ private static class FieldInfo {
+ private final Transform transform;
+ private final DataType type;
+
+ public FieldInfo(Transform transform, DataType type) {
+ this.transform = transform;
+ this.type = type;
+ }
+
+ public Transform transform() {
+ return transform;
+ }
+
+ public DataType type() {
+ return type;
+ }
+ }
+
+ private FieldInfo resolveField(String field) {
+ String[] parts = field.split("\\.");
+ int topLevelIndex = rowType.getFieldIndex(parts[0]);
+ if (topLevelIndex == -1) {
throw new UnsupportedOperationException(
- String.format("Nested field '%s' is unsupported.", field));
+ String.format("Field '%s' is not found in table schema.",
parts[0]));
+ }
+
+ DataType fieldType;
+ if (parts.length == 1) {
+ fieldType = rowType.getTypeAt(topLevelIndex);
+ } else {
+ fieldType = getNestedFieldType(rowType, parts);
+ if (fieldType == null) {
+ throw new UnsupportedOperationException(
+ String.format("Nested field '%s' is unsupported.",
field));
+ }
+ }
+
+ Transform transform = new FieldTransform(new FieldRef(topLevelIndex,
field, fieldType));
Review Comment:
This does not actually evaluate the nested field. FieldTransform only reads
InternalRowUtils.get(row, fieldRef.index(), fieldRef.type()), so for a filter
like a.b = 1 this FieldRef still reads top-level column a (index 0) but with b
type. That can make predicate.test(row) and statistics pruning read the struct
column as an int/string instead of traversing into b. The new test only checks
toString(), so it misses this runtime behavior. Please add a real nested-field
transform/path traversal, or keep these filters unsupported until evaluation
and stats pruning can handle nested paths correctly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
