szehon-ho commented on code in PR #17578:
URL: https://github.com/apache/iceberg/pull/17578#discussion_r4019774978
##########
parquet/src/main/java/org/apache/iceberg/parquet/PruneColumns.java:
##########
@@ -162,9 +164,83 @@ public Type variant(
@Override
public Type primitive(
org.apache.iceberg.types.Type.PrimitiveType expected, PrimitiveType
primitive) {
+ validatePrimitive(expected, primitive, String.join(".", currentPath()));
return null;
}
+ static void validatePrimitive(
+ org.apache.iceberg.types.Type.PrimitiveType expected, PrimitiveType
primitive, String path) {
+ if (expected != null
+ && (expected.typeId() == TypeID.GEOMETRY || expected.typeId() ==
TypeID.GEOGRAPHY)) {
+ Preconditions.checkArgument(
+
TypeUtil.isPromotionAllowed(MessageTypeToType.convertPrimitive(primitive),
expected),
+ "Cannot read Parquet type %s as Iceberg type %s for field %s",
+ primitive,
+ expected,
+ path);
+ }
+ }
+
+ static void validateFallbackType(org.apache.iceberg.types.Type expected,
Type type, String path) {
+ if (expected == null) {
+ return;
+ }
+
+ if (expected.isPrimitiveType() || type.isPrimitive()) {
+ if (expected.isPrimitiveType() && type.isPrimitive()) {
+ validatePrimitive(expected.asPrimitiveType(), type.asPrimitiveType(),
path);
+ }
+
+ return;
+ }
+
+ GroupType group = type.asGroupType();
+ if (expected.isStructType()) {
+ validateFallbackStruct(expected.asStructType(), group, path);
+ } else if (expected.isListType()) {
+ validateFallbackList(expected.asListType(), group, path);
+ } else if (expected.isMapType()) {
+ validateFallbackMap(expected.asMapType(), group, path);
+ }
+ }
+
+ private static void validateFallbackStruct(StructType expected, GroupType
struct, String path) {
+ List<NestedField> expectedFields = expected.fields();
+ int fieldCount = Math.min(expectedFields.size(), struct.getFieldCount());
+ for (int i = 0; i < fieldCount; i += 1) {
Review Comment:
Do not zip the compressed projected struct with the file fields. When a
preceding nested child is not projected, `expected.fields().get(i)` shifts and
validates the selected child geospatial parameters against the wrong Parquet
field, causing false rejection or missed mismatches. Align projected children
to their original file positions, or validate only when that alignment is known.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
