codope commented on code in PR #12350:
URL: https://github.com/apache/hudi/pull/12350#discussion_r1862714024
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -2514,6 +2532,27 @@ public static HoodieData<HoodieRecord>
convertMetadataToPartitionStatsRecords(Ho
}
}
+ /**
+ * Given table schema and field to index, checks if field's data type are
supported.
+ *
+ * @param columnToIndex column to index
+ * @param tableSchema table schema
+ * @return true if field's data type is supported, false otherwise
+ */
+ public static boolean validateDataTypeForPartitionStats(String
columnToIndex, Schema tableSchema) {
+ Schema fieldSchema = getNestedFieldSchemaFromWriteSchema(tableSchema,
columnToIndex);
+ // Exclude fields based on logical type
+ if ((fieldSchema.getType() == Schema.Type.INT || fieldSchema.getType() ==
Schema.Type.LONG)
+ && fieldSchema.getLogicalType() != null) {
+
+ // Skip fields with logical types DATE or TIME_MILLIS for INT,
TIMESTAMP_MILLIS for LONG
Review Comment:
I was checking the timestamop based logical types in avro -
https://avro.apache.org/docs/1.11.0/spec.html#Logical+Types
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
