yihua commented on code in PR #11597:
URL: https://github.com/apache/hudi/pull/11597#discussion_r1704793858
##########
hudi-common/src/main/java/org/apache/hudi/metadata/MetadataPartitionType.java:
##########
@@ -99,18 +210,96 @@ public boolean
isMetadataPartitionAvailable(HoodieTableMetaClient metaClient) {
}
return false;
}
+
+ @Override
+ public void constructMetadataPayload(HoodieMetadataPayload payload,
GenericRecord record) {
+ GenericRecord secondaryIndexRecord = getNestedFieldValue(record,
SCHEMA_FIELD_ID_SECONDARY_INDEX);
+ checkState(secondaryIndexRecord != null, "Valid SecondaryIndexMetadata
record expected for type: " +
MetadataPartitionType.SECONDARY_INDEX.getRecordType());
+ payload.secondaryIndexMetadata = new HoodieSecondaryIndexInfo(
+
secondaryIndexRecord.get(SECONDARY_INDEX_FIELD_RECORD_KEY).toString(),
+ (Boolean)
secondaryIndexRecord.get(SECONDARY_INDEX_FIELD_IS_DELETED));
+ }
},
- PARTITION_STATS(HoodieTableMetadataUtil.PARTITION_NAME_PARTITION_STATS,
"partition-stats-") {
+ PARTITION_STATS(HoodieTableMetadataUtil.PARTITION_NAME_PARTITION_STATS,
"partition-stats-", 6) {
Review Comment:
Does PARTITION_STATS index needs custom combine logic in
`#combineMetadataPayloads`?
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieTableMetadataUtil.java:
##########
@@ -2141,6 +2147,118 @@ private static
List<HoodieColumnRangeMetadata<Comparable>> translateWriteStatToF
return getFileStatsRangeMetadata(writeStat.getPartitionPath(),
writeStat.getPath(), datasetMetaClient, columnsToIndex, false);
}
+ public static String getPartitionStatsIndexKey(String partitionPath, String
columnName) {
Review Comment:
In the refactoring of the indexing code to come, we can move these
index-specific util methods to index subclass.
##########
hudi-common/src/main/java/org/apache/hudi/metadata/MetadataPartitionType.java:
##########
@@ -137,6 +327,37 @@ public String getFileIdPrefix() {
return fileIdPrefix;
}
+ public int getRecordType() {
+ return recordType;
+ }
+
+ /**
+ * Construct metadata payload from the given record.
+ */
+ public void constructMetadataPayload(HoodieMetadataPayload payload,
GenericRecord record) {
+ throw new UnsupportedOperationException("MetadataPayload construction not
supported for partition type: " + this);
+ }
+
+ /**
+ * Merge old and new metadata payloads. By default, it returns the newer
payload.
+ * Implementations can override this method to merge the payloads depending
on the partition type.
+ */
+ public HoodieMetadataPayload combineMetadataPayloads(HoodieMetadataPayload
older, HoodieMetadataPayload newer) {
+ return newer;
+ }
+
+ /**
+ * Get the metadata partition type for the given record type.
+ */
+ public static MetadataPartitionType get(int type) {
+ for (MetadataPartitionType partitionType : values()) {
+ if (partitionType.getRecordType() == type) {
+ return partitionType;
+ }
+ }
Review Comment:
Maybe we can construct a map for faster retrieval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
