lokeshj1703 commented on code in PR #9467:
URL: https://github.com/apache/hudi/pull/9467#discussion_r1299179444
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/SixToFiveDowngradeHandler.java:
##########
@@ -65,13 +79,31 @@ public Map<ConfigProperty, String>
downgrade(HoodieWriteConfig config, HoodieEng
}
/**
- * Record-level index, a new partition in metadata table, was first added in
- * 0.14.0 ({@link HoodieTableVersion#SIX}. Any downgrade from this version
- * should remove this partition.
+ * Utility method to run compaction for MOR table as part of downgrade step.
*/
- private static void removeRecordIndexIfNeeded(HoodieTable table,
HoodieEngineContext context) {
- HoodieTableMetaClient metaClient = table.getMetaClient();
- deleteMetadataTablePartition(metaClient, context,
MetadataPartitionType.RECORD_INDEX, false);
+ private void runCompaction(HoodieTable table, HoodieEngineContext context,
HoodieWriteConfig config,
+ SupportsUpgradeDowngrade upgradeDowngradeHelper) {
+ try {
+ if (table.getMetaClient().getTableType() ==
HoodieTableType.MERGE_ON_READ) {
+ // The log block version has been upgraded in version six so
compaction is required for downgrade.
+ // set required configs for scheduling compaction.
+
HoodieInstantTimeGenerator.setCommitTimeZone(table.getMetaClient().getTableConfig().getTimelineTimezone());
+ HoodieWriteConfig compactionConfig =
HoodieWriteConfig.newBuilder().withProps(config.getProps()).build();
+ compactionConfig.setValue(HoodieCompactionConfig.INLINE_COMPACT.key(),
"true");
+
compactionConfig.setValue(HoodieCompactionConfig.INLINE_COMPACT_NUM_DELTA_COMMITS.key(),
"1");
+
compactionConfig.setValue(HoodieCompactionConfig.INLINE_COMPACT_TRIGGER_STRATEGY.key(),
CompactionTriggerStrategy.NUM_COMMITS.name());
+
compactionConfig.setValue(HoodieCompactionConfig.COMPACTION_STRATEGY.key(),
UnBoundedCompactionStrategy.class.getName());
+ compactionConfig.setValue(HoodieMetadataConfig.ENABLE.key(), "false");
+ EmbeddedTimelineServerHelper.createEmbeddedTimelineService(context,
config);
Review Comment:
Addressed. This was required earlier but not needed any more.
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/SixToFiveDowngradeHandler.java:
##########
@@ -39,20 +47,26 @@
import static
org.apache.hudi.common.table.HoodieTableConfig.TABLE_METADATA_PARTITIONS;
import static
org.apache.hudi.common.table.HoodieTableConfig.TABLE_METADATA_PARTITIONS_INFLIGHT;
-import static
org.apache.hudi.metadata.HoodieTableMetadataUtil.deleteMetadataTablePartition;
/**
* Downgrade handle to assist in downgrading hoodie table from version 6 to 5.
* To ensure compatibility, we need recreate the compaction requested file to
* .aux folder.
+ * Since version 6 includes a new schema field for metadata table(MDT),
+ * the MDT needs to be deleted during downgrade to avoid column drop error.
+ * Also log block version was upgraded in version 6, therefore full compaction
needs
+ * to be completed during downgrade to avoid write failures.
Review Comment:
Addressed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
