nsivabalan commented on code in PR #8771:
URL: https://github.com/apache/hudi/pull/8771#discussion_r1199675883
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -252,8 +253,9 @@ private void enablePartition(final MetadataPartitionType
partitionType, final Ho
* @param writeConfig {@code HoodieWriteConfig} of the main
dataset writer
* @param failedWritesCleaningPolicy Cleaning policy on failed writes
*/
- private HoodieWriteConfig createMetadataWriteConfig(
+ public static HoodieWriteConfig createMetadataWriteConfig(
Review Comment:
can we move static methods to HoodieTableMetadataUtil ?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/UpgradeDowngrade.java:
##########
@@ -105,6 +108,25 @@ public boolean needsUpgradeOrDowngrade(HoodieTableVersion
toVersion) {
* @param instantTime current instant time that should not be touched.
*/
public void run(HoodieTableVersion toVersion, String instantTime) {
+ // Change metadata table version automatically
+ if (toVersion.versionCode() >= HoodieTableVersion.FOUR.versionCode()) {
+ String metadataTablePath = HoodieTableMetadata.getMetadataTableBasePath(
+ metaClient.getBasePathV2().toString());
+ try {
+ if (metaClient.getFs().exists(new Path(metadataTablePath))) {
+ HoodieTableMetaClient mdtMetaClient = HoodieTableMetaClient.builder()
+
.setConf(metaClient.getHadoopConf()).setBasePath(metadataTablePath).build();
+ HoodieWriteConfig mdtWriteConfig =
HoodieBackedTableMetadataWriter.createMetadataWriteConfig(
+ config, HoodieFailedWritesCleaningPolicy.LAZY);
Review Comment:
So, this takes care of both upgrade and downgrade right. i.e. upgrade will
upgrade both data table and mdt table and downgrade will downgrade both ?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/table/upgrade/UpgradeDowngrade.java:
##########
@@ -105,6 +108,25 @@ public boolean needsUpgradeOrDowngrade(HoodieTableVersion
toVersion) {
* @param instantTime current instant time that should not be touched.
*/
public void run(HoodieTableVersion toVersion, String instantTime) {
+ // Change metadata table version automatically
+ if (toVersion.versionCode() >= HoodieTableVersion.FOUR.versionCode()) {
+ String metadataTablePath = HoodieTableMetadata.getMetadataTableBasePath(
+ metaClient.getBasePathV2().toString());
+ try {
+ if (metaClient.getFs().exists(new Path(metadataTablePath))) {
+ HoodieTableMetaClient mdtMetaClient = HoodieTableMetaClient.builder()
+
.setConf(metaClient.getHadoopConf()).setBasePath(metadataTablePath).build();
+ HoodieWriteConfig mdtWriteConfig =
HoodieBackedTableMetadataWriter.createMetadataWriteConfig(
+ config, HoodieFailedWritesCleaningPolicy.LAZY);
Review Comment:
why this has to be lazy? we can rollback any pending operations right ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
