nsivabalan commented on a change in pull request #1858:
URL: https://github.com/apache/hudi/pull/1858#discussion_r458365915
##########
File path:
hudi-client/src/main/java/org/apache/hudi/client/AbstractHoodieWriteClient.java
##########
@@ -190,6 +192,7 @@ public HoodieMetrics getMetrics() {
*/
protected HoodieTable getTableAndInitCtx(WriteOperationType operationType) {
HoodieTableMetaClient metaClient = createMetaClient(true);
+ mayBeUpradeOrDowngrade(metaClient);
Review comment:
@vinothchandar : is this the right place to call upgrade/downgrade. If
not, please advise.
##########
File path:
hudi-client/src/main/java/org/apache/hudi/table/UpgradeDowngradeHelper.java
##########
@@ -0,0 +1,175 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hudi.table;
+
+import org.apache.hudi.common.table.HoodieTableConfig;
+import org.apache.hudi.common.table.HoodieTableMetaClient;
+import org.apache.hudi.common.table.HoodieTableVersion;
+import org.apache.hudi.exception.HoodieException;
+
+import org.apache.hadoop.fs.FSDataInputStream;
+import org.apache.hadoop.fs.FSDataOutputStream;
+import org.apache.hadoop.fs.FileUtil;
+import org.apache.hadoop.fs.Path;
+
+import java.io.IOException;
+import java.util.Properties;
+
+/**
+ * Helper class to assist in upgrading/downgrading Hoodie when there is a
version change.
+ */
+public class UpgradeDowngradeHelper {
+
+ public static final String HOODIE_ORIG_PROPERTY_FILE =
"hoodie.properties.orig";
+
+ /**
+ * Perform Upgrade or Downgrade steps if required and updated table version
if need be.
+ * <p>
+ * Starting from version 0.6.0, this upgrade/downgrade step will be added in
all write paths.
+ * Essentially, if a dataset was created using any pre 0.6.0(for eg 0.5.3),
and Hoodie verion was upgraded to 0.6.0, there are some upgrade steps need
+ * to be executed before doing any writes.
+ * Similarly, if a dataset was created using 0.6.0 and then hoodie was
downgraded, some downgrade steps need to be executed before proceeding w/ any
writes.
+ * On a high level, these are the steps performed
+ * Step1 : Understand current hoodie version and table version from
hoodie.properties file
+ * Step2 : Fix any residues from previous upgrade/downgrade
+ * Step3 : Check for version upgrade/downgrade.
+ * Step4 : If upgrade/downgrade is required, perform the steps required for
the same.
+ * Step5 : Copy hoodie.properties -> hoodie.properties.orig
+ * Step6 : Update hoodie.properties file with current table version
+ * Step7 : Delete hoodie.properties.orig
+ * </p>
+ * @param metaClient instance of {@link HoodieTableMetaClient} to use
+ * @throws IOException
+ */
+ public static void doUpgradeOrDowngrade(HoodieTableMetaClient metaClient)
throws IOException {
+ // Fetch version from property file and current version
+ HoodieTableVersion versionFromPropertyFile =
metaClient.getTableConfig().getHoodieTableVersionFromPropertyFile();
+ HoodieTableVersion currentVersion =
metaClient.getTableConfig().getCurrentHoodieTableVersion();
+
+ Path metaPath = new Path(metaClient.getMetaPath());
+ Path originalHoodiePropertyPath =
getOrigHoodiePropertyFilePath(metaPath.toString());
+
+ boolean updateTableVersionInPropertyFile = false;
+
+ if (metaClient.getFs().exists(originalHoodiePropertyPath)) {
+ // if hoodie.properties.orig exists, rename to hoodie.properties and
skip upgrade/downgrade step
+ metaClient.getFs().rename(originalHoodiePropertyPath,
getHoodiePropertyFilePath(metaPath.toString()));
+ updateTableVersionInPropertyFile = true;
+ } else {
+ // upgrade or downgrade if there is a version mismatch
+ if (versionFromPropertyFile != currentVersion) {
+ updateTableVersionInPropertyFile = true;
+ if (versionFromPropertyFile == HoodieTableVersion.PRE_ZERO_SIZE_ZERO
&& currentVersion == HoodieTableVersion.ZERO_SIX_ZERO) {
+ upgradeFromOlderToZeroSixZero();
+ } else if (versionFromPropertyFile == HoodieTableVersion.ZERO_SIX_ZERO
&& currentVersion == HoodieTableVersion.PRE_ZERO_SIZE_ZERO) {
+ downgradeFromZeroSixZeroToPreZeroSixZero();
+ } else {
+ throw new HoodieException("Illegal state wrt table versions. Version
from proerpty file " + versionFromPropertyFile + " and current version " +
currentVersion);
+ }
+ }
+ }
+
+ /**
+ * If table version needs to be updated in hoodie.properties file.
+ * Step1: Copy hoodie.properties to hoodie.properties.orig
+ * Step2: add table.version to hoodie.properties
+ * Step3: delete hoodie.properties.orig
+ */
+ if (updateTableVersionInPropertyFile) {
+ updateTableVersionInMetaPath(metaClient);
Review comment:
@vinothchandar : after updating the hoodie.properties file, I haven't
reloaded the meta client as of now. It is just the table version in memory that
has changed and no other code blocks should access table.version. Do you think
is reload of metaclient mandatory if we update hoodie.properties w/ new table
version?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
