[
https://issues.apache.org/jira/browse/HUDI-1268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liwei updated HUDI-1268:
------------------------
Status: In Progress (was: Open)
> Fix UpgradeDowngrade fs Rename issue for hdfs and aliyun oss
> ------------------------------------------------------------
>
> Key: HUDI-1268
> URL: https://issues.apache.org/jira/browse/HUDI-1268
> Project: Apache Hudi
> Issue Type: Bug
> Components: Writer Core
> Reporter: leesf
> Assignee: liwei
> Priority: Major
> Labels: pull-request-available
> Fix For: 0.6.1
>
>
> 一、issue：in UpgradeDowngrade.run()
> fs.rename(updatedPropsFilePath, propsFilePath);
> the fs.rename have different action,such as:
> // a) for hdfs : if propsFilePath already exist,fs.rename will not replace
> propsFilePath, but just return false
> // b) for localfs: if propsFilePath already exist,fs.rename will replace
> propsFilePath, and return ture
> // c) for aliyun ossfs: if propsFilePath already exist,will throw
> FileAlreadyExistsException
> // so we should delete the old propsFilePath. also upgrade and downgrade is
> Idempotent
>
> 二、for aliyun oss filesystem
> when using HoodieWriteClient API to write data to hudi with following config:
> ```
> Properties properties = new Properties();
> properties.setProperty(HoodieTableConfig.HOODIE_TABLE_NAME_PROP_NAME,
> tableName);
> properties.setProperty(HoodieTableConfig.HOODIE_TABLE_TYPE_PROP_NAME,
> tableType.name());
> properties.setProperty(HoodieTableConfig.HOODIE_PAYLOAD_CLASS_PROP_NAME,
> OverwriteWithLatestAvroPayload.class.getName());
> properties.setProperty(HoodieTableConfig.HOODIE_ARCHIVELOG_FOLDER_PROP_NAME,
> "archived");
> return HoodieTableMetaClient.initTableAndGetMetaClient(hadoopConf, basePath,
> properties);
> ```
> the exception will be thrown with FileAlreadyExistsException in aliyun OSS,
> after debugging, it is the following code throws the exception.
>
> ```
> // Rename the .updated file to hoodie.properties. This is atomic in hdfs, but
> not in cloud stores.
> // But as long as this does not leave a partial hoodie.properties file, we
> are okay.
> fs.rename(updatedPropsFilePath, propsFilePath);
> ```
> however, we would ignore the FileAlreadyExistsException since
> hoodie.properties already exists.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
