stream2000 opened a new pull request, #8804:
URL: https://github.com/apache/hudi/pull/8804
### Change Logs
Improve update mechanism for HoodieTableProperties.
Currently, the update mechanism for HoodieTableProperties has two problem.
First, when fetching configs from file system, when encountering
IOEXception, we will check whether the cfg file exists. If not, we will
fallback to read the backup file. If the cfg file exists we will throw out the
exception as following.
```txt
Caused by:
java.io.FileNotFoundException:path_to_table/.hoodoie/hoodie.properties: No such
file or directory!
at
org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem.getFileStatus(AliyunOSSFileSystem.java:279)
~[aps-api.jar:?]
at
org.apache.hadoop.fs.aliyun.oss.AliyunOSSFileSystem.open(AliyunOSSFileSystem.java:622)
~[aps-api.jar:?]
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:976)
~[aps-api.jar:?]
at
org.apache.hudi.common.fs.HoodieWrapperFileSystem.open(HoodieWrapperFileSystem.java:460)
~[aps-api.jar:?]
at
org.apache.hudi.common.table.HoodieTableConfig.fetchConfigs(HoodieTableConfig.java:331)
~[aps-api.jar:?]
at
org.apache.hudi.common.table.HoodieTableConfig.<init>(HoodieTableConfig.java:273)
~[aps-api.jar:?]
... 27 more
```
This is because there is another writer updating the config, it will delete
the file first and then write it back afterward.
Secondly, when updating the table config, we have the following code which
will create an empty config file first then write the content to it. Reader
will find a totally empty config files in some cases so we need to update the
config file atomicly.
```java
try (FSDataInputStream in = fs.open(backupCfgPath);
FSDataOutputStream out = fs.create(cfgPath, true)) {
Properties props = new TypedProperties();
props.load(in);
modifyFn.accept(props, modifyProps);
checksum = storeProperties(props, out);
}
```
### Impact
Improve update mechanism for HoodieTableProperties.
### Risk level (write none, low medium or high below)
Low
### Documentation Update
None
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Change Logs and Impact were stated clearly
- [x] Adequate tests were added if applicable
- [ ] CI passed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
