vinothchandar commented on a change in pull request #4038:
URL: https://github.com/apache/hudi/pull/4038#discussion_r753688167
##########
File path:
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java
##########
@@ -191,16 +194,103 @@ public HoodieTableConfig(FileSystem fs, String metaPath,
String payloadClassName
/**
* For serializing and de-serializing.
- *
*/
public HoodieTableConfig() {
super();
}
+ private void fetchConfigs(FileSystem fs, String metaPath) throws IOException
{
+ Path cfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE);
+ try (FSDataInputStream is = fs.open(cfgPath)) {
+ props.load(is);
+ } catch (IOException ioe) {
+ if (!fs.exists(cfgPath)) {
Review comment:
yes why incur that cost everytime? this way we only incur it in degraded
state
##########
File path:
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java
##########
@@ -191,16 +194,103 @@ public HoodieTableConfig(FileSystem fs, String metaPath,
String payloadClassName
/**
* For serializing and de-serializing.
- *
*/
public HoodieTableConfig() {
super();
}
+ private void fetchConfigs(FileSystem fs, String metaPath) throws IOException
{
+ Path cfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE);
+ try (FSDataInputStream is = fs.open(cfgPath)) {
+ props.load(is);
+ } catch (IOException ioe) {
+ if (!fs.exists(cfgPath)) {
+ LOG.warn("Run `table recover-configs` if config update/delete failed
midway. Falling back to backed up configs.");
+ // try the backup. this way no query ever fails if update fails midway.
+ Path backupCfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE_BACKUP);
+ try (FSDataInputStream is = fs.open(backupCfgPath)) {
+ props.load(is);
+ }
+ } else {
+ throw ioe;
+ }
+ }
+ }
+
+ public static void recover(FileSystem fs, Path metadataFolder) throws
IOException {
+ Path cfgPath = new Path(metadataFolder, HOODIE_PROPERTIES_FILE);
+ Path backupCfgPath = new Path(metadataFolder,
HOODIE_PROPERTIES_FILE_BACKUP);
+ recoverIfNeeded(fs, cfgPath, backupCfgPath);
+ }
+
+ static void recoverIfNeeded(FileSystem fs, Path cfgPath, Path backupCfgPath)
throws IOException {
+ if (!fs.exists(cfgPath)) {
Review comment:
there is not in cloud storage. its atomic. On hdfs there is a small
window for this - but very rare since the write will fit within the block size.
It's solvable by adding a checksum entry to the file and validating. I have a
note in the description around this.
##########
File path:
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java
##########
@@ -191,16 +194,103 @@ public HoodieTableConfig(FileSystem fs, String metaPath,
String payloadClassName
/**
* For serializing and de-serializing.
- *
*/
public HoodieTableConfig() {
super();
}
+ private void fetchConfigs(FileSystem fs, String metaPath) throws IOException
{
+ Path cfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE);
+ try (FSDataInputStream is = fs.open(cfgPath)) {
+ props.load(is);
+ } catch (IOException ioe) {
+ if (!fs.exists(cfgPath)) {
+ LOG.warn("Run `table recover-configs` if config update/delete failed
midway. Falling back to backed up configs.");
+ // try the backup. this way no query ever fails if update fails midway.
+ Path backupCfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE_BACKUP);
+ try (FSDataInputStream is = fs.open(backupCfgPath)) {
+ props.load(is);
+ }
+ } else {
+ throw ioe;
+ }
+ }
+ }
+
+ public static void recover(FileSystem fs, Path metadataFolder) throws
IOException {
+ Path cfgPath = new Path(metadataFolder, HOODIE_PROPERTIES_FILE);
+ Path backupCfgPath = new Path(metadataFolder,
HOODIE_PROPERTIES_FILE_BACKUP);
+ recoverIfNeeded(fs, cfgPath, backupCfgPath);
+ }
+
+ static void recoverIfNeeded(FileSystem fs, Path cfgPath, Path backupCfgPath)
throws IOException {
+ if (!fs.exists(cfgPath)) {
Review comment:
cc @prashantwason I filed HUDI-2809 for this. if you have time, please
help! :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
