codope commented on code in PR #8609:
URL: https://github.com/apache/hudi/pull/8609#discussion_r1246489239
##########
hudi-common/src/main/java/org/apache/hudi/common/table/HoodieTableConfig.java:
##########
@@ -334,22 +337,43 @@ public HoodieTableConfig() {
super();
}
- private void fetchConfigs(FileSystem fs, String metaPath) throws IOException
{
+ private static TypedProperties fetchConfigs(FileSystem fs, String metaPath)
throws IOException {
Path cfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE);
- try (FSDataInputStream is = fs.open(cfgPath)) {
- props.load(is);
- } catch (IOException ioe) {
- if (!fs.exists(cfgPath)) {
- LOG.warn("Run `table recover-configs` if config update/delete failed
midway. Falling back to backed up configs.");
- // try the backup. this way no query ever fails if update fails midway.
- Path backupCfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE_BACKUP);
- try (FSDataInputStream is = fs.open(backupCfgPath)) {
+ Path backupCfgPath = new Path(metaPath, HOODIE_PROPERTIES_FILE_BACKUP);
+ int readRetryCount = 0;
+ boolean found = false;
+
+ TypedProperties props = new TypedProperties();
+ while (readRetryCount++ < MAX_READ_RETRIES) {
+ for (Path path : Arrays.asList(cfgPath, backupCfgPath)) {
+ // Read the properties and validate that it is a valid file
+ try (FSDataInputStream is = fs.open(path)) {
+ props.clear();
props.load(is);
+ found = true;
+ ValidationUtils.checkArgument(validateChecksum(props));
+ return props;
+ } catch (IOException e) {
+ LOG.warn(String.format("Could not read properties from %s: %s",
path, e));
+ } catch (IllegalArgumentException e) {
+ LOG.warn(String.format("Invalid properties file %s: %s", path,
props));
}
- } else {
- throw ioe;
+ }
+
+ // Failed to read all files so wait before retrying. This can happen in
cases of parallel updates to the properties.
+ try {
+ Thread.sleep(READ_RETRY_DELAY_MSEC);
+ } catch (InterruptedException e) {
+ LOG.warn("Interrupted while waiting");
}
}
+
+ // If we are here then after all retries either no hoodie.properties was
found or only an invalid file was found.
+ if (found) {
+ throw new IllegalArgumentException("hoodie.properties file seems
invalid. Please check for left over `.updated` files if any, manually copy it
to hoodie.properties and retry");
+ } else {
+ throw new HoodieIOException("Could not load Hoodie properties from " +
cfgPath);
Review Comment:
Fixed the deltastreamer tests by modifying the exception message here as
deltastreamer depends on specific messae. Pitfalls of depending on exception
message as business logic! We should try to avoid that as much as apossible.
https://github.com/apache/hudi/blob/b95248e011931f4748a7a9fbb8298cbbb71bda88/hudi-utilities/src/main/java/org/apache/hudi/utilities/deltastreamer/HoodieDeltaStreamer.java#L695-L697
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
