zhangyue19921010 commented on code in PR #6133:
URL: https://github.com/apache/hudi/pull/6133#discussion_r1028240012
##########
hudi-common/src/main/java/org/apache/hudi/common/util/MarkerUtils.java:
##########
@@ -208,10 +216,56 @@ public static Set<String> readMarkersFromFile(Path
markersFilePath, Serializable
fsDataInputStream = fs.open(markersFilePath);
markers = new
HashSet<>(FileIOUtils.readAsUTFStringLines(fsDataInputStream));
} catch (IOException e) {
- throw new HoodieIOException("Failed to read MARKERS file " +
markersFilePath, e);
+ if (ignoreException) {
+ LOG.warn("IOException occurs during read MARKERS file, ", e);
+ } else {
+ throw new HoodieIOException("Failed to read MARKERS file " +
markersFilePath, e);
+ }
} finally {
closeQuietly(fsDataInputStream);
}
return markers;
}
+
+ /**
+ * Reads files containing the markers written by timeline-server-based
marker mechanism locally instead of using cluster Context.
+ *
+ * @param markerDir marker directory.
+ * @param fileSystem file system to use.
+ * @return A {@code Map} of file name to the set of markers stored in the
file.
+ */
+ public static Set<String>
readTimelineServerBasedMarkersFromFileSystemLocally(String markerDir,
FileSystem fileSystem) {
Review Comment:
Nice catch. Changed.
##########
hudi-timeline-service/src/main/java/org/apache/hudi/timeline/service/handlers/MarkerHandler.java:
##########
@@ -109,6 +126,7 @@ public void stop() {
}
dispatchingExecutorService.shutdown();
batchingExecutorService.shutdown();
+ checkers.values().forEach(ExecutorService::shutdown);
Review Comment:
Removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
