[
https://issues.apache.org/jira/browse/HUDI-1138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17393927#comment-17393927
]
ASF GitHub Bot commented on HUDI-1138:
--------------------------------------
nsivabalan commented on a change in pull request #3233:
URL: https://github.com/apache/hudi/pull/3233#discussion_r683052454
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java
##########
@@ -601,4 +604,84 @@ public static HoodieWrapperFileSystem getFs(String path,
SerializableConfigurati
.filter(fileStatus ->
!fileStatus.getPath().toString().contains(HoodieTableMetaClient.METAFOLDER_NAME))
.collect(Collectors.toList());
}
+
+ /**
+ * Deletes a directory by deleting sub-paths in parallel on the file system.
+ *
+ * @param hoodieEngineContext {@code HoodieEngineContext} instance
+ * @param fs file system
+ * @param dirPath directory path
+ * @param parallelism parallelism to use for sub-paths
+ * @return {@code true} if the directory is delete; {@code false} otherwise.
+ */
+ public static boolean deleteDir(
+ HoodieEngineContext hoodieEngineContext, FileSystem fs, Path dirPath,
int parallelism) {
+ try {
+ if (fs.exists(dirPath)) {
+ FSUtils.parallelizeSubPathProcess(hoodieEngineContext, fs, dirPath,
parallelism, e -> true,
+ pairOfSubPathAndConf ->
deleteSubPath(pairOfSubPathAndConf.getKey(), pairOfSubPathAndConf.getValue())
+ );
+
+ boolean result = fs.delete(dirPath, true);
+ LOG.info("Removed directory at " + dirPath);
+ return result;
+ }
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ return false;
+ }
+
+ /**
+ * Processes sub-path in parallel.
+ *
+ * @param hoodieEngineContext {@code HoodieEngineContext} instance
+ * @param fs file system
+ * @param dirPath directory path
+ * @param parallelism parallelism to use for sub-paths
+ * @param subPathPredicate predicate to use to filter sub-paths for
processing
+ * @param pairFunction actual processing logic for each sub-path
+ * @param <T> type of result to return for each sub-path
+ * @return a map of sub-path to result of the processing
+ */
+ public static <T> Map<String, T> parallelizeSubPathProcess(
Review comment:
looks neat :)
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java
##########
@@ -601,4 +604,84 @@ public static HoodieWrapperFileSystem getFs(String path,
SerializableConfigurati
.filter(fileStatus ->
!fileStatus.getPath().toString().contains(HoodieTableMetaClient.METAFOLDER_NAME))
.collect(Collectors.toList());
}
+
+ /**
+ * Deletes a directory by deleting sub-paths in parallel on the file system.
+ *
+ * @param hoodieEngineContext {@code HoodieEngineContext} instance
+ * @param fs file system
+ * @param dirPath directory path
+ * @param parallelism parallelism to use for sub-paths
+ * @return {@code true} if the directory is delete; {@code false} otherwise.
+ */
+ public static boolean deleteDir(
+ HoodieEngineContext hoodieEngineContext, FileSystem fs, Path dirPath,
int parallelism) {
+ try {
+ if (fs.exists(dirPath)) {
+ FSUtils.parallelizeSubPathProcess(hoodieEngineContext, fs, dirPath,
parallelism, e -> true,
+ pairOfSubPathAndConf ->
deleteSubPath(pairOfSubPathAndConf.getKey(), pairOfSubPathAndConf.getValue())
+ );
+
+ boolean result = fs.delete(dirPath, true);
+ LOG.info("Removed directory at " + dirPath);
+ return result;
+ }
+ } catch (IOException ioe) {
+ throw new HoodieIOException(ioe.getMessage(), ioe);
+ }
+ return false;
+ }
+
+ /**
+ * Processes sub-path in parallel.
+ *
+ * @param hoodieEngineContext {@code HoodieEngineContext} instance
+ * @param fs file system
+ * @param dirPath directory path
+ * @param parallelism parallelism to use for sub-paths
+ * @param subPathPredicate predicate to use to filter sub-paths for
processing
+ * @param pairFunction actual processing logic for each sub-path
+ * @param <T> type of result to return for each sub-path
+ * @return a map of sub-path to result of the processing
+ */
+ public static <T> Map<String, T> parallelizeSubPathProcess(
Review comment:
Do we have unit tests for these newly added methods. Other devs might
possible start using it in future. So, lets ensure we have good test coverage.
##########
File path: hudi-common/src/main/java/org/apache/hudi/common/fs/FSUtils.java
##########
@@ -601,4 +604,84 @@ public static HoodieWrapperFileSystem getFs(String path,
SerializableConfigurati
.filter(fileStatus ->
!fileStatus.getPath().toString().contains(HoodieTableMetaClient.METAFOLDER_NAME))
.collect(Collectors.toList());
}
+
+ /**
+ * Deletes a directory by deleting sub-paths in parallel on the file system.
+ *
+ * @param hoodieEngineContext {@code HoodieEngineContext} instance
+ * @param fs file system
+ * @param dirPath directory path
+ * @param parallelism parallelism to use for sub-paths
+ * @return {@code true} if the directory is delete; {@code false} otherwise.
+ */
+ public static boolean deleteDir(
Review comment:
since this is refactoring even existing DirectMaker files path, we just
need to be mindful of any regression. We can rely on CI test result. but just a
reminder.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
> Re-implement marker files via timeline server
> ---------------------------------------------
>
> Key: HUDI-1138
> URL: https://issues.apache.org/jira/browse/HUDI-1138
> Project: Apache Hudi
> Issue Type: Improvement
> Components: Writer Core
> Affects Versions: 0.9.0
> Reporter: Vinoth Chandar
> Assignee: Ethan Guo
> Priority: Blocker
> Labels: pull-request-available
> Fix For: 0.9.0
>
>
> Even as you can argue that RFC-15/consolidated metadata, removes the need for
> deleting partial files written due to spark task failures/stage retries. It
> will still leave extra files inside the table (and users will pay for it
> every month) and we need the marker mechanism to be able to delete these
> partial files.
> Here we explore if we can improve the current marker file mechanism, that
> creates one marker file per data file written, by
> Delegating the createMarker() call to the driver/timeline server, and have it
> create marker metadata into a single file handle, that is flushed for
> durability guarantees
>
> P.S: I was tempted to think Spark listener mechanism can help us deal with
> failed tasks, but it has no guarantees. the writer job could die without
> deleting a partial file. i.e it can improve things, but cant provide
> guarantees
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
