This is an automated email from the ASF dual-hosted git repository.
zhouky pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-celeborn.git
The following commit(s) were added to refs/heads/main by this push:
new 413e4f3b8 [CELEBORN-1222] Fix Celeborn worker won't record HDFS writer
413e4f3b8 is described below
commit 413e4f3b8015b6e0c121162c8a60154934e23d4f
Author: mingji <[email protected]>
AuthorDate: Tue Jan 16 10:08:03 2024 +0800
[CELEBORN-1222] Fix Celeborn worker won't record HDFS writer
### What changes were proposed in this pull request?
To record hdfs writer in worker.
To fix a bug introduced by
https://github.com/apache/incubator-celeborn/pull/2130.
### Why are the changes needed?
If the hdfs writer won't be recorded, the worker won't clean the HDFS
shuffle file if a partition is broken until the master cleans the HDFS.
### Does this PR introduce _any_ user-facing change?
NO.
### How was this patch tested?
GA and cluster.
Closes #2227 from FMX/b1222.
Authored-by: mingji <[email protected]>
Signed-off-by: zky.zhoukeyong <[email protected]>
---
.../apache/celeborn/service/deploy/worker/storage/StorageManager.scala | 2 ++
1 file changed, 2 insertions(+)
diff --git
a/worker/src/main/scala/org/apache/celeborn/service/deploy/worker/storage/StorageManager.scala
b/worker/src/main/scala/org/apache/celeborn/service/deploy/worker/storage/StorageManager.scala
index 6fcc98050..f2171b7f5 100644
---
a/worker/src/main/scala/org/apache/celeborn/service/deploy/worker/storage/StorageManager.scala
+++
b/worker/src/main/scala/org/apache/celeborn/service/deploy/worker/storage/StorageManager.scala
@@ -399,6 +399,8 @@ final private[worker] class StorageManager(conf:
CelebornConf, workerSource: Abs
workingDirWriters.computeIfAbsent(workingDir,
workingDirWriterListFunc).put(
diskFileInfo.getFilePath,
writer)
+ } else {
+ hdfsWriters.put(diskFileInfo.getFilePath, writer)
}
writer
}
