(celeborn) branch main updated: [CELEBORN-2178] Close hadoopFs FileSystem for stopping master

[nicholasjiang]

This is an automated email from the ASF dual-hosted git repository.

nicholasjiang pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/celeborn.git


The following commit(s) were added to refs/heads/main by this push:
     new a518f222d [CELEBORN-2178] Close hadoopFs FileSystem for stopping master
a518f222d is described below

commit a518f222d48b74868f981b57bd79b07edb591fc3
Author: xxx <953396...@qq.com>
AuthorDate: Tue Oct 28 10:35:47 2025 +0800

    [CELEBORN-2178] Close hadoopFs FileSystem for stopping master
    
    ### What changes were proposed in this pull request?
    
    Close hadoopFs FileSystem when master is stopped.
    
    ### Why are the changes needed?
    
    Close hadoopFs FileSystem when master is stopped.
    
    ### Does this PR introduce _any_ user-facing change?
    
    NO
    
    ### How was this patch tested?
    
    CI
    
    Closes #3509 from xy2953396112/CELEBORN-2178.
    
    Authored-by: xxx <953396...@qq.com>
    Signed-off-by: SteNicholas <programg...@163.com>
---
 .../org/apache/celeborn/service/deploy/master/Master.scala | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git 
a/master/src/main/scala/org/apache/celeborn/service/deploy/master/Master.scala 
b/master/src/main/scala/org/apache/celeborn/service/deploy/master/Master.scala
index 92a7dfe09..0799af45c 100644
--- 
a/master/src/main/scala/org/apache/celeborn/service/deploy/master/Master.scala
+++ 
b/master/src/main/scala/org/apache/celeborn/service/deploy/master/Master.scala
@@ -389,7 +389,19 @@ private[celeborn] class Master(
     messagesHelper.close()
 
     metricsSystem.stop()
-
+    if (hadoopFs != null) {
+      hadoopFs.asScala.foreach {
+        case (storageType, fs) =>
+          if (fs != null) {
+            try {
+              fs.close()
+            } catch {
+              case t: Throwable =>
+                logError(s"Close $storageType FileSystem ${fs.getUri} 
failed.", t)
+            }
+          }
+      }
+    }
     logInfo("Celeborn Master is stopped.")
   }