(incubator-celeborn) branch main updated: [CELEBORN-1230] Check working directory read and write error without init delay

[zhouky]

This is an automated email from the ASF dual-hosted git repository.

zhouky pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/incubator-celeborn.git


The following commit(s) were added to refs/heads/main by this push:
     new e54a2b7e2 [CELEBORN-1230] Check working directory read and write error 
without init delay
e54a2b7e2 is described below

commit e54a2b7e2f463e201288596e9bec075f88ee80ed
Author: Fei Wang <fwan...@ebay.com>
AuthorDate: Fri Feb 16 00:07:01 2024 +0800

    [CELEBORN-1230] Check working directory read and write error without init 
delay
    
    ### What changes were proposed in this pull request?
    In this pr, when getting device disk info, we check the dir writable to 
make sure that the capacity reported to celeborn master are correct and does no 
include non-writable directories.
    ### Why are the changes needed?
    To ignore bad disk when initializing the worker.
    
    ### Does this PR introduce _any_ user-facing change?
    No.
    
    ### How was this patch tested?
    
    Existing UT.
    
    Closes #2233 from turboFei/check_disk_init.
    
    Lead-authored-by: Fei Wang <fwan...@ebay.com>
    Co-authored-by: Keyong Zhou <zky.zhoukey...@alibaba-inc.com>
    Signed-off-by: zky.zhoukeyong <zky.zhoukey...@alibaba-inc.com>
---
 .../apache/celeborn/service/deploy/worker/storage/DeviceMonitor.scala   | 2 +-
 .../org/apache/celeborn/service/deploy/worker/storage/WorkerSuite.scala | 1 +
 2 files changed, 2 insertions(+), 1 deletion(-)

diff --git 
a/worker/src/main/scala/org/apache/celeborn/service/deploy/worker/storage/DeviceMonitor.scala
 
b/worker/src/main/scala/org/apache/celeborn/service/deploy/worker/storage/DeviceMonitor.scala
index ea78358c8..7efa568b1 100644
--- 
a/worker/src/main/scala/org/apache/celeborn/service/deploy/worker/storage/DeviceMonitor.scala
+++ 
b/worker/src/main/scala/org/apache/celeborn/service/deploy/worker/storage/DeviceMonitor.scala
@@ -158,7 +158,7 @@ class LocalDeviceMonitor(
           }
         }
       },
-      diskCheckInterval,
+      0,
       diskCheckInterval,
       TimeUnit.MILLISECONDS)
   }
diff --git 
a/worker/src/test/scala/org/apache/celeborn/service/deploy/worker/storage/WorkerSuite.scala
 
b/worker/src/test/scala/org/apache/celeborn/service/deploy/worker/storage/WorkerSuite.scala
index 77a881896..a8572c371 100644
--- 
a/worker/src/test/scala/org/apache/celeborn/service/deploy/worker/storage/WorkerSuite.scala
+++ 
b/worker/src/test/scala/org/apache/celeborn/service/deploy/worker/storage/WorkerSuite.scala
@@ -42,6 +42,7 @@ class WorkerSuite extends AnyFunSuite with BeforeAndAfterEach 
{
 
   override def beforeEach(): Unit = {
     assert(null == worker)
+    conf.set(s"${CelebornConf.WORKER_DISK_MONITOR_CHECKLIST.key}", "readwrite")
   }
 
   override def afterEach(): Unit = {