Re: [PR] [GLUTEN-10707] Improve SparkDirectoryUtil with AtomicReference [incubator-gluten]

2025-09-16 Thread via GitHub 


github-actions[bot] commented on PR #10708:
URL: 
https://github.com/apache/incubator-gluten/pull/10708#issuecomment-3296867964

   Run Gluten Clickhouse CI on x86


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Re: [PR] [GLUTEN-10707] Improve SparkDirectoryUtil with AtomicReference [incubator-gluten]

2025-09-16 Thread via GitHub 


beliefer commented on code in PR #10708:
URL: 
https://github.com/apache/incubator-gluten/pull/10708#discussion_r2351572271


##
gluten-core/src/main/scala/org/apache/spark/util/SparkDirectoryUtil.scala:
##
@@ -67,29 +68,31 @@ class SparkDirectoryUtil private (val roots: Array[String]) 
extends Logging {
 }
 
 object SparkDirectoryUtil extends Logging {
-  private var INSTANCE: SparkDirectoryUtil = _
+  private val INSTANCE = new AtomicReference[SparkDirectoryUtil]()
 
-  def init(conf: SparkConf): Unit = synchronized {
+  def init(conf: SparkConf): Unit = {
 val roots = Utils.getConfiguredLocalDirs(conf)
 init(roots)
   }
 
-  private def init(roots: Array[String]): Unit = synchronized {
-if (INSTANCE == null) {
-  INSTANCE = new SparkDirectoryUtil(roots)
+  private def init(roots: Array[String]): Unit = {
+if (INSTANCE.get() == null) {
+  INSTANCE.compareAndSet(null, new SparkDirectoryUtil(roots))

Review Comment:
   Got it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Re: [PR] [GLUTEN-10707] Improve SparkDirectoryUtil with AtomicReference [incubator-gluten]

2025-09-15 Thread via GitHub 


beliefer commented on code in PR #10708:
URL: 
https://github.com/apache/incubator-gluten/pull/10708#discussion_r2350501074


##
gluten-core/src/main/scala/org/apache/spark/util/SparkDirectoryUtil.scala:
##
@@ -67,29 +68,31 @@ class SparkDirectoryUtil private (val roots: Array[String]) 
extends Logging {
 }
 
 object SparkDirectoryUtil extends Logging {
-  private var INSTANCE: SparkDirectoryUtil = _
+  private val INSTANCE = new AtomicReference[SparkDirectoryUtil]()
 
-  def init(conf: SparkConf): Unit = synchronized {
+  def init(conf: SparkConf): Unit = {
 val roots = Utils.getConfiguredLocalDirs(conf)
 init(roots)
   }
 
-  private def init(roots: Array[String]): Unit = synchronized {
-if (INSTANCE == null) {
-  INSTANCE = new SparkDirectoryUtil(roots)
+  private def init(roots: Array[String]): Unit = {
+if (INSTANCE.get() == null) {
+  INSTANCE.compareAndSet(null, new SparkDirectoryUtil(roots))

Review Comment:
   I'm sorry. I didn't understand what you said. Could you tell me more ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Re: [PR] [GLUTEN-10707] Improve SparkDirectoryUtil with AtomicReference [incubator-gluten]

2025-09-15 Thread via GitHub 


beliefer commented on PR #10708:
URL: 
https://github.com/apache/incubator-gluten/pull/10708#issuecomment-3294614589

   Run Gluten Clickhouse CI on x86


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Re: [PR] [GLUTEN-10707] Improve SparkDirectoryUtil with AtomicReference [incubator-gluten]

2025-09-15 Thread via GitHub 


github-actions[bot] commented on PR #10708:
URL: 
https://github.com/apache/incubator-gluten/pull/10708#issuecomment-3291973105

   Run Gluten Clickhouse CI on x86


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Re: [PR] [GLUTEN-10707] Improve SparkDirectoryUtil with AtomicReference [incubator-gluten]

2025-09-15 Thread via GitHub 


github-actions[bot] commented on PR #10708:
URL: 
https://github.com/apache/incubator-gluten/pull/10708#issuecomment-3291973108

   https://github.com/apache/incubator-gluten/issues/10707


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Re: [PR] [GLUTEN-10707] Improve SparkDirectoryUtil with AtomicReference [incubator-gluten]

2025-09-15 Thread via GitHub 


zhztheplayer commented on code in PR #10708:
URL: 
https://github.com/apache/incubator-gluten/pull/10708#discussion_r2349607173


##
gluten-core/src/main/scala/org/apache/spark/util/SparkDirectoryUtil.scala:
##
@@ -67,29 +68,31 @@ class SparkDirectoryUtil private (val roots: Array[String]) 
extends Logging {
 }
 
 object SparkDirectoryUtil extends Logging {
-  private var INSTANCE: SparkDirectoryUtil = _
+  private val INSTANCE = new AtomicReference[SparkDirectoryUtil]()
 
-  def init(conf: SparkConf): Unit = synchronized {
+  def init(conf: SparkConf): Unit = {
 val roots = Utils.getConfiguredLocalDirs(conf)
 init(roots)
   }
 
-  private def init(roots: Array[String]): Unit = synchronized {
-if (INSTANCE == null) {
-  INSTANCE = new SparkDirectoryUtil(roots)
+  private def init(roots: Array[String]): Unit = {
+if (INSTANCE.get() == null) {
+  INSTANCE.compareAndSet(null, new SparkDirectoryUtil(roots))

Review Comment:
   The code is workable but the constructor of `SparkDirectoryUtil` may be 
unnecessarily called in concurrent invocations.
   
   We can use `AtomicBoolean` instead if unable to avoid this.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]