Github user tgravescs commented on a diff in the pull request:
https://github.com/apache/spark/pull/21953#discussion_r207258035
--- Diff: core/src/main/scala/org/apache/spark/util/Utils.scala ---
@@ -460,7 +461,14 @@ private[spark] object Utils extends Logging {
if (useCache && fetchCacheEnabled) {
val cachedFileName = s"${url.hashCode}${timestamp}_cache"
val lockFileName = s"${url.hashCode}${timestamp}_lock"
- val localDir = new File(getLocalDir(conf))
+ var localDir: File = null
+ // Set the cachedLocalDir for the first time and re-use it later
+ this.synchronized {
--- End diff --
if we want to be more efficient to not hit the synchronized block each time
we could do one extra check before it to check cachedLocalDir.isEmpty. Only
if its empty do we enter synchronized and then recheck if still empty.
this would be very similar to getOrCreateLocalRootDirs
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
