Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/18537#discussion_r125620580
--- Diff:
mllib/src/test/scala/org/apache/spark/ml/recommendation/ALSSuite.scala ---
@@ -777,15 +778,34 @@ class ALSSuite
}
}
-class ALSCleanerSuite extends SparkFunSuite {
+class ALSCleanerSuite extends SparkFunSuite with BeforeAndAfterEach {
+
+ private var localDir: File = _
--- End diff --
Just to make sure the issue, I debugged as below:
**Before**
```diff
@@ -785,7 +785,10 @@ class ALSCleanerSuite extends SparkFunSuite {
def getAllFiles: Set[File] =
FileUtils.listFiles(localDir, TrueFileFilter.INSTANCE,
TrueFileFilter.INSTANCE).asScala.toSet
try {
+ println(s"Before local dir : ${Utils.getLocalDir(conf)}")
conf.set("spark.local.dir", localDir.getAbsolutePath)
+ println(s"Actual local dir : ${Utils.getLocalDir(conf)}")
+ println(s"Expected local dir root : ${localDir.getAbsolutePath}")
val sc = new SparkContext("local[2]", "test", conf)
try {
sc.setCheckpointDir(checkpointDir.getAbsolutePath)
@@ -854,77 +857,77 @@ class ALSCleanerSuite extends SparkFunSuite {
```
```
Before local dir :
/private/var/folders/9j/gf_c342d7d150mwrxvkqnc180000gn/T/spark-bd7219ae-e205-49d4-bebf-2970ae4bab4d
Actual local dir :
/private/var/folders/9j/gf_c342d7d150mwrxvkqnc180000gn/T/spark-bd7219ae-e205-49d4-bebf-2970ae4bab4d
Expected local dir root :
/private/var/folders/9j/gf_c342d7d150mwrxvkqnc180000gn/T/spark-167efe6b-ec4a-4fda-80f3-3dc970439d51
```
**After**
```diff
@@ -785,7 +785,11 @@ class ALSCleanerSuite extends SparkFunSuite {
def getAllFiles: Set[File] =
FileUtils.listFiles(localDir, TrueFileFilter.INSTANCE,
TrueFileFilter.INSTANCE).asScala.toSet
try {
+ println(s"Before local dir : ${Utils.getLocalDir(conf)}")
conf.set("spark.local.dir", localDir.getAbsolutePath)
+ Utils.clearLocalRootDirs()
+ println(s"Actual local dir : ${Utils.getLocalDir(conf)}")
+ println(s"Expected local dir root : ${localDir.getAbsolutePath}")
val sc = new SparkContext("local[2]", "test", conf)
try {
sc.setCheckpointDir(checkpointDir.getAbsolutePath)
@@ -854,77 +858,77 @@ class ALSCleanerSuite extends SparkFunSuite {
```
```
Before local dir :
/private/var/folders/9j/gf_c342d7d150mwrxvkqnc180000gn/T/spark-f5f2e39f-b4da-48aa-b6e9-f26b5b80c53e
Actual local dir :
/private/var/folders/9j/gf_c342d7d150mwrxvkqnc180000gn/T/spark-6e4ead48-2a0f-49b4-8cd7-4a81803aafee/spark-22dff601-c972-4a31-8357-cb595346487b
Expected local dir root :
/private/var/folders/9j/gf_c342d7d150mwrxvkqnc180000gn/T/spark-6e4ead48-2a0f-49b4-8cd7-4a81803aafee
```
I just double checked `getLocalDir` is called with initiation of
`SparkContext` as below:
<img width="273" alt="2017-07-05 8 30 28"
src="https://user-images.githubusercontent.com/6477701/27862873-6d434778-61c2-11e7-89b8-5d4c6a8506e2.png";>
<img width="478" alt="2017-07-05 8 30 21"
src="https://user-images.githubusercontent.com/6477701/27862876-6e221fe8-61c2-11e7-89d1-f566ccf9f7f3.png";>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
![https://user-images.githubusercontent.com/6477701/27862873-6d434778-61c2-11e7-89b8-5d4c6a8506e2.png"](https://user-images.githubusercontent.com/6477701/27862873-6d434778-61c2-11e7-89b8-5d4c6a8506e2.png"){kind=link}
![https://user-images.githubusercontent.com/6477701/27862876-6e221fe8-61c2-11e7-89d1-f566ccf9f7f3.png"](https://user-images.githubusercontent.com/6477701/27862876-6e221fe8-61c2-11e7-89d1-f566ccf9f7f3.png"){kind=link}