Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/19688#discussion_r149846521
--- Diff: bin/load-spark-env.cmd ---
@@ -24,10 +24,8 @@ rem conf/ subdirectory.
if [%SPARK_ENV_LOADED%] == [] (
set SPARK_ENV_LOADED=1
- if not [%SPARK_CONF_DIR%] == [] (
- set user_conf_dir=%SPARK_CONF_DIR%
- ) else (
- set user_conf_dir=..\conf
+ if [%SPARK_CONF_DIR%] == [] (
+ set SPARK_CONF_DIR=..\conf
--- End diff --
I think this should actually be like:
https://github.com/apache/spark/blob/2ba9b6a2dfff8eb06b6f93024f5140e784b8be49/bin/spark-class2.cmd#L21
assuming from:
https://github.com/apache/spark/blob/24e6c187fbaa6874eedbdda6b3b5dc6ff9e1de36/conf/spark-env.sh.template#L59
So, to be clear:
```
set SPARK_CONF_DIR=%~dp0..\conf
```
otherwise, it looks ending up:
```cmd
C:\Users\IEUser\workspace\spark>.\bin\spark-shell
```
```scala
scala> sys.env.get("SPARK_CONF_DIR")
res0: Option[String] = Some(..\conf)
scala> import java.nio.file.{Paths, Files}
import java.nio.file.{Paths, Files}
scala> Files.exists(Paths.get(sys.env.get("SPARK_CONF_DIR").get))
res2: Boolean = false
```
After the change:
```scala
scala> sys.env.get("SPARK_CONF_DIR")
res0: Option[String] = Some(C:\Users\IEUser\workspace\spark\bin\..\conf)
scala> import java.nio.file.{Paths, Files}
import java.nio.file.{Paths, Files}
scala> Files.exists(Paths.get(sys.env.get("SPARK_CONF_DIR").get))
res2: Boolean = true
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
