[GitHub] spark pull request #19688: [SPARK-22466][Spark Submit]export SPARK_CONF_DIR ...

[HyukjinKwon]

Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/19688#discussion_r149846521
  
    --- Diff: bin/load-spark-env.cmd ---
    @@ -24,10 +24,8 @@ rem conf/ subdirectory.
     if [%SPARK_ENV_LOADED%] == [] (
       set SPARK_ENV_LOADED=1
     
    -  if not [%SPARK_CONF_DIR%] == [] (
    -    set user_conf_dir=%SPARK_CONF_DIR%
    -  ) else (
    -    set user_conf_dir=..\conf
    +  if [%SPARK_CONF_DIR%] == [] (
    +    set SPARK_CONF_DIR=..\conf
    --- End diff --
    
    I think this should actually be like:
    
    
https://github.com/apache/spark/blob/2ba9b6a2dfff8eb06b6f93024f5140e784b8be49/bin/spark-class2.cmd#L21
    
    assuming from:
    
    
https://github.com/apache/spark/blob/24e6c187fbaa6874eedbdda6b3b5dc6ff9e1de36/conf/spark-env.sh.template#L59
    
    So, to be clear:
    
    ```
    set SPARK_CONF_DIR=%~dp0..\conf
    ```
    
    otherwise, it looks ending up:
    
    ```cmd
    C:\Users\IEUser\workspace\spark>.\bin\spark-shell
    ```
    
    ```scala
    scala> sys.env.get("SPARK_CONF_DIR")
    res0: Option[String] = Some(..\conf)
    
    scala> import java.nio.file.{Paths, Files}
    import java.nio.file.{Paths, Files}
    
    scala> Files.exists(Paths.get(sys.env.get("SPARK_CONF_DIR").get))
    res2: Boolean = false
    ```
    
    After the change:
    
    ```scala
    scala> sys.env.get("SPARK_CONF_DIR")
    res0: Option[String] = Some(C:\Users\IEUser\workspace\spark\bin\..\conf)
    
    scala> import java.nio.file.{Paths, Files}
    import java.nio.file.{Paths, Files}
    
    scala> Files.exists(Paths.get(sys.env.get("SPARK_CONF_DIR").get))
    res2: Boolean = true
    ```


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org