Github user HyukjinKwon commented on the issue:
https://github.com/apache/spark/pull/19370
I suggest this:
```cmd
rem Path to Python script finding SPARK_HOME
set FIND_SPARK_HOME_PYTHON_SCRIPT=%~dp0find_spark_home.py
rem Default to standard python interpreter unless told otherwise
set PYTHON_RUNNER=python
rem If PYSPARK_DRIVER_PYTHON is set, it overwrites the python version
if not "x%PYSPARK_DRIVER_PYTHON%" =="x" (
set PYTHON_RUNNER=%PYSPARK_DRIVER_PYTHON%
)
rem If PYSPARK_PYTHON is set, it overwrites the python version
if not "x%PYSPARK_PYTHON%" =="x" (
set PYTHON_RUNNER=%PYSPARK_PYTHON%
)
rem If there is python installed, trying to use the root dir as SPARK_HOME
where %PYTHON_RUNNER% > nul 2>$1
if %ERRORLEVEL% neq 0 (
if not exist %PYTHON_RUNNER% (
echo Missing Python executable '%PYTHON_RUNNER%', defaulting to
'%~dp0..' for SPARK_HOME. ^
Please install Python or specify the correct Python executable for
PYSPARK_DRIVER_PYTHON ^
or PYSPARK_PYTHON.
if "x%SPARK_HOME%"=="x" (
set SPARK_HOME=%~dp0..
)
)
)
rem Only attempt to find SPARK_HOME if it is not set.
if "x%SPARK_HOME%"=="x" (
rem We are pip installed, use the Python script to resolve a reasonable
SPARK_HOME
if not exist "%FIND_SPARK_HOME_PYTHON_SCRIPT%" (
rem If we are not in the same directory as find_spark_home.py we are
not pip installed so we don't
rem need to search the different Python directories for a Spark
installation.
rem Note only that, if the user has pip installed PySpark but is
directly calling pyspark-shell or
rem spark-submit in another directory we want to use that version of
PySpark rather than the
rem pip installed version of PySpark.
set SPARK_HOME=%~dp0..
) else (
rem If there is no python installed it will fail with message:
rem 'python' is not recognized as an internal or external command,
for /f "delims=" %%i in ('%PYTHON_RUNNER%
%FIND_SPARK_HOME_PYTHON_SCRIPT%') do set SPARK_HOME=%%i
)
)
```
I manually tested each branch. This address the concern in
https://github.com/apache/spark/pull/19370#discussion_r152449365. The error
message shows like:
```cmd
C:\...>pyspark
```
```
Missing Python executable 'C:\foo\bar.exe', defaulting to
'C:\Python27\Scripts\.
.' for SPARK_HOME. Please install Python or specify the correct Python
executabl
e for PYSPARK_DRIVER_PYTHON or PYSPARK_PYTHON.
```
```cmd
C:\...>pyspark
```
```
Missing Python executable 'bar', defaulting to 'C:\Python27\Scripts\..' for
SPAR
K_HOME. Please install Python or specify the correct Python executable for
PYSPA
RK_DRIVER_PYTHON or PYSPARK_PYTHON.
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
