Shooter23 commented on a change in pull request #35043:
URL: https://github.com/apache/spark/pull/35043#discussion_r776118212
##########
File path: bin/find-spark-home
##########
@@ -37,5 +37,5 @@ else
if [[ -z "$PYSPARK_DRIVER_PYTHON" ]]; then
PYSPARK_DRIVER_PYTHON="${PYSPARK_PYTHON:-"python3"}"
fi
- export SPARK_HOME=$($PYSPARK_DRIVER_PYTHON "$FIND_SPARK_HOME_PYTHON_SCRIPT")
+ export SPARK_HOME=$($PYSPARK_DRIVER_PYTHON "$FIND_SPARK_HOME_PYTHON_SCRIPT"
| cut -d $'\007' -f 2)
Review comment:
It appears to be the same problem but with a different solution.
I hadn't tried it with jupyter before so I don't know what I should expect.
Running it with `PYSPARK_DRIVER_PYTHON=jupyter` made it attempt to run
`jupyter-/path/to/find_spark_home.py`, which failed with it failing to find
/bin/load-spark-env.sh or /bin/spark-submit.
Running it with `PYSPARK_DRIVER_PYTHON='jupyter notebook'` started the
kernel but initially opened a 403 forbidden page in the browser, complaining of
lacking a referrer. I pasted the URL with the token appearing in the jupyter
log and it took me to the bin directory under the virtual environment.
It doesn't appear to me that `$PYSPARK_DRIVER_PYTHON
"$FIND_SPARK_HOME_PYTHON_SCRIPT"` should run with PYSPARK_DRIVER_PYTHON set to
"jupyter".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
