Github user rayburgemeestre commented on the issue:
https://github.com/apache/spark/pull/20451
Yes, absolutely, but I made those changes in a hardcoded way though, just
to try it out. So I'm not sure how helpful it is.
Near the end of this file:
`/usr/local/share/jupyter/kernels/apache_toree_pyspark/bin/run.sh` (which is
invoked by the pyspark kernel)
Added these options:
```
--master k8s://https://rb-spark:6443 \
--deploy-mode cluster \
--conf spark.executor.instances=3 \
--conf
spark.kubernetes.container.image=node001:5000/brightcomputing/spark-py:v2.3.0 \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
```
So the exec call looks like this:
```
eval exec \
"${SPARK_HOME}/bin/spark-submit" \
--name "'Apache Toree'" \
"${SPARK_OPTS}" \
--master k8s://https://rb-spark:6443 \
--conf spark.executor.instances=3 \
--conf
spark.kubernetes.container.image=node001:5000/brightcomputing/spark-py:v2.3.0 \
--conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
--class org.apache.toree.Main \
"${TOREE_ASSEMBLY}" \
"${TOREE_OPTS}" \
"$@"
```
A few notes:
- node001:5000 is my docker registry
- I merged some changes from master into your PR before manually building
the 2.4.0-SNAPSHOT jar file. Changes that make an additional "spark-py" image
available) The PR for that was: https://github.com/apache/spark/pull/21092
- The spark image in my shell script says "spark-py:v2.3.0" but it's
actually a newer version v2.4.0-SNAPSHOT. I just called the
docker_image_tool.sh script with the wrong version number in this case.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
