Github user rayburgemeestre commented on the issue: https://github.com/apache/spark/pull/20451 Yes, absolutely, but I made those changes in a hardcoded way though, just to try it out. So I'm not sure how helpful it is. Near the end of this file: `/usr/local/share/jupyter/kernels/apache_toree_pyspark/bin/run.sh` (which is invoked by the pyspark kernel) Added these options: ``` --master k8s://https://rb-spark:6443 \ --deploy-mode cluster \ --conf spark.executor.instances=3 \ --conf spark.kubernetes.container.image=node001:5000/brightcomputing/spark-py:v2.3.0 \ --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \ ``` So the exec call looks like this: ``` eval exec \ "${SPARK_HOME}/bin/spark-submit" \ --name "'Apache Toree'" \ "${SPARK_OPTS}" \ --master k8s://https://rb-spark:6443 \ --conf spark.executor.instances=3 \ --conf spark.kubernetes.container.image=node001:5000/brightcomputing/spark-py:v2.3.0 \ --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \ --class org.apache.toree.Main \ "${TOREE_ASSEMBLY}" \ "${TOREE_OPTS}" \ "$@" ``` A few notes: - node001:5000 is my docker registry - I merged some changes from master into your PR before manually building the 2.4.0-SNAPSHOT jar file. Changes that make an additional "spark-py" image available) The PR for that was: https://github.com/apache/spark/pull/21092 - The spark image in my shell script says "spark-py:v2.3.0" but it's actually a newer version v2.4.0-SNAPSHOT. I just called the docker_image_tool.sh script with the wrong version number in this case.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org