Github user lucashu1 commented on the issue: https://github.com/apache/spark/pull/21092 Sorry in advance if this is the wrong place to be asking this! Does this PR mean that we'll be able to create SparkContexts using PySpark's [`SparkSession.Builder`](https://spark.apache.org/docs/preview/api/python/pyspark.sql.html#pyspark.sql.SparkSession.Builder) with `master` set to `k8s://<...>:<...>`, and have the resulting jobs run on spark-on-k8s, instead of on local/standalone? E.g.: ``` from pyspark.sql import SparkSession spark = SparkSession.builder.master('k8s://https://kubernetes:443').getOrCreate() ``` I'm trying to use PySpark in a Jupyter notebook that's running inside a Kubernetes pod, and have it use spark-on-k8s instead of resorting to using `local[*]` as `master`. Till now, I've been getting an error saying that: > Error: Python applications are currently not supported for Kubernetes. whenever I try to use `k8s://<...>` as `master`. Thanks!
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org