Thomas Graves created SPARK-33406:
-------------------------------------
Summary: k8s pyspark shell doesn't honor pyspark memory setting
Key: SPARK-33406
URL: https://issues.apache.org/jira/browse/SPARK-33406
Project: Spark
Issue Type: Bug
Components: Kubernetes
Affects Versions: 3.1.0
Reporter: Thomas Graves
When running the interactive pyspark shell in client mode on k8s, it doesn't
honor thespark.executor.pyspark.memory memory setting when requesting
containers from k8s.
If you run it in cluster mode and specify a python script then it does work.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
