Thomas Graves created SPARK-31444:
-------------------------------------
Summary: Pyspark memory and cores calculation doesn't account for
task cpus
Key: SPARK-31444
URL: https://issues.apache.org/jira/browse/SPARK-31444
Project: Spark
Issue Type: Bug
Components: PySpark
Affects Versions: 2.4.5
Reporter: Thomas Graves
during changes for stage level scheduling, I discovered a possible issue that
the calculation for splitting pyspark memory up doesn't take into account the
spark.task.cpus setting.
Discussion here:
[https://github.com/apache/spark/pull/28085#discussion_r407573038]
See PythonRunner.scala:
[https://github.com/apache/spark/blob/6b88d136deb99afd9363b208fd6fe5684fe8c3b8/core/src/main/scala/org/apache/spark/api/python/PythonRunner.scala#L90]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
