tgravescs commented on a change in pull request #28085:
URL: https://github.com/apache/spark/pull/28085#discussion_r412983650
##########
File path: core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala
##########
@@ -1135,6 +1136,27 @@ private[spark] class DAGScheduler(
}
}
+ /**
+ * `PythonRunner` needs to know what the pyspark memory and cores settings
are for the profile
+ * being run. Pass them in the local properties of the task if it's set for
the stage profile.
+ */
+ private def addPysparkConfigsToProperties(stage: Stage, properties:
Properties): Unit = {
Review comment:
I assume you mean addPySparkConfigsToProperties with capital on Spark
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
