Thomas Graves created SPARK-33447:
-------------------------------------
Summary: Stage level scheduling, allow specifying other spark
configs via ResourceProfile
Key: SPARK-33447
URL: https://issues.apache.org/jira/browse/SPARK-33447
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 3.1.0
Reporter: Thomas Graves
With the addition of stage level scheduling and ResourceProfiles, we only allow
certain resources to specified currently. There are many other configs that
users may want to change between changes. We should look at adding in perhaps
SparkConf settings.
One very specific one that was brought up in review was ability to change yarn
queue (if that is even possible in yarn) between stages because someone might
want ot use one queue for ETL and a separate queue for ML where there are nodes
with GPU. Or perhaps node labels separately.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
