[ https://issues.apache.org/jira/browse/SPARK-29149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Graves reassigned SPARK-29149: ------------------------------------- Assignee: Thomas Graves > Update YARN cluster manager For Stage Level Scheduling > ------------------------------------------------------ > > Key: SPARK-29149 > URL: https://issues.apache.org/jira/browse/SPARK-29149 > Project: Spark > Issue Type: Story > Components: YARN > Affects Versions: 3.0.0 > Reporter: Thomas Graves > Assignee: Thomas Graves > Priority: Major > > For Stage Level Scheduling, we need to update the YARN allocator to handle > requesting executors for multiple ResourceProfiles. > * The container requests have to be updated to be based on a number of > containers per ResourceProfile. This is a larger change then you might > expect because on YARN you can’t ask for different container sizes within the > same YARN container priority. So we will have to ask for containers at > different priorities to be able to get different container sizes. Other YARN > applications like Tez handle this now so it shouldn’t be a big deal just a > matter of mapping stages to different priorities. > * The allocation response from YARN has to match the containers to a > resource profile. > * We need to launch the container with additional parameters so the executor > knows its resource profile to report back to the ExecutorMonitor. > -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org