[ https://issues.apache.org/jira/browse/SPARK-19941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15923363#comment-15923363 ]
Karthik Palaniappan commented on SPARK-19941: --------------------------------------------- To repro: Set up Spark on YARN (Hadoop 2.8). Configure YARN to have a list of included and excluded nodes (yarn.resourcemanager.nodes.include-path and yarn.resourcemanager.nodes.exclude-path). Start with all nodes included, and node excluded. Run a spark job. I used: ``` spark-submit --class org.apache.spark.examples.SparkPi spark-examples.jar 100000 ``` While the job is running, add all nodes to the excluded file, and run `yarn rmadmin -refreshNodes -g 3600 -client`. Expected: Spark does not schedule any more tasks on executors, and they exit after being idle for 60s. The job hangs. Actual: Spark continues to schedule tasks and the job completes successfully. The nodes are only decommissioned when the job finishes. A less dramatic example is to just decommission a subset of the nodes and expect that tasks are not scheduled on executors on those hosts. > Spark should not schedule tasks on executors on decommissioning YARN nodes > -------------------------------------------------------------------------- > > Key: SPARK-19941 > URL: https://issues.apache.org/jira/browse/SPARK-19941 > Project: Spark > Issue Type: Bug > Components: Scheduler, YARN > Affects Versions: 2.2.0 > Environment: Hadoop 2.8.0-rc1 > Reporter: Karthik Palaniappan > > Hadoop 2.8 added a mechanism to gracefully decommission Node Managers in > YARN: https://issues.apache.org/jira/browse/YARN-914 > Essentially you can mark nodes to be decommissioned, and let them a) finish > work in progress and b) finish serving shuffle data. But no new work will be > scheduled on the node. > Spark should respect when NMs are set to decommissioned, and similarly > decommission executors on those nodes by not scheduling any more tasks on > them. > It looks like in the future YARN may inform the app master when containers > will be killed: https://issues.apache.org/jira/browse/YARN-3784. However, I > don't think Spark should schedule based on a timeout. We should gracefully > decommission the executor as fast as possible (which is the spirit of > YARN-914). The app master can query the RM for NM statuses (if it doesn't > already have them) and stop scheduling on executors on NMs that are > decommissioning. > Stretch feature: The timeout may be useful in determining whether running > further tasks on the executor is even helpful. Spark may be able to tell that > shuffle data will not be consumed by the time the node is decommissioned, so > it is not worth computing. The executor can be killed immediately. -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org