[
https://issues.apache.org/jira/browse/SPARK-60914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-60914:
-----------------------------------
Labels: pull-request-available (was: )
> YarnAllocator never removes a recommissioned node from
> decommissioningNodesCache, so a second decommissioning of the same node is
> ignored
> -----------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-60914
> URL: https://issues.apache.org/jira/browse/SPARK-60914
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 3.4.0, 3.5.0, 4.1.0, 4.0.0
> Reporter: Norio Akagi
> Priority: Minor
> Labels: pull-request-available
>
> YarnAllocator keeps a \{{decommissioningNodesCache}} (added in SPARK-39018)
> so that the driver is notified only once per node that enters the
> DECOMMISSIONING state. Hosts are only ever added to the cache in
> \{{handleNodesInDecommissioningState}}. Nothing removes them except LRU
> eviction once the cache exceeds 200 entries.
> When a node is recommissioned, the RM reports it to the AM in
> \{{AllocateResponse.getUpdatedNodes}} with state RUNNING (YARN-10538, Hadoop
> 3.2.3 / 3.3.1 / 3.4.0+). YarnAllocator ignores every node state other than
> DECOMMISSIONING, so the host stays in the cache after it is back to RUNNING.
> If YARN later allocates new containers on that host and the host is
> decommissioned again, the DECOMMISSIONING report is filtered out by the stale
> cache entry. \{{DecommissionExecutorsOnHost}} is never sent to the driver,
> the new executors keep running tasks, and when the node is finally shut down
> their tasks and shuffle data are lost without the graceful decommission that
> \{{spark.decommission.enabled}} is supposed to provide.
> Sequence:
> # host1 is reported as DECOMMISSIONING. YarnAllocator sends
> DecommissionExecutorsOnHost(host1) and adds host1 to the cache.
> # host1 is recommissioned. The RM reports host1 as RUNNING, YarnAllocator
> ignores it. host1 remains in the cache.
> # YARN allocates new containers on host1 and executors start there.
> # host1 is reported as DECOMMISSIONING again. The report is dropped because
> host1 is already in the cache. The executors from step 3 are never
> decommissioned.
> This is easy to hit on clusters where the same hostname goes through
> decommission and recommission repeatedly, for example autoscaling setups that
> cancel a scale-in.
> Proposed fix: in \{{handleNodesInDecommissioningState}}, also look at RUNNING
> node reports and remove the host from \{{decommissioningNodesCache}}. A unit
> test that replays the sequence above against YarnAllocator fails on current
> master at step 4 ("Wanted 2 times, but was 1 time") and passes with the fix.
> References:
> * Original implementation: SPARK-39018,
> https://github.com/apache/spark/pull/35683
> * RM side notification on recommission: YARN-10538
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
