Prabhu Joseph created YARN-11466:
------------------------------------
Summary: Graceful Decommission for Shuffle Services
Key: YARN-11466
URL: https://issues.apache.org/jira/browse/YARN-11466
Project: Hadoop YARN
Issue Type: New Feature
Reporter: Prabhu Joseph
Assignee: Prabhu Joseph
Currently, YARN Graceful Decommission waits for the completion of both running
containers and the running applications of those containers launched on the
node under decommission. This adds unnecessary cost to users on cloud
deployments. This feature aims to improve the Graceful Decommission logic by
waiting for the actual shuffle data to be consumed by dependent tasks rather
than the entire application.
Below is the high-level design I have in mind.
Add a new interface (say AuxiliaryShuffleService extends AuxiliaryService)
through which the workloads (Spark, Tez, MapReduce) ShuffleHandler exposes
shuffle data metrics (like shuffle data being present or not). NodeManager
periodically collects the shuffle data metrics from the configured
AuxiliaryShuffleServices and sends them along with the heartbeat to the
ResourceManager. The graceful decommission logic runs inside ResourceManager
waits until the shuffle data is consumed, with a maximum wait time up to the
configured graceful decommission timeout.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
