Ming Ma created MAPREDUCE-6694:
----------------------------------
Summary: Make AM more resilient to potential lost of any completed
container notification
Key: MAPREDUCE-6694
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6694
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Ming Ma
YARN tries to guarantee any completed container notification is delivered to AM
under any circumstance, YARN-1372 is an example to make sure for the case of RM
restart. However, under some corner cases, it is still possible a completed
container notifications is lost or significantly delayed. For example, if NM
host becomes dead when RM fails over.
AM won't preempt reducers if it thought there is at least one mapper running.
{noformat}
void preemptReducesIfNeeded() {
...
if (assignedRequests.maps.size() > 0) {
// there are assigned mappers
return;
}
...
{noformat}
Instead of completely depending on notification from RM, it can use
TaskUmbilicalProtocol to help to decide if there is any mapper running. That
will make AM more resilient to any bugs in YARN.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
