Tao Yang created YARN-9413:
------------------------------
Summary: Queue resource leak after app fail for CapacityScheduler
Key: YARN-9413
URL: https://issues.apache.org/jira/browse/YARN-9413
Project: Hadoop YARN
Issue Type: Bug
Components: capacityscheduler
Affects Versions: 3.1.2
Reporter: Tao Yang
Assignee: Tao Yang
To reproduce this problem:
# Submit an app which is configured to keep containers across app attempts and
should fail after AM finished at first time (am-max-attempts=1).
# App is started with 2 containers running on NM1 node.
# Fail the AM of the application with PREEMPTED exit status which should not
count towards max attempt retry but app will fail immediately.
# Used resource of this queue leaks after app fail.
The root cause is the inconsistency of handling app attempt failure between
RMAppAttemptImpl$BaseFinalTransition#transition and
RMAppImpl$AttemptFailedTransition#transition:
# After app fail, RMAppFailedAttemptEvent will be sent in
RMAppAttemptImpl$BaseFinalTransition#transition, if exit status of AM container
is PREEMPTED/ABORTED/DISKS_FAILED/KILLED_BY_RESOURCEMANAGER, it will not count
towards max attempt retry, so that it will send AppAttemptRemovedSchedulerEvent
with keepContainersAcrossAppAttempts=true and RMAppFailedAttemptEvent with
transferStateFromPreviousAttempt=true.
# RMAppImpl$AttemptFailedTransition#transition handle RMAppFailedAttemptEvent
and will fail the app if its max app attempts is 1.
# CapacityScheduler handles AppAttemptRemovedSchedulerEvent in
CapcityScheduler#doneApplicationAttempt, it will skip killing and calling
completion process for containers belong to this app, so that queue resource
leak happens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
