Tao Yang created YARN-9413: ------------------------------ Summary: Queue resource leak after app fail for CapacityScheduler Key: YARN-9413 URL: https://issues.apache.org/jira/browse/YARN-9413 Project: Hadoop YARN Issue Type: Bug Components: capacityscheduler Affects Versions: 3.1.2 Reporter: Tao Yang Assignee: Tao Yang
To reproduce this problem: # Submit an app which is configured to keep containers across app attempts and should fail after AM finished at first time (am-max-attempts=1). # App is started with 2 containers running on NM1 node. # Fail the AM of the application with PREEMPTED exit status which should not count towards max attempt retry but app will fail immediately. # Used resource of this queue leaks after app fail. The root cause is the inconsistency of handling app attempt failure between RMAppAttemptImpl$BaseFinalTransition#transition and RMAppImpl$AttemptFailedTransition#transition: # After app fail, RMAppFailedAttemptEvent will be sent in RMAppAttemptImpl$BaseFinalTransition#transition, if exit status of AM container is PREEMPTED/ABORTED/DISKS_FAILED/KILLED_BY_RESOURCEMANAGER, it will not count towards max attempt retry, so that it will send AppAttemptRemovedSchedulerEvent with keepContainersAcrossAppAttempts=true and RMAppFailedAttemptEvent with transferStateFromPreviousAttempt=true. # RMAppImpl$AttemptFailedTransition#transition handle RMAppFailedAttemptEvent and will fail the app if its max app attempts is 1. # CapacityScheduler handles AppAttemptRemovedSchedulerEvent in CapcityScheduler#doneApplicationAttempt, it will skip killing and calling completion process for containers belong to this app, so that queue resource leak happens. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org