Robert Kanter commented on YARN-4408:

I haven't been able to reproduce this issue, and I agree that it's not a common 
occurrence; but we have seen the number of running containers go negative 
internally on two different clusters and also on a customer's cluster.  So I 
started through the code and state machine for how we could decrement the gauge 
without first incrementing it.  As far as I can tell, this is the only way 
where this can happen because we don't check {{container.wasLaunched}} like in 
the other two places where we decrement the gauge.  

> NodeManager still reports negative running containers
> -----------------------------------------------------
>                 Key: YARN-4408
>                 URL: https://issues.apache.org/jira/browse/YARN-4408
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>    Affects Versions: 2.4.0
>            Reporter: Robert Kanter
>            Assignee: Robert Kanter
>         Attachments: YARN-4408.001.patch
> YARN-1697 fixed a problem where the NodeManager metrics could report a 
> negative number of running containers.  However, it missed a rare case where 
> this can still happen.
> YARN-1697 added a flag to indicate if the container was actually launched 
> ({{LOCALIZED}} to {{RUNNING}}) or not ({{LOCALIZED}} to {{KILLING}}), which 
> is then checked when transitioning from {{CONTAINER_CLEANEDUP_AFTER_KILL}} to 
> {{DONE}} and {{EXITED_WITH_FAILURE}} to {{DONE}} to only decrement the gauge 
> if we actually ran the container and incremented the gauge .  However, this 
> flag is not checked while transitioning from {{EXITED_WITH_SUCCESS}} to 
> {{DONE}}.

This message was sent by Atlassian JIRA

Reply via email to