Jason Lowe commented on YARN-3496:

How are we going to support recovery of containers without tracking which 
resources we have localized?  We cannot re-localize resources for already 
running containers -- they're already running on the existing localized 
resources.  If we fail to track those resources then we leak them on the local 
filesystem, potentially colliding with them later.  So from my perspective 
there already is a way to enable/disable this behavior, and that's 
enabling/disabling NM recovery.  I don't see how we can support recovery 
without storing localization state.

Do we have hard evidence storing to leveldb is a huge NM performance impact, or 
is this theoretical?  In our testing we have not seen enabling NM recovery as 
having a significant impact on NM performance.

> Add a configuration to disable/enable storing localization state in 
> NMLeveldbStateStore
> ---------------------------------------------------------------------------------------
>                 Key: YARN-3496
>                 URL: https://issues.apache.org/jira/browse/YARN-3496
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: nodemanager
>    Affects Versions: 2.7.0
>            Reporter: zhihai xu
>            Assignee: zhihai xu
> Add a configuration to disable/enable storing localization state in 
> NMLeveldbStateStore.
> Store Localization state in the levelDB may have some overhead, which may 
> affect NM performance.
> It would better to have a configuration to disable/enable it.

This message was sent by Atlassian JIRA

Reply via email to