Varun Saxena commented on YARN-2962:

[~vinodkv] / [~asuresh], supporting 2 separate hierarchies will increase 
complexity. Let us consider option 1 i.e. having {{RM_APP_ROOT/hierarchies}}. 
Here, we also need to consider the case where split index can be changed from 
say 2 to 3. To handle this case we can have multiple folders under hierarchies 
to indicate split index. But, this would mean that for an app we may have to 
look under upto 5 locations till we succeed. 
Option 2 can also be done. Here we can check whether data exists under a znode 
or not to determine whether we found the app or not. Here also we may have to 
look up multiple times before finding an app though.

We can also do as under :
1. As Vinod suggested, write a tool or utility like "yarn resourcemanager 
-format-state-store" to migrate apps from the current scheme to the newly 
configured scheme. Can also allow giving the app index from command line.  Not 
sure though how much time migrating 10000 apps(default value of max number of 
apps in store) in state store will take.
2. Current code will continue as it is. We can abort running of RM if we find 
mismatch in the scheme used for storing of apps. We can then warn the admin to 
run the tool above before he tries to restart RM.

> ZKRMStateStore: Limit the number of znodes under a znode
> --------------------------------------------------------
>                 Key: YARN-2962
>                 URL: https://issues.apache.org/jira/browse/YARN-2962
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: resourcemanager
>    Affects Versions: 2.6.0
>            Reporter: Karthik Kambatla
>            Assignee: Varun Saxena
>            Priority: Critical
>         Attachments: YARN-2962.01.patch, YARN-2962.2.patch, YARN-2962.3.patch
> We ran into this issue where we were hitting the default ZK server message 
> size configs, primarily because the message had too many znodes even though 
> they individually they were all small.

This message was sent by Atlassian JIRA

Reply via email to