Anubhav Dhoot commented on YARN-2229:

We cannot simply add a field and have old code not know about it. That will 
cause it to silently work with a wrong id (missing field). And because of the 
way we construct containerIds we need to add the new field (details in 

The only way i see it working (without a cluster shutdown) is if we support 
deserializing both the older format and newer format. When serializing we can 
choose to emit a new field based on a condition (flag or version number of the 
So the first rolling upgrade will not turn on the condition but will ensure all 
the code supports deserializing the newer field if it exists. In the next 
rolling upgrade we can turn on the condition to serialize the new field.

RM can ensure that  NMs are upgraded to a specific version (support 
deserializing the new field) before allowing the flag to be turned on. That 
will take care of the case when someone does not follow the approach above.
Any problems with this approach?

> ContainerId can overflow with RM restart
> ----------------------------------------
>                 Key: YARN-2229
>                 URL: https://issues.apache.org/jira/browse/YARN-2229
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: resourcemanager
>            Reporter: Tsuyoshi OZAWA
>            Assignee: Tsuyoshi OZAWA
>         Attachments: YARN-2229.1.patch, YARN-2229.10.patch, 
> YARN-2229.10.patch, YARN-2229.2.patch, YARN-2229.2.patch, YARN-2229.3.patch, 
> YARN-2229.4.patch, YARN-2229.5.patch, YARN-2229.6.patch, YARN-2229.7.patch, 
> YARN-2229.8.patch, YARN-2229.9.patch
> On YARN-2052, we changed containerId format: upper 10 bits are for epoch, 
> lower 22 bits are for sequence number of Ids. This is for preserving 
> semantics of {{ContainerId#getId()}}, {{ContainerId#toString()}}, 
> {{ContainerId#compareTo()}}, {{ContainerId#equals}}, and 
> {{ConverterUtils#toContainerId}}. One concern is epoch can overflow after RM 
> restarts 1024 times.
> To avoid the problem, its better to make containerId long. We need to define 
> the new format of container Id with preserving backward compatibility on this 

This message was sent by Atlassian JIRA

Reply via email to