Jason Lowe commented on YARN-1354:

Thanks for taking a look, Junping!

bq. what would happen if storeApplication(), finishApplication(), 
removeApplication() failed with application related information get 
inconsistent after restart?

If storeApplication fails then it will throw an IOException which will bubble 
up and fail the container start request on the client.  As long as we're unable 
to store a new application, containers for that application will not start, 
which I believe is the desired behavior.  That prevents the state store from 
being inconsistent in this particular scenario.

If finishApplication fails then the NM will proceed as if it did succeed but 
the state store will still have the application present.  This should be 
corrected when the NM restarts and registers with the RM with those 
applications still running.  The RM should correct the situation by telling the 
NM that the application has finished (see YARN-1885), and the NM will proceed 
to perform application finish processing (e.g.: log aggregation, etc.).  I 
think worst-case it will upload all of the app container logs again, but when 
it goes to rename to the final destination name that will fail because the name 
already exists.  Thus there could be some wasted work, but it should sort 
itself out and not do something catastrophic.

If removeApplication fails then the NM will proceed as if it did succeed but 
the state store will still have the application present.  This should be 
corrected when the NM finishes application processing (per above or if it was 
already recorded as finished) and it will again try to remove it from the state 
store.  As above I think there could be some unnecessary work performed, but I 
think in the end the application should eventually be removed from the NM on 
restart.  It could still remain in the state store if the second removal also 
fails, but a subsequent restart should behave the same.

bq. Do we need special warning if get failed on deserializing credential here?

I'm not sure how credential processing is fundamentally all that different from 
protocol buffer parsing which could also fail.  If the credentials can't be 
read then we can't recover the application.  Currently recovery errors are 
fatal to NM startup.  Do you have something specific in mind for handling the 
credentials if the writable changes (e.g.: some pseudo code to show the 

> Recover applications upon nodemanager restart
> ---------------------------------------------
>                 Key: YARN-1354
>                 URL: https://issues.apache.org/jira/browse/YARN-1354
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager
>    Affects Versions: 2.3.0
>            Reporter: Jason Lowe
>            Assignee: Jason Lowe
>         Attachments: YARN-1354-v1.patch, 
> YARN-1354-v2-and-YARN-1987-and-YARN-1362.patch, YARN-1354-v3.patch, 
> YARN-1354-v4.patch, YARN-1354-v5.patch
> The set of active applications in the nodemanager context need to be 
> recovered for work-preserving nodemanager restart

This message was sent by Atlassian JIRA

Reply via email to