Zhijie Shen commented on YARN-2837:

bq.  It’ll throw inCompatible exception, even though it should not.

I guess it's way we want to do. When currently CURRENT_VERSION_INFO = 1.0, we 
make it compatible with the existing store without version info. When 
CURRENT_VERSION_INFO is upgraded to 2.0, the db schema is no longer compatible 
to 1.0 and the exception is thrown. Going directly from no version to 2.0 may 
not be a valid use case.

Here I simply reuse the same logic that other leveldb impl is using. If you 
have some concern, we can figure out it separately.

bq. RMDelegationTokenIdentifierData

I noticed it before, but I don't think it is really necessary to create another 
protobuf obj just because we have one additional long integer to ser/des.

bq. it’s always adding new token, the old token still remain, we should remove 
the old token

{{db.put(k, v);}} will update the value if key already exists.

bq. the following looks ok , a simpler way might be to just concatenate two 

That's different. See KeyBuilder  and KeyParser for more detail. It's need to 
be broken into two section for the convenience of parsing it.

bq. Or we could force user to configure the path properly and throw exception 

I prefer to keep state store and data store under the same dir by default.  For 
advanced deployment, the user is free to config them separately.

bq. we need to keep track of the latest sequenceNumber so that it can be 

It is updated here.
    public void recover(TimelineServiceState state) throws IOException {
      LOG.info("Recovering " + getClass().getSimpleName());
      for (DelegationKey key : state.getTokenMasterKeyState()) {
      for (Entry<TimelineDelegationTokenIdentifier, Long> entry :
          state.getTokenState().entrySet()) {
        addPersistedDelegationToken(entry.getKey(), entry.getValue());

Otherwise, I updated the patch accordingly. BTW, I've removed the cache config, 
as it is not so important to the state store.

> Timeline server needs to recover the timeline DT when restarting
> ----------------------------------------------------------------
>                 Key: YARN-2837
>                 URL: https://issues.apache.org/jira/browse/YARN-2837
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: timelineserver
>            Reporter: Zhijie Shen
>            Assignee: Zhijie Shen
>            Priority: Blocker
>             Fix For: 2.7.0
>         Attachments: YARN-2837.1.patch, YARN-2837.2.patch, YARN-2837.3.patch
> Timeline server needs to recover the stateful information when restarting as 
> RM/NM/JHS does now. So far the stateful information only includes the 
> timeline DT. Without recovery, the timeline DT of the existing YARN apps is 
> not long valid, and cannot be renewed any more after the timeline server is 
> restarted.

This message was sent by Atlassian JIRA

Reply via email to