Zhijie Shen commented on YARN-2033:

[~djp], thanks for raising this question explicitly. Here're two points I'd 
like to highlight for this work:

1. This patch doesn't intend to remove the existing FS based history store, but 
deprecate it by removing the default configs about loading FS based history 
store. On the other hand, the patch adds the history store that rides the 
timeline store, and use it as the default. Given the user who is the early 
adopter of the generic history service wants to continue with FS based history 
store, he needs to set the old configs explicitly (actually he should have done 
it because by default the generic history service is not enabled), and the new 
generic history service is still going to horner old configs for backward 

2. Though the generic history service (previously we call it application 
history server) is introduced to Hadoop since 2.4, but it is not production 
ready. We have explicitly highlighted it in the 
 I agree it seems to be a bit aggressive to move from FS based history store to 
timeline store based one as the default, however, I'm afraid it's the best 
choice at the current stage, because FS based history store has several 
critical limitations: no caching, no retention, not scalable and not supporting 
the secure mode. Unless we're able to solve all these problems (obviously we 
don't have the bandwidth to do it now), it's risky to use FS based history 
store as the default, in particular when the timeline server is going to be 
production ready. On the other side, the aforementioned limitations have 
already been addressed by the timeline store (scalability will be ensured by 
HBase timeline store). Hence timeline store based history store should be a 
more reasonable and reliable default of new users.

> Investigate merging generic-history into the Timeline Store
> -----------------------------------------------------------
>                 Key: YARN-2033
>                 URL: https://issues.apache.org/jira/browse/YARN-2033
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Vinod Kumar Vavilapalli
>            Assignee: Zhijie Shen
>         Attachments: ProposalofStoringYARNMetricsintotheTimelineStore.pdf, 
> YARN-2033.1.patch, YARN-2033.2.patch, YARN-2033.3.patch, YARN-2033.4.patch, 
> YARN-2033.5.patch, YARN-2033.Prototype.patch, YARN-2033_ALL.1.patch, 
> YARN-2033_ALL.2.patch, YARN-2033_ALL.3.patch, YARN-2033_ALL.4.patch
> Having two different stores isn't amicable to generic insights on what's 
> happening with applications. This is to investigate porting generic-history 
> into the Timeline Store.
> One goal is to try and retain most of the client side interfaces as close to 
> what we have today.

This message was sent by Atlassian JIRA

Reply via email to