bc Wong commented on YARN-1530:

bq. The current writing channel allows the data to be available on the timeline 
server immediately

Let's have reliability before speed. I think one of the requirement of ATS is: 
*The channel for writing events should be reliable.*

I'm using *reliable* here in a strong sense, not the TCP-best-effort style 
reliability. HDFS is reliable. Kafka is reliable. (They are also scalable and 
robust.) A normal RPC connection is not. I don't want the ATS to be able to 
slow down my writes, and therefore, my applications, at all. For example, an 
ATS failover shouldn't pause all my applications for N seconds. A direct RPC to 
the ATS for writing seems a poor choice in general.

Yes, you could make a distributed reliable scalable "ATS service" to accept 
writing events. But that seems a lot of work, while we can leverage existing 

If the channel itself is pluggable, then we have lots of options. Kafka is a 
very good choice, for sites that already deploy Kafka and know how to operate 
it. Using HDFS as a channel is also a good default implementation, for people 
already know how to scale and manage HDFS. Embedding a Kafka broker with each 
ATS daemon is also an option, if we're ok with that dependency.

> [Umbrella] Store, manage and serve per-framework application-timeline data
> --------------------------------------------------------------------------
>                 Key: YARN-1530
>                 URL: https://issues.apache.org/jira/browse/YARN-1530
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Vinod Kumar Vavilapalli
>         Attachments: ATS-Write-Pipeline-Design-Proposal.pdf, 
> ATS-meet-up-8-28-2014-notes.pdf, application timeline design-20140108.pdf, 
> application timeline design-20140116.pdf, application timeline 
> design-20140130.pdf, application timeline design-20140210.pdf
> This is a sibling JIRA for YARN-321.
> Today, each application/framework has to do store, and serve per-framework 
> data all by itself as YARN doesn't have a common solution. This JIRA attempts 
> to solve the storage, management and serving of per-framework data from 
> various applications, both running and finished. The aim is to change YARN to 
> collect and store data in a generic manner with plugin points for frameworks 
> to do their own thing w.r.t interpretation and serving.

This message was sent by Atlassian JIRA

Reply via email to