Sangjin Lee commented on YARN-4061:

I don't think the MR JHS is an apt comparison. First, we're dealing with a 
totally distributed writer situation (individual jobs) for the MR JHS whereas 
the RM timeline collector would be a single significant writer (again, it's the 
RM collector that I'm most worried about). Also, JHS writes only a few large 
files (job conf, job history files, etc.), whereas the timeline service will 
write a huge number of tiny writes. The volume of writes will be much larger 
than the JHS use case.

Regarding the synchronous semantics, we really need to think it through. On the 
one hand, we might consider handling the synchronous calls separate from the 
rest and outside the log queue, but it's not clear how one can make it work 
alongside the asynchronous writes that are going on.

> [Fault tolerance] Fault tolerant writer for timeline v2
> -------------------------------------------------------
>                 Key: YARN-4061
>                 URL: https://issues.apache.org/jira/browse/YARN-4061
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Li Lu
>            Assignee: Li Lu
>         Attachments: FaulttolerantwriterforTimelinev2.pdf
> We need to build a timeline writer that can be resistant to backend storage 
> down time and timeline collector failures. 

This message was sent by Atlassian JIRA

Reply via email to