[ 
https://issues.apache.org/jira/browse/YARN-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14711671#comment-14711671
 ] 

Junping Du commented on YARN-4061:
----------------------------------

Thanks [~gtCarrera9] for working on a document for this. I have some high level 
comments on current design before moving to the details:
1. We should be very careful to use HDFS to cache incremental updates, i.e. 
incoming timeline entities. HDFS itself is not optimized for random writing 
performance especially with large scale of writers (assume each NM has a 
TimelineWriter). 
2. Implementing a redo log based on HDFS is very complicated, and this should 
achieve the similar goal as WAL (Write Ahead Log) in HBase. Isn't it? If so, do 
we plan to borrow code/components from HBase on this?
3. I think making HDFS serve as backup storage make more sense.

> [Fault tolerance] Fault tolerant writer for timeline v2
> -------------------------------------------------------
>
>                 Key: YARN-4061
>                 URL: https://issues.apache.org/jira/browse/YARN-4061
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Li Lu
>            Assignee: Li Lu
>         Attachments: FaulttolerantwriterforTimelinev2.pdf
>
>
> We need to build a timeline writer that can be resistant to backend storage 
> down time and timeline collector failures. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Reply via email to