Junping Du commented on YARN-3949:

Thanks [~sjlee0] for working out the patch! I have some high level comments on 
API design here: 
Do we think flush() should be a generic behavior that TimelineCollectorManager 
should handle or it belongs to specific storage tiers that should be handled in 
HBase/Phoenix writer level? At least FSTimelineWriter don't need flush behavior 
so may be other storage implementation in future also doesn't need it. Instead, 
may be we could handle it inside of specific writer? Even for critical event 
that we want to flush it immediately after writing to backend storage, we can 
differentiate it through write API with sync or async (as parameter or name) 
rather than make caller explicitly to call flush(). The additional benefit 
could be: we can make writer to provide three different flush SLAs for 
different data being written:
1. Most critical entities, like lifecycle events of application, final 
counters, etc., writer do a flush immediately after write. 
2. Some important entities, like status update events of application, or 
increasing counters, etc., writer can do flush within a specific time (this 
patch work).
3. Less important entities, like: container metrics, etc. writer don't do flush 
but wait it reach to some buffer size to flush it automatically (original 
The time interval in above case 2 for trigger flush could be adjusted: if case 
1 write() happens, the timer should be reset; if only case 3 write() but no 
case 2 write(), the timing flush could be delayed.

> ensure timely flush of timeline writes
> --------------------------------------
>                 Key: YARN-3949
>                 URL: https://issues.apache.org/jira/browse/YARN-3949
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>    Affects Versions: YARN-2928
>            Reporter: Sangjin Lee
>            Assignee: Sangjin Lee
>         Attachments: YARN-3949-YARN-2928.001.patch, 
> YARN-3949-YARN-2928.002.patch, YARN-3949-YARN-2928.002.patch
> Currently flushing of timeline writes is not really handled. For example, 
> {{HBaseTimelineWriterImpl}} relies on HBase's {{BufferedMutator}} to batch 
> and write puts asynchronously. However, {{BufferedMutator}} may not flush 
> them to HBase unless the internal buffer fills up.
> We do need a flush functionality first to ensure that data are written in a 
> reasonably timely manner, and to be able to ensure some critical writes are 
> done synchronously (e.g. key lifecycle events).

This message was sent by Atlassian JIRA

Reply via email to