Zhijie Shen commented on YARN-3949:

IMHO, given write + flush, it's not necessary to have sync write and async 
write api at the writer level, while we already have the analogy at the app 
collector level. App collector level knows better than writer to decide if it 
should flush after one write, two writes or more.

The current approach seems to be good now, I propose to go with it, and unblock 
viewing the app timeline data after it gets finished. Thoughts? From my point 
of view, async write is more than the flush (e.g, queueing the entities in the 
collector, combining the updates of the same entity and etc.)

For the patch details:

1. "writer.flush.interval.seconds" \->  "writer.flush-interval-seconds". YARN 
convention is to use "." to separate namespaces (sub components) and "-" to 
concat words. Please move the default to YarnConfiguration as well, which is 
part of API, and ad this config to yarn-default.xml.

2. Shall we use shutdown and then waitTermination to gracefully stop the 
service? Otherwise, if there's a scheduled flush task that is running while the 
manager invokes writer.close(), will it cause any problem? Or is it just thread 
safe, such that we don't need to worry about it?

> ensure timely flush of timeline writes
> --------------------------------------
>                 Key: YARN-3949
>                 URL: https://issues.apache.org/jira/browse/YARN-3949
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>    Affects Versions: YARN-2928
>            Reporter: Sangjin Lee
>            Assignee: Sangjin Lee
>         Attachments: YARN-3949-YARN-2928.001.patch, 
> YARN-3949-YARN-2928.002.patch, YARN-3949-YARN-2928.002.patch
> Currently flushing of timeline writes is not really handled. For example, 
> {{HBaseTimelineWriterImpl}} relies on HBase's {{BufferedMutator}} to batch 
> and write puts asynchronously. However, {{BufferedMutator}} may not flush 
> them to HBase unless the internal buffer fills up.
> We do need a flush functionality first to ensure that data are written in a 
> reasonably timely manner, and to be able to ensure some critical writes are 
> done synchronously (e.g. key lifecycle events).

This message was sent by Atlassian JIRA

Reply via email to