Zhijie Shen commented on YARN-3047:

I've a general question about reading from collector to get the timeline data 
of a running app. I'm not sure why we cannot read the timeline data from the 
backend for a running app. While the app is running, it will send the timeline 
data to its corresponding collector, and collector won't hold this data until 
the app is finished to start persisting it in the backend, right? The problem I 
can think of is that the collector process the timeline data in async manner, 
and the app is sending the timeline data intensively, such that there's still 
some timeline data which is already sent to the collector, but not persisted in 
the backend. In this case, to completely answer a user's query, the reader 
needs to combine the persisted data in the backend and the pending data in 
collector. That said, for running app, we can and should still read from the 
backend. Thoughts?

Moreover, as the timeline data keeps flowing into the backend when the app is 
running, we can get the final consistent results from the backend only, if we 
don't need to support real-time use case immediately. IMHO, in terms of the 
steps to move the reader work forward, we can first focus on querying the data 
from the backend, and then from collector as well.

> [Data Serving] Set up ATS reader with basic request serving structure and 
> lifecycle
> -----------------------------------------------------------------------------------
>                 Key: YARN-3047
>                 URL: https://issues.apache.org/jira/browse/YARN-3047
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Sangjin Lee
>            Assignee: Varun Saxena
>         Attachments: Timeline_Reader(draft).pdf, YARN-3047.001.patch, 
> YARN-3047.003.patch, YARN-3047.02.patch, YARN-3047.04.patch
> Per design in YARN-2938, set up the ATS reader as a service and implement the 
> basic structure as a service. It includes lifecycle management, request 
> serving, and so on.

This message was sent by Atlassian JIRA

Reply via email to