[
https://issues.apache.org/jira/browse/YARN-3047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14381103#comment-14381103
]
Zhijie Shen commented on YARN-3047:
-----------------------------------
I've a general question about reading from collector to get the timeline data
of a running app. I'm not sure why we cannot read the timeline data from the
backend for a running app. While the app is running, it will send the timeline
data to its corresponding collector, and collector won't hold this data until
the app is finished to start persisting it in the backend, right? The problem I
can think of is that the collector process the timeline data in async manner,
and the app is sending the timeline data intensively, such that there's still
some timeline data which is already sent to the collector, but not persisted in
the backend. In this case, to completely answer a user's query, the reader
needs to combine the persisted data in the backend and the pending data in
collector. That said, for running app, we can and should still read from the
backend. Thoughts?
Moreover, as the timeline data keeps flowing into the backend when the app is
running, we can get the final consistent results from the backend only, if we
don't need to support real-time use case immediately. IMHO, in terms of the
steps to move the reader work forward, we can first focus on querying the data
from the backend, and then from collector as well.
> [Data Serving] Set up ATS reader with basic request serving structure and
> lifecycle
> -----------------------------------------------------------------------------------
>
> Key: YARN-3047
> URL: https://issues.apache.org/jira/browse/YARN-3047
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
> Reporter: Sangjin Lee
> Assignee: Varun Saxena
> Attachments: Timeline_Reader(draft).pdf, YARN-3047.001.patch,
> YARN-3047.003.patch, YARN-3047.02.patch, YARN-3047.04.patch
>
>
> Per design in YARN-2938, set up the ATS reader as a service and implement the
> basic structure as a service. It includes lifecycle management, request
> serving, and so on.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
