[
https://issues.apache.org/jira/browse/YARN-3981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15448925#comment-15448925
]
Rohith Sharma K S commented on YARN-3981:
-----------------------------------------
Hi [~sjlee0], thanks for the summary of the discussion.
bq. This client would launch a special YARN app under the covers whose app
master and its associated timeline writer can serve as the proxy for timeline
data the client may write. When this special timeline client shuts down, it
would tear down the associated YARN app also.
I assume launching a special is just same as normal-app where in YARN
frameworks take care of writing all the corresponding data to ATS2. Are you
referring special app as Uber job? Even uber job, for each application there
will be corresponding another job in the list. This doubles the storage are
also.
One another approach is
As part of NM daemon, start new service same as TimeLineWriterWebService. Idea
is NM reports all these collector address to RM. Introduce new API in
clientRMservice to get collector address. Address is given by RM in random(This
can be decided later). This address is used by timeline client. TimeLineClient
exposes new constructor with an flowName. So system properties can be written
at flow level.
These special entities are stored in separate table i.e with key
*clusterid!flowName!entityType!entityId*.
I would appreciate if folks give their suggestions/comments on the new approach.
> support timeline clients not associated with an application
> -----------------------------------------------------------
>
> Key: YARN-3981
> URL: https://issues.apache.org/jira/browse/YARN-3981
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
> Affects Versions: YARN-2928
> Reporter: Sangjin Lee
> Assignee: Rohith Sharma K S
> Labels: YARN-5355
>
> In the current v.2 design, all timeline writes must belong in a
> flow/application context (cluster + user + flow + flow run + application).
> But there are use cases that require writing data outside the context of an
> application. One such example is a higher level client (e.g. tez client or
> hive/oozie/cascading client) writing flow-level data that spans multiple
> applications. We need to find a way to support them.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
