[jira] [Commented] (YARN-3041) create the ATS entity/event API
[ https://issues.apache.org/jira/browse/YARN-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14316597#comment-14316597 ] Sangjin Lee commented on YARN-3041: --- Thanks for the clarification Robert. > create the ATS entity/event API > --- > > Key: YARN-3041 > URL: https://issues.apache.org/jira/browse/YARN-3041 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Robert Kanter > Attachments: YARN-3041.preliminary.001.patch > > > Per design in YARN-2928, create the ATS entity and events API. > Also, as part of this JIRA, create YARN system entities (e.g. cluster, user, > flow, flow run, YARN app, ...). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3041) create the ATS entity/event API
[ https://issues.apache.org/jira/browse/YARN-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14315842#comment-14315842 ] Robert Kanter commented on YARN-3041: - An event in the current ATS is state transitions. For example, all of the stuff that goes into a jhist file (JOB_FAILED, JOB_INFO_CHANGED, etc). See MAPREDUCE-5933 where I updated the MR AM to post events to the ATS for specific examples. The difficulty in explicitly defining the states is that these are AM states, so they are framework-specific; while the ATS is YARN, so we can't do that. e.g. The ATS shouldn't know what a MAP_ATTEMPT_STARTED event is. > create the ATS entity/event API > --- > > Key: YARN-3041 > URL: https://issues.apache.org/jira/browse/YARN-3041 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Robert Kanter > Attachments: YARN-3041.preliminary.001.patch > > > Per design in YARN-2928, create the ATS entity and events API. > Also, as part of this JIRA, create YARN system entities (e.g. cluster, user, > flow, flow run, YARN app, ...). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3041) create the ATS entity/event API
[ https://issues.apache.org/jira/browse/YARN-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14314752#comment-14314752 ] Sangjin Lee commented on YARN-3041: --- Hitesh on YARN-2928 brought up an interesting point regarding the events (also see my reply). For my own education, what is an event in current ATS? Is it explicitly about affecting state changes in entities? Or can it be something else? How should events be defined in the next gen timeline service? And/or should the notion of the "state" be explicitly defined? Thoughts? > create the ATS entity/event API > --- > > Key: YARN-3041 > URL: https://issues.apache.org/jira/browse/YARN-3041 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Robert Kanter > Attachments: YARN-3041.preliminary.001.patch > > > Per design in YARN-2928, create the ATS entity and events API. > Also, as part of this JIRA, create YARN system entities (e.g. cluster, user, > flow, flow run, YARN app, ...). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3041) create the ATS entity/event API
[ https://issues.apache.org/jira/browse/YARN-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14310026#comment-14310026 ] Zhijie Shen commented on YARN-3041: --- bq. IMO it might make sense to define all YARN system entities as explicit types Make sense to me. > create the ATS entity/event API > --- > > Key: YARN-3041 > URL: https://issues.apache.org/jira/browse/YARN-3041 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Robert Kanter > Attachments: YARN-3041.preliminary.001.patch > > > Per design in YARN-2928, create the ATS entity and events API. > Also, as part of this JIRA, create YARN system entities (e.g. cluster, user, > flow, flow run, YARN app, ...). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3041) create the ATS entity/event API
[ https://issues.apache.org/jira/browse/YARN-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309612#comment-14309612 ] Sangjin Lee commented on YARN-3041: --- [~rkanter], [~Naganarasimha], IMO it might make sense to define all YARN system entities as explicit types. It would include flow runs, YARN apps, app attempts, and containers. They have well-defined meaning and relationship, so it seems natural to me? Thoughts? > create the ATS entity/event API > --- > > Key: YARN-3041 > URL: https://issues.apache.org/jira/browse/YARN-3041 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Robert Kanter > Attachments: YARN-3041.preliminary.001.patch > > > Per design in YARN-2928, create the ATS entity and events API. > Also, as part of this JIRA, create YARN system entities (e.g. cluster, user, > flow, flow run, YARN app, ...). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3041) create the ATS entity/event API
[ https://issues.apache.org/jira/browse/YARN-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14307365#comment-14307365 ] Naganarasimha G R commented on YARN-3041: - hi [~rkanter], I have one query : As part of 3045 NM needs to write container lifecycle events and container system metrics, So was wondering whether we need to also treat container as a special entity (TimelineServiceAppContainer or something). > create the ATS entity/event API > --- > > Key: YARN-3041 > URL: https://issues.apache.org/jira/browse/YARN-3041 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Robert Kanter > Attachments: YARN-3041.preliminary.001.patch > > > Per design in YARN-2928, create the ATS entity and events API. > Also, as part of this JIRA, create YARN system entities (e.g. cluster, user, > flow, flow run, YARN app, ...). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3041) create the ATS entity/event API
[
https://issues.apache.org/jira/browse/YARN-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14299548#comment-14299548
]
Robert Kanter commented on YARN-3041:
-
I'm glad it mostly matches up with the doc.
1. I think that makes sense. A Metric doesn't need all of the stuff that it's
inheriting from the {{TimelineServiceEntity}}. I'm already using the old
{{TimelineEvent}}, which matches up with what you had in the doc (other than
having {{eventInfo}} instead of {{metadata}}
2. It sounds like we may need more discussion on this area. As [~sjlee0]
pointed out, we had originally said a single parent to have a linear hierarchy
for aggregation. This is different than the "Relates to" and "Is related to"
in the doc and having a DAG. I wonder if it makes sense to have a parent-child
relationship only to relate the entities to each other (e.g. Application is a
child of Run, etc), and some other structure (not sure what) for aggregation?
That would help us capture other aggregation paths for things that don't fit in
the parental hierarchy. Though that makes things more complicated :(
3. You're right: they don't really need all the stuff they're inheriting from
{{TimelineServiceEntity}}. I think they really only need the relationship
field(s) and an id.
I'll do some refactoring for another prelim version.
> create the ATS entity/event API
> ---
>
> Key: YARN-3041
> URL: https://issues.apache.org/jira/browse/YARN-3041
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
>Reporter: Sangjin Lee
>Assignee: Robert Kanter
> Attachments: YARN-3041.preliminary.001.patch
>
>
> Per design in YARN-2928, create the ATS entity and events API.
> Also, as part of this JIRA, create YARN system entities (e.g. cluster, user,
> flow, flow run, YARN app, ...).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (YARN-3041) create the ATS entity/event API
[ https://issues.apache.org/jira/browse/YARN-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14299517#comment-14299517 ] Sangjin Lee commented on YARN-3041: --- bq. I suggest using more generalized in/outbound relationship instead of parent-child one. One parent can have multiple children obviously, but we said in the current design that we want to limit the parent to be one. The consideration was that the parent-child relationship is used really to handle the aggregation along the linear hierarchy, and multiple parents complicate that significantly. > create the ATS entity/event API > --- > > Key: YARN-3041 > URL: https://issues.apache.org/jira/browse/YARN-3041 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Robert Kanter > Attachments: YARN-3041.preliminary.001.patch > > > Per design in YARN-2928, create the ATS entity and events API. > Also, as part of this JIRA, create YARN system entities (e.g. cluster, user, > flow, flow run, YARN app, ...). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3041) create the ATS entity/event API
[ https://issues.apache.org/jira/browse/YARN-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14298182#comment-14298182 ] Zhijie Shen commented on YARN-3041: --- [~rkanter], thanks for the patch. I've just attached my data model proposal in the umbrella jira. It seems that we're almost on the same page. Several major differences I've noticed: 1. Metric and event are separate from entity as they are compound too. 2. I suggest using more generalized in/outbound relationship instead of parent-child one. For example, multiple vertex in a dag. 3. I'm wondering if cluster and user are really necessary to be standalone entity instead of one field of an entity. Thoughts? > create the ATS entity/event API > --- > > Key: YARN-3041 > URL: https://issues.apache.org/jira/browse/YARN-3041 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Robert Kanter > Attachments: YARN-3041.preliminary.001.patch > > > Per design in YARN-2928, create the ATS entity and events API. > Also, as part of this JIRA, create YARN system entities (e.g. cluster, user, > flow, flow run, YARN app, ...). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
