[jira] [Updated] (YARN-3041) [Data Model] create overall data objects of TS next gen
[ https://issues.apache.org/jira/browse/YARN-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Saxena updated YARN-3041: --- Fix Version/s: 2.9.0 > [Data Model] create overall data objects of TS next gen > --- > > Key: YARN-3041 > URL: https://issues.apache.org/jira/browse/YARN-3041 > Project: Hadoop YARN > Issue Type: Sub-task > Components: timelineserver >Reporter: Sangjin Lee >Assignee: Zhijie Shen > Fix For: 2.9.0, 3.0.0-alpha1 > > Attachments: Data_model_proposal_v2.pdf, YARN-3041.2.patch, > YARN-3041.3.patch, YARN-3041.4.patch, YARN-3041.5.patch, > YARN-3041.preliminary.001.patch > > > Per design in YARN-2928, create the ATS entity and events API. > Also, as part of this JIRA, create YARN system entities (e.g. cluster, user, > flow, flow run, YARN app, ...). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-3041) [Data Model] create overall data objects of TS next gen
[ https://issues.apache.org/jira/browse/YARN-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-3041: -- Attachment: YARN-3041.5.patch Thanks for the feedback, Sangjin, Vrushali and Joep! We had an offline discussion. I updated the patch according to it. Here's the summary of the major changes: 1. It is necessary to have both Flow and FlowRun in the taxonomy, as the concepts of them are most the same. FlowRun is more likely to model an individual flow instance of a number applications while Flow sounds like a the generic perspective of application organization, which may be nested multiple FlowRun instances. Hence, we just need to have FlowRun only, but rename FlowRun to Flow for simplicity. 2. To address the aggregation interval, which means we may want to query the aggregated information for a particular time window, I change TimelineMetric to have starttime and endtime attributes. 3. The types of the first class citizen entities are defined centrally as the enums, and the parent-child relationship is defined there too. 4. In the write path, queue is the string attribute of application while user is the string attribute of the flow, while we still have the entities of both to put the aggregated data at the reader side. One additional implication is that all the applications are going to be run by the same user of the parent flow. 5. Flow id is the composite: user@flow_name(or id)/version/run, which will uniquely identify a flow in the storage. Joep has raised a great point of keeping the type generic to extend the data model beyond YARN, such as Mesos. I think we can think and discuss more around it, but let's file a separate Jira to tackle this direction. Here, as mentioned above, let's try to get the first draft of data model in asap to unblock the aggregator and the reader work. Hopefully it makes sense to the folks here. [Data Model] create overall data objects of TS next gen --- Key: YARN-3041 URL: https://issues.apache.org/jira/browse/YARN-3041 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Zhijie Shen Attachments: Data_model_proposal_v2.pdf, YARN-3041.2.patch, YARN-3041.3.patch, YARN-3041.4.patch, YARN-3041.5.patch, YARN-3041.preliminary.001.patch Per design in YARN-2928, create the ATS entity and events API. Also, as part of this JIRA, create YARN system entities (e.g. cluster, user, flow, flow run, YARN app, ...). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3041) [Data Model] create overall data objects of TS next gen
[ https://issues.apache.org/jira/browse/YARN-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-3041: -- Summary: [Data Model] create overall data objects of TS next gen (was: [Data Model] create the ATS entity/event API) [Data Model] create overall data objects of TS next gen --- Key: YARN-3041 URL: https://issues.apache.org/jira/browse/YARN-3041 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Zhijie Shen Attachments: Data_model_proposal_v2.pdf, YARN-3041.2.patch, YARN-3041.3.patch, YARN-3041.preliminary.001.patch Per design in YARN-2928, create the ATS entity and events API. Also, as part of this JIRA, create YARN system entities (e.g. cluster, user, flow, flow run, YARN app, ...). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3041) [Data Model] create overall data objects of TS next gen
[ https://issues.apache.org/jira/browse/YARN-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-3041: -- Attachment: (was: YARN-3041.4.patch) [Data Model] create overall data objects of TS next gen --- Key: YARN-3041 URL: https://issues.apache.org/jira/browse/YARN-3041 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Zhijie Shen Attachments: Data_model_proposal_v2.pdf, YARN-3041.2.patch, YARN-3041.3.patch, YARN-3041.4.patch, YARN-3041.preliminary.001.patch Per design in YARN-2928, create the ATS entity and events API. Also, as part of this JIRA, create YARN system entities (e.g. cluster, user, flow, flow run, YARN app, ...). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3041) [Data Model] create overall data objects of TS next gen
[ https://issues.apache.org/jira/browse/YARN-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-3041: -- Attachment: YARN-3041.4.patch [Data Model] create overall data objects of TS next gen --- Key: YARN-3041 URL: https://issues.apache.org/jira/browse/YARN-3041 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Zhijie Shen Attachments: Data_model_proposal_v2.pdf, YARN-3041.2.patch, YARN-3041.3.patch, YARN-3041.4.patch, YARN-3041.preliminary.001.patch Per design in YARN-2928, create the ATS entity and events API. Also, as part of this JIRA, create YARN system entities (e.g. cluster, user, flow, flow run, YARN app, ...). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (YARN-3041) [Data Model] create overall data objects of TS next gen
[ https://issues.apache.org/jira/browse/YARN-3041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijie Shen updated YARN-3041: -- Attachment: YARN-3041.4.patch [Data Model] create overall data objects of TS next gen --- Key: YARN-3041 URL: https://issues.apache.org/jira/browse/YARN-3041 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Zhijie Shen Attachments: Data_model_proposal_v2.pdf, YARN-3041.2.patch, YARN-3041.3.patch, YARN-3041.4.patch, YARN-3041.preliminary.001.patch Per design in YARN-2928, create the ATS entity and events API. Also, as part of this JIRA, create YARN system entities (e.g. cluster, user, flow, flow run, YARN app, ...). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
