[jira] [Commented] (YARN-3046) [Event producers] Implement MapReduce AM writing MR events to v2 ATS
[ https://issues.apache.org/jira/browse/YARN-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505580#comment-14505580 ] Junping Du commented on YARN-3046: -- bq. If that turns out to be the case, we could conceivably create subclasses in a standard place for MR, and have all use cases use those concrete subclasses. But I'm fine deferring that aspect a little bit. It's not a critical point. Another option is to make HierarchicalTimelineEntity non abstract? I also agree we should open a new JIRA to discuss this later. [Event producers] Implement MapReduce AM writing MR events to v2 ATS Key: YARN-3046 URL: https://issues.apache.org/jira/browse/YARN-3046 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Junping Du Attachments: YARN-3046-no-test-v2.patch, YARN-3046-no-test.patch, YARN-3046-v1-rebase.patch, YARN-3046-v1.patch, YARN-3046-v2.patch, YARN-3046-v3.patch, YARN-3046-v4.patch, YARN-3046-v5.patch, YARN-3046-v6.patch Per design in YARN-2928, select a handful of MR metrics (e.g. HDFS bytes written) and have the MR AM write the framework-specific metrics to ATS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3046) [Event producers] Implement MapReduce AM writing MR events to v2 ATS
[ https://issues.apache.org/jira/browse/YARN-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504712#comment-14504712 ] Junping Du commented on YARN-3046: -- Thanks [~zjshen] and [~sjlee0] for review and comments! bq. createEntity's comments need to be updated to reflect the latest code changes. Nice catch! Updated in v6 patch. bq. The following code does nothing, and can be removed. The only thing it does is get rid of falling into default (for unrecognized event) where it will be return directly. Let's keep it here. bq. OK, this is another existing bug I'll leave it up to you to decide whether we want to fix this in ATS v.1 in a separate JIRA. Let's fix in refactor patch (MAPREDUCE-6318) given we already roll back a prefix tiny bug on v1. bq. I.150: same issue Good catch! Fix it in v6 patch. [Event producers] Implement MapReduce AM writing MR events to v2 ATS Key: YARN-3046 URL: https://issues.apache.org/jira/browse/YARN-3046 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Junping Du Attachments: YARN-3046-no-test-v2.patch, YARN-3046-no-test.patch, YARN-3046-v1-rebase.patch, YARN-3046-v1.patch, YARN-3046-v2.patch, YARN-3046-v3.patch, YARN-3046-v4.patch, YARN-3046-v5.patch Per design in YARN-2928, select a handful of MR metrics (e.g. HDFS bytes written) and have the MR AM write the framework-specific metrics to ATS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3046) [Event producers] Implement MapReduce AM writing MR events to v2 ATS
[ https://issues.apache.org/jira/browse/YARN-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505259#comment-14505259 ] Zhijie Shen commented on YARN-3046: --- Will commit the last patch late today to give Sangjin some time to check the last patch. [Event producers] Implement MapReduce AM writing MR events to v2 ATS Key: YARN-3046 URL: https://issues.apache.org/jira/browse/YARN-3046 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Junping Du Attachments: YARN-3046-no-test-v2.patch, YARN-3046-no-test.patch, YARN-3046-v1-rebase.patch, YARN-3046-v1.patch, YARN-3046-v2.patch, YARN-3046-v3.patch, YARN-3046-v4.patch, YARN-3046-v5.patch, YARN-3046-v6.patch Per design in YARN-2928, select a handful of MR metrics (e.g. HDFS bytes written) and have the MR AM write the framework-specific metrics to ATS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3046) [Event producers] Implement MapReduce AM writing MR events to v2 ATS
[ https://issues.apache.org/jira/browse/YARN-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14505286#comment-14505286 ] Sangjin Lee commented on YARN-3046: --- LGTM. Just to close on one point, at a later point let's consider whether HierarchicalTimelineEntity makes more sense for the MR entities. If the parent-child navigation becomes more important (it's not clear if it will), then there seems to be a slight advantage with using HierarchicalTimelineEntity. If that turns out to be the case, we could conceivably create subclasses in a standard place for MR, and have all use cases use those concrete subclasses. But I'm fine deferring that aspect a little bit. It's not a critical point. [Event producers] Implement MapReduce AM writing MR events to v2 ATS Key: YARN-3046 URL: https://issues.apache.org/jira/browse/YARN-3046 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Junping Du Attachments: YARN-3046-no-test-v2.patch, YARN-3046-no-test.patch, YARN-3046-v1-rebase.patch, YARN-3046-v1.patch, YARN-3046-v2.patch, YARN-3046-v3.patch, YARN-3046-v4.patch, YARN-3046-v5.patch, YARN-3046-v6.patch Per design in YARN-2928, select a handful of MR metrics (e.g. HDFS bytes written) and have the MR AM write the framework-specific metrics to ATS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3046) [Event producers] Implement MapReduce AM writing MR events to v2 ATS
[ https://issues.apache.org/jira/browse/YARN-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504118#comment-14504118 ] Sangjin Lee commented on YARN-3046: --- Looks fine for the most part. One small point: (JobHistoryEventHandler.java) - l.851: OK, this is another existing bug :) I'll leave it up to you to decide whether we want to fix this in ATS v.1 in a separate JIRA (JobFinishedEvent.java) - l.150: same issue [Event producers] Implement MapReduce AM writing MR events to v2 ATS Key: YARN-3046 URL: https://issues.apache.org/jira/browse/YARN-3046 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Junping Du Attachments: YARN-3046-no-test-v2.patch, YARN-3046-no-test.patch, YARN-3046-v1-rebase.patch, YARN-3046-v1.patch, YARN-3046-v2.patch, YARN-3046-v3.patch, YARN-3046-v4.patch, YARN-3046-v5.patch Per design in YARN-2928, select a handful of MR metrics (e.g. HDFS bytes written) and have the MR AM write the framework-specific metrics to ATS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3046) [Event producers] Implement MapReduce AM writing MR events to v2 ATS
[
https://issues.apache.org/jira/browse/YARN-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504066#comment-14504066
]
Zhijie Shen commented on YARN-3046:
---
Hopefully we can agree the scope of this jira to add ATS v2 integration
side-by-side to v1 (in addition to the fix of TASK_ATTEMPT_ENTITY, which we
missed in v1).
+1 for the latest patch except two nits:
1. createEntity's comments need to be updated to reflect the latest code
changes.
2. The following code does nothing, and can be removed.
{code}
1117 // Handle job events
1118 case JOB_SUBMITTED:
1119 case JOB_STATUS_CHANGED:
1120 case JOB_INFO_CHANGED:
1121 case JOB_INITED:
1122 case JOB_PRIORITY_CHANGED:
1123 case JOB_QUEUE_CHANGED:
1124 case JOB_FAILED:
1125 case JOB_KILLED:
1126 case JOB_ERROR:
1127 case JOB_FINISHED:
1128 case AM_STARTED:
1129 case NORMALIZED_RESOURCE:
1130break;
{code}
[~sjlee0], any comment on the last patch?
[Event producers] Implement MapReduce AM writing MR events to v2 ATS
Key: YARN-3046
URL: https://issues.apache.org/jira/browse/YARN-3046
Project: Hadoop YARN
Issue Type: Sub-task
Components: timelineserver
Reporter: Sangjin Lee
Assignee: Junping Du
Attachments: YARN-3046-no-test-v2.patch, YARN-3046-no-test.patch,
YARN-3046-v1-rebase.patch, YARN-3046-v1.patch, YARN-3046-v2.patch,
YARN-3046-v3.patch, YARN-3046-v4.patch, YARN-3046-v5.patch
Per design in YARN-2928, select a handful of MR metrics (e.g. HDFS bytes
written) and have the MR AM write the framework-specific metrics to ATS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
