[jira] [Commented] (YARN-3046) [Event producers] Implement MapReduce AM writing some MR metrics to ATS
[ https://issues.apache.org/jira/browse/YARN-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14503551#comment-14503551 ] Junping Du commented on YARN-3046: -- Thanks [~sjlee0] for review and comments! bq. So am I correct in understanding that with this patch it emits events to the timeline service? Are metrics emitted as well? That's right. Sorry for missing metrics here. Can we file a separated MR metric JIRA to track with adjusting current JIRA title a bit? May be we can make our end-to-end flow work first and shooting MR metric/counters need involve more work that can happen next? bq. Why not use HierarchicalTimelineEntity for job entity and task entities? If hierarchical timeline entities don't work for MR jobs and tasks, I don't know where it will. Shouldn't we try to use it for the MR job/task/etc.? I was thinking on this before, and I agree that HierarchicalTimelineEntity is the best for describing parent-children relationship. The quick reason that we don't use HierarchicalTimelineEntity here is because HierarchicalTimelineEntity is an abstract class, and we don't have related Task/Job entity as implementations so far. Given Task/Job entity is MR specific concept rather than generic application concept, it seems a little overweight to create the subclass of HierarchicalTimelineEntity for Task/Job entities? I think TimelineEntity provide enough flexibility to describe this app-specific relationship (especially for describing other relationships, like: Tez's DAG), as we don't expect every application have to add new sub entities of TimelineEntity. Isn't it? bq. For that matter, do we need to think about emitting task attempts, or should we consider as the next step after this? Nice catch! We addressed task attempt event as task event for now but it should be a separated type of event which has Task_Attempt_ID as entity ID (point to Task_ID). Will address this in next patch. bq. JobHistoryEventHandler.java:775: is that change related with this JIRA, or is it something different? That fix a typo of v1 timeline service. I can remove it if you think the change is not necessary. [Event producers] Implement MapReduce AM writing some MR metrics to ATS --- Key: YARN-3046 URL: https://issues.apache.org/jira/browse/YARN-3046 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Junping Du Attachments: YARN-3046-no-test-v2.patch, YARN-3046-no-test.patch, YARN-3046-v1-rebase.patch, YARN-3046-v1.patch, YARN-3046-v2.patch, YARN-3046-v3.patch, YARN-3046-v4.patch Per design in YARN-2928, select a handful of MR metrics (e.g. HDFS bytes written) and have the MR AM write the framework-specific metrics to ATS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3046) [Event producers] Implement MapReduce AM writing some MR metrics to ATS
[ https://issues.apache.org/jira/browse/YARN-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14503175#comment-14503175 ] Junping Du commented on YARN-3046: -- bq. Shall we do nullcheck in TimelineEvent. Theoretically, we shouldn't block users from putting null value, though in this case we may want to avoid it. However, we can do it at the HistoryEvent side. I see. Let's remove nullcheck in TimelineEvent part. HistoryEvent is also a consumer. If we have assumption to allow consumer to put null value, should be fine in this case. [Event producers] Implement MapReduce AM writing some MR metrics to ATS --- Key: YARN-3046 URL: https://issues.apache.org/jira/browse/YARN-3046 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Junping Du Attachments: YARN-3046-no-test-v2.patch, YARN-3046-no-test.patch, YARN-3046-v1-rebase.patch, YARN-3046-v1.patch, YARN-3046-v2.patch, YARN-3046-v3.patch Per design in YARN-2928, select a handful of MR metrics (e.g. HDFS bytes written) and have the MR AM write the framework-specific metrics to ATS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3046) [Event producers] Implement MapReduce AM writing some MR metrics to ATS
[ https://issues.apache.org/jira/browse/YARN-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14503133#comment-14503133 ] Zhijie Shen commented on YARN-3046: --- Another issue: Shall we do nullcheck in TimelineEvent. Theoretically, we shouldn't block users from putting null value, though in this case we may want to avoid it. However, we can do it at the HistoryEvent side. [Event producers] Implement MapReduce AM writing some MR metrics to ATS --- Key: YARN-3046 URL: https://issues.apache.org/jira/browse/YARN-3046 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Junping Du Attachments: YARN-3046-no-test-v2.patch, YARN-3046-no-test.patch, YARN-3046-v1-rebase.patch, YARN-3046-v1.patch, YARN-3046-v2.patch, YARN-3046-v3.patch Per design in YARN-2928, select a handful of MR metrics (e.g. HDFS bytes written) and have the MR AM write the framework-specific metrics to ATS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3046) [Event producers] Implement MapReduce AM writing some MR metrics to ATS
[ https://issues.apache.org/jira/browse/YARN-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14503363#comment-14503363 ] Sangjin Lee commented on YARN-3046: --- Sorry my review is late for this [~djp]! Some high level comments: - So am I correct in understanding that with this patch it emits events to the timeline service? Are metrics emitted as well? I didn't see any changes regarding writing metrics to the timeline service. I might have missed that detail. - Why not use HierarchicalTimelineEntity for job entity and task entities? If hierarchical timeline entities don't work for MR jobs and tasks, I don't know where it will. Shouldn't we try to use it for the MR job/task/etc.? - For that matter, do we need to think about emitting task attempts, or should we consider as the next step after this? One small question: - JobHistoryEventHandler.java:775: is that change related with this JIRA, or is it something different? Thanks! [Event producers] Implement MapReduce AM writing some MR metrics to ATS --- Key: YARN-3046 URL: https://issues.apache.org/jira/browse/YARN-3046 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Junping Du Attachments: YARN-3046-no-test-v2.patch, YARN-3046-no-test.patch, YARN-3046-v1-rebase.patch, YARN-3046-v1.patch, YARN-3046-v2.patch, YARN-3046-v3.patch, YARN-3046-v4.patch Per design in YARN-2928, select a handful of MR metrics (e.g. HDFS bytes written) and have the MR AM write the framework-specific metrics to ATS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3046) [Event producers] Implement MapReduce AM writing some MR metrics to ATS
[ https://issues.apache.org/jira/browse/YARN-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500037#comment-14500037 ] Junping Du commented on YARN-3046: -- Thanks [~rkanter] and [~zjshen] for review and comments! bq. One minor thing: Is there a JIRA for this TODO? Yes. YARN-3367. Will add JIRA number in TODO comment here. bq. Task entity Id should be be the job Id, but the task Id. PS: there's a typo here Nice catch! This is definitely a bug. Fix it (and typo) in v3 patch. [Event producers] Implement MapReduce AM writing some MR metrics to ATS --- Key: YARN-3046 URL: https://issues.apache.org/jira/browse/YARN-3046 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Junping Du Attachments: YARN-3046-no-test-v2.patch, YARN-3046-no-test.patch, YARN-3046-v1-rebase.patch, YARN-3046-v1.patch, YARN-3046-v2.patch Per design in YARN-2928, select a handful of MR metrics (e.g. HDFS bytes written) and have the MR AM write the framework-specific metrics to ATS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3046) [Event producers] Implement MapReduce AM writing some MR metrics to ATS
[ https://issues.apache.org/jira/browse/YARN-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500998#comment-14500998 ] Zhijie Shen commented on YARN-3046: --- In the new patch, the task entity ID is set correctly. But can we use reflect or make getTaskId the interface method of TaskEvent to simply the code change? [Event producers] Implement MapReduce AM writing some MR metrics to ATS --- Key: YARN-3046 URL: https://issues.apache.org/jira/browse/YARN-3046 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Junping Du Attachments: YARN-3046-no-test-v2.patch, YARN-3046-no-test.patch, YARN-3046-v1-rebase.patch, YARN-3046-v1.patch, YARN-3046-v2.patch, YARN-3046-v3.patch Per design in YARN-2928, select a handful of MR metrics (e.g. HDFS bytes written) and have the MR AM write the framework-specific metrics to ATS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3046) [Event producers] Implement MapReduce AM writing some MR metrics to ATS
[ https://issues.apache.org/jira/browse/YARN-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14501015#comment-14501015 ] Junping Du commented on YARN-3046: -- bq. In the new patch, the task entity ID is set correctly. But can we use reflect or make getTaskId the interface method of TaskEvent to simply the code change? I agree that it could make code more concisely if we can differentiate job entities from task entities with refection or some other ways. Coincidentally, I also put some ideas and comments to MAPREDUCE-6318. However, can we do this refactor together with v1 timeline service in MAPREDUCE-6318? The code here far more than acceptable, I believe. [Event producers] Implement MapReduce AM writing some MR metrics to ATS --- Key: YARN-3046 URL: https://issues.apache.org/jira/browse/YARN-3046 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Junping Du Attachments: YARN-3046-no-test-v2.patch, YARN-3046-no-test.patch, YARN-3046-v1-rebase.patch, YARN-3046-v1.patch, YARN-3046-v2.patch, YARN-3046-v3.patch Per design in YARN-2928, select a handful of MR metrics (e.g. HDFS bytes written) and have the MR AM write the framework-specific metrics to ATS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3046) [Event producers] Implement MapReduce AM writing some MR metrics to ATS
[ https://issues.apache.org/jira/browse/YARN-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14497872#comment-14497872 ] Junping Du commented on YARN-3046: -- Hi [~zjshen] and [~rkanter], any comments on latest patch? Thanks! [Event producers] Implement MapReduce AM writing some MR metrics to ATS --- Key: YARN-3046 URL: https://issues.apache.org/jira/browse/YARN-3046 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Junping Du Attachments: YARN-3046-no-test-v2.patch, YARN-3046-no-test.patch, YARN-3046-v1-rebase.patch, YARN-3046-v1.patch, YARN-3046-v2.patch Per design in YARN-2928, select a handful of MR metrics (e.g. HDFS bytes written) and have the MR AM write the framework-specific metrics to ATS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3046) [Event producers] Implement MapReduce AM writing some MR metrics to ATS
[ https://issues.apache.org/jira/browse/YARN-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14499087#comment-14499087 ] Robert Kanter commented on YARN-3046: - LGTM +1 But let's wait on any additional comments from [~zjshen] One minor thing: Is there a JIRA for this TODO? {noformat} // TODO remove threadPool after adding non-blocking call in TimelineClient {noformat} [Event producers] Implement MapReduce AM writing some MR metrics to ATS --- Key: YARN-3046 URL: https://issues.apache.org/jira/browse/YARN-3046 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Junping Du Attachments: YARN-3046-no-test-v2.patch, YARN-3046-no-test.patch, YARN-3046-v1-rebase.patch, YARN-3046-v1.patch, YARN-3046-v2.patch Per design in YARN-2928, select a handful of MR metrics (e.g. HDFS bytes written) and have the MR AM write the framework-specific metrics to ATS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3046) [Event producers] Implement MapReduce AM writing some MR metrics to ATS
[ https://issues.apache.org/jira/browse/YARN-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14494233#comment-14494233 ] Junping Du commented on YARN-3046: -- Thanks [~zjshen] for review and comments! bq. I'm not sure if we should have a MR config to determine is new or old timeline service. If this MR config is set to true, but YARN cluster is still setup with old timeline service. It still doesn't work. Theoretically, the most beautiful solution is to let applications (MR, DS, etc.) doesn't aware any version of timeline service. However, we already decided to go with different methods/structures between v1 and v2 for TimelineClient, so application have to be aware of which version timeline service get used. The next option is to let application figure out timeline related info from YARN/RM, it can be done through registerApplicationMaster() in ApplicationMasterProtocol with return value for service off, v1_on, or v2_on. The last option is as v1 patch shows which along the existing way for v1 service to enable timeline service in a separated configuration: MRJobConfig.MAPREDUCE_JOB_EMIT_TIMELINE_DATA. Personally, I would prefer the 2nd option. The reason is just like you mentioned, application owner doesn't have to aware RM/YARN infrastructure details. However, this need change to YARN AM protocol, and changes on different applications (distributed shell, etc.) and mark existing MR configuration deprecated (or it would have conflict in principle of similar configurations). I would prefer to file a separated JIRA to track this more carefully as this is important but not the focus of this JIRA's scope. What do you think? bq. Node need to have JobHistoryEventUtils, you can move util method to JobHistoryUtils if you want. I tried to do so before I created JobHistoryEventUtils. However, I found we cannot do it because JobHistoryUtils is in hadoop-mapreduce-client-common component, but some consumer of method is in hadoop-mapreduce-client-core component (like: ReduceAttemptFinishedEvent, TaskAttemptFinishedEvent, etc.). Currently, hadoop-mapreduce-client-common has dependency on hadoop-mapreduce-client-core, so we don't allow these events under hadoop-mapreduce-client-core to depend on JobHistoryUtils which will cause bidirectional dependency issue. The bad news is we cannot move JobHistoryUtils to hadoop-mapreduce-client-core either, because it has reference to other classes (like: MRApps) that still in hadoop-mapreduce-client-common. That's why I create JobHistoryEventUtils for shared methods. bq. In the current way of shutting down the threadpool, is it guaranteed that the pending entity is going to be published before shutting down? It will have delay (60 secs) to wait pending entity get posted, and the delay is typically much larger than service discovery time (typically saying, heartbeat interval, not counting collector failed over case) and timeline entity REST posting time. It also larger than every entity posting time in case of failure with maximum retry (30 * 1 sec). So I think it could be safe to do so here. I will address other comments in new patch. [Event producers] Implement MapReduce AM writing some MR metrics to ATS --- Key: YARN-3046 URL: https://issues.apache.org/jira/browse/YARN-3046 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Junping Du Attachments: YARN-3046-no-test-v2.patch, YARN-3046-no-test.patch, YARN-3046-v1-rebase.patch, YARN-3046-v1.patch Per design in YARN-2928, select a handful of MR metrics (e.g. HDFS bytes written) and have the MR AM write the framework-specific metrics to ATS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3046) [Event producers] Implement MapReduce AM writing some MR metrics to ATS
[ https://issues.apache.org/jira/browse/YARN-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14495534#comment-14495534 ] Junping Du commented on YARN-3046: -- Also, verify end-to-end test for putting MR history events to poc storage backend of new timeline service works. [Event producers] Implement MapReduce AM writing some MR metrics to ATS --- Key: YARN-3046 URL: https://issues.apache.org/jira/browse/YARN-3046 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Junping Du Attachments: YARN-3046-no-test-v2.patch, YARN-3046-no-test.patch, YARN-3046-v1-rebase.patch, YARN-3046-v1.patch, YARN-3046-v2.patch Per design in YARN-2928, select a handful of MR metrics (e.g. HDFS bytes written) and have the MR AM write the framework-specific metrics to ATS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3046) [Event producers] Implement MapReduce AM writing some MR metrics to ATS
[ https://issues.apache.org/jira/browse/YARN-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14495462#comment-14495462 ] Robert Kanter commented on YARN-3046: - Sure, another JIRA sounds fine. [Event producers] Implement MapReduce AM writing some MR metrics to ATS --- Key: YARN-3046 URL: https://issues.apache.org/jira/browse/YARN-3046 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Junping Du Attachments: YARN-3046-no-test-v2.patch, YARN-3046-no-test.patch, YARN-3046-v1-rebase.patch, YARN-3046-v1.patch Per design in YARN-2928, select a handful of MR metrics (e.g. HDFS bytes written) and have the MR AM write the framework-specific metrics to ATS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3046) [Event producers] Implement MapReduce AM writing some MR metrics to ATS
[ https://issues.apache.org/jira/browse/YARN-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14495191#comment-14495191 ] Junping Du commented on YARN-3046: -- Thanks [~rkanter] for review and comments! bq. I think something that really hasn't been properly discussed is how compatible the ATS v1 APIs are with the v2 APIs, or how aware the user needs to be about which ATS they are using. (We're running into that in the other direction in YARN-2423). Thanks for pointing out YARN-2423 which I already watched for a while. I agree that this is very important to discuss our compatibility story between v1 and v2 timeline services. However, given we already have NM and AM to leverage the same new TimelinetClient API for posting related new TimelineEntity, we can go through this way for MR in the short term. Given we already have a separated JIRA to track this: YARN-3049, may be we should go ahead to start some discussions from there? bq. This isn't a GET call, so it should just say putEntityNonBlocking failed: + e Sounds good. Will update it soon. bq. Why not set the TimelineEvent Id in the toTimelineEvent() methods (or a parent method)? Nice catch! Will update it soon. bq. Given that all of the toTimelineEvent() methods are copy-paste of the switch statement in JobHistoryEventHandler#processEventForTimelineServer(), can you refactor processEventForTimelineServer() to use the toTimelineEvent() methods instead? That's a cleaner way of doing this and it's getting the same information. I know this is used by ATS v1, but we can still clean it up; especially given that it's all copy-pasted here anyway. I had the same thoughts when I was starting the effort last week. However, I had a quick conversation offline with other guys, like [~vinodkv] and [~zjshen] and get buy-in with an opinion that ATS v1 on MR metrics is a released production and there could be some ongoing effort (bug fix, etc.) could happen in parallel on trunk and branch-2. Doing such a refactor on major piece could bring more risk of conflict or even involving with new bug which is not necessary for current phase. A better plan sounds like we do this refactor work after phase 2 (end-2-end can work properly) when we are merging back YARN-2928 to trunk and branch-2. What do you think? If you also agree, I can file a separated JIRA against trunk to do the refactor work you mentioned here after we merge back. Thoughts? [Event producers] Implement MapReduce AM writing some MR metrics to ATS --- Key: YARN-3046 URL: https://issues.apache.org/jira/browse/YARN-3046 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Junping Du Attachments: YARN-3046-no-test-v2.patch, YARN-3046-no-test.patch, YARN-3046-v1-rebase.patch, YARN-3046-v1.patch Per design in YARN-2928, select a handful of MR metrics (e.g. HDFS bytes written) and have the MR AM write the framework-specific metrics to ATS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3046) [Event producers] Implement MapReduce AM writing some MR metrics to ATS
[ https://issues.apache.org/jira/browse/YARN-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14495097#comment-14495097 ] Robert Kanter commented on YARN-3046: - I think something that really hasn't been properly discussed is how compatible the ATS v1 APIs are with the v2 APIs, or how aware the user needs to be about which ATS they are using. (We're running into that in the other direction in YARN-2423). Here's some more specific comments about the patch: # In this message, {code}LOG.error(putEntityNonBlocking get failed: + e);{code} This isn't a GET call, so it should just say {{putEntityNonBlocking failed: + e}} # Why not set the TimelineEvent Id in the {{toTimelineEvent()}} methods (or a parent method)? # Given that all of the {{toTimelineEvent()}} methods are copy-paste of the switch statement in {{JobHistoryEventHandler#processEventForTimelineServer()}}, can you refactor processEventForTimelineServer() to use the {{toTimelineEvent()}} methods instead? That's a cleaner way of doing this and it's getting the same information. I know this is used by ATS v1, but we can still clean it up; especially given that it's all copy-pasted here anyway. [Event producers] Implement MapReduce AM writing some MR metrics to ATS --- Key: YARN-3046 URL: https://issues.apache.org/jira/browse/YARN-3046 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Junping Du Attachments: YARN-3046-no-test-v2.patch, YARN-3046-no-test.patch, YARN-3046-v1-rebase.patch, YARN-3046-v1.patch Per design in YARN-2928, select a handful of MR metrics (e.g. HDFS bytes written) and have the MR AM write the framework-specific metrics to ATS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3046) [Event producers] Implement MapReduce AM writing some MR metrics to ATS
[ https://issues.apache.org/jira/browse/YARN-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14492823#comment-14492823 ] Zhijie Shen commented on YARN-3046: --- Junping, thanks for the patch. Overall it looks good to me. Some comments about the patch details. 1. I'm not sure if we should have a MR config to determine is new or old timeline service. If this MR config is set to true, but YARN cluster is still setup with old timeline service. It still doesn't work. {code} 433 public static final String MAPREDUCE_JOB_NEW_TIMELINE_SERVICE_ENABLED = 434 mapreduce.job.new-timeline-service.enabled; {code} 2. Node need to have JobHistoryEventUtils, you can move util method to JobHistoryUtils if you want. 3. {{TimelineClient getTimelineClient();}} shouldn't belong to AppContext, which will be presented to the history server too. It's better to put it into RunningAppContext. 4. In the current way of shutting down the threadpool, is it guaranteed that the pending entity is going to be published before shutting down? 5. It's not documented well, but can we treat parent - child relationship as the outbound edge from parent and inbound edge to child. Therefore,{{ taskEntity.addIsRelatedToEntity(jobEntity)}}? {code} entity.addRelatesToEntity(relatedJobEntity, jobId.toString()); {code} 6. In {{TaskAttemptStartedEvent}}, TASK_TYPE is written twice. 7. createTimelineEntityFromEvent - createJobEntity and createTimelineEntityFromEventWithRelatedJobEntity - createTaskEntity? 8. Use {{IOException | YarnException e}}? {code} 1122try { 1123 timelineClient.putEntities(tEntity); 1124} catch (IOException ex) { 1125 LOG.error(Error putting entity + tEntity.getId() + to new Timeline 1126 + Service, ex); 1127} catch (YarnException ex) { {code} [Event producers] Implement MapReduce AM writing some MR metrics to ATS --- Key: YARN-3046 URL: https://issues.apache.org/jira/browse/YARN-3046 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Junping Du Attachments: YARN-3046-no-test-v2.patch, YARN-3046-no-test.patch, YARN-3046-v1-rebase.patch, YARN-3046-v1.patch Per design in YARN-2928, select a handful of MR metrics (e.g. HDFS bytes written) and have the MR AM write the framework-specific metrics to ATS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3046) [Event producers] Implement MapReduce AM writing some MR metrics to ATS
[ https://issues.apache.org/jira/browse/YARN-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14483588#comment-14483588 ] Junping Du commented on YARN-3046: -- Linked with MAPREDUCE-6189 - the test failure on trunk is solid, not only on my local test bed. [Event producers] Implement MapReduce AM writing some MR metrics to ATS --- Key: YARN-3046 URL: https://issues.apache.org/jira/browse/YARN-3046 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Junping Du Attachments: YARN-3046-no-test-v2.patch, YARN-3046-no-test.patch Per design in YARN-2928, select a handful of MR metrics (e.g. HDFS bytes written) and have the MR AM write the framework-specific metrics to ATS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3046) [Event producers] Implement MapReduce AM writing some MR metrics to ATS
[ https://issues.apache.org/jira/browse/YARN-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14393089#comment-14393089 ] Junping Du commented on YARN-3046: -- Also, looks like no YARN changes get involved in this JIRA, will migrate it to MAPREDUCE project later. [Event producers] Implement MapReduce AM writing some MR metrics to ATS --- Key: YARN-3046 URL: https://issues.apache.org/jira/browse/YARN-3046 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Junping Du Attachments: YARN-3046-no-test.patch Per design in YARN-2928, select a handful of MR metrics (e.g. HDFS bytes written) and have the MR AM write the framework-specific metrics to ATS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3046) [Event producers] Implement MapReduce AM writing some MR metrics to ATS
[ https://issues.apache.org/jira/browse/YARN-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391142#comment-14391142 ] Junping Du commented on YARN-3046: -- Hi [~rkanter], as I mentioned offline early, can I take on this JIRA and move it forward? Thanks! [Event producers] Implement MapReduce AM writing some MR metrics to ATS --- Key: YARN-3046 URL: https://issues.apache.org/jira/browse/YARN-3046 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Robert Kanter Per design in YARN-2928, select a handful of MR metrics (e.g. HDFS bytes written) and have the MR AM write the framework-specific metrics to ATS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-3046) [Event producers] Implement MapReduce AM writing some MR metrics to ATS
[ https://issues.apache.org/jira/browse/YARN-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391436#comment-14391436 ] Junping Du commented on YARN-3046: -- Thanks [~rkanter]! I will try to deliver a patch soon. Hopefully, you can help to review it. [Event producers] Implement MapReduce AM writing some MR metrics to ATS --- Key: YARN-3046 URL: https://issues.apache.org/jira/browse/YARN-3046 Project: Hadoop YARN Issue Type: Sub-task Components: timelineserver Reporter: Sangjin Lee Assignee: Junping Du Per design in YARN-2928, select a handful of MR metrics (e.g. HDFS bytes written) and have the MR AM write the framework-specific metrics to ATS. -- This message was sent by Atlassian JIRA (v6.3.4#6332)