Junping Du commented on YARN-3046:

Thanks [~sjlee0] for review and comments!
bq. So am I correct in understanding that with this patch it emits events to 
the timeline service? Are metrics emitted as well?
That's right. Sorry for missing metrics here. Can we file a separated MR metric 
JIRA to track with adjusting current JIRA title a bit? May be we can make our 
end-to-end flow work first and shooting MR metric/counters need involve more 
work that can happen next? 

bq. Why not use HierarchicalTimelineEntity for job entity and task entities? If 
hierarchical timeline entities don't work for MR jobs and tasks, I don't know 
where it will. Shouldn't we try to use it for the MR job/task/etc.?
I was thinking on this before, and I agree that HierarchicalTimelineEntity is 
the best for describing parent-children relationship. The quick reason that we 
don't use HierarchicalTimelineEntity here is because HierarchicalTimelineEntity 
is an abstract class, and we don't have related Task/Job entity as 
implementations so far. Given Task/Job entity is MR specific concept rather 
than generic application concept, it seems a little overweight to create the 
subclass of HierarchicalTimelineEntity for Task/Job entities? I think 
TimelineEntity provide enough  flexibility to describe this app-specific 
relationship (especially for describing other relationships, like: Tez's DAG), 
as we don't expect every application have to add new sub entities of 
TimelineEntity. Isn't it?

bq. For that matter, do we need to think about emitting task attempts, or 
should we consider as the next step after this?
Nice catch! We addressed task attempt event as task event for now but it should 
be a separated type of event which has Task_Attempt_ID as entity ID (point to 
Task_ID). Will address this in next patch.

bq. JobHistoryEventHandler.java:775: is that change related with this JIRA, or 
is it something different?
That fix a typo of v1 timeline service. I can remove it if you think the change 
is not necessary.

> [Event producers] Implement MapReduce AM writing some MR metrics to ATS
> -----------------------------------------------------------------------
>                 Key: YARN-3046
>                 URL: https://issues.apache.org/jira/browse/YARN-3046
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: timelineserver
>            Reporter: Sangjin Lee
>            Assignee: Junping Du
>         Attachments: YARN-3046-no-test-v2.patch, YARN-3046-no-test.patch, 
> YARN-3046-v1-rebase.patch, YARN-3046-v1.patch, YARN-3046-v2.patch, 
> YARN-3046-v3.patch, YARN-3046-v4.patch
> Per design in YARN-2928, select a handful of MR metrics (e.g. HDFS bytes 
> written) and have the MR AM write the framework-specific metrics to ATS.

This message was sent by Atlassian JIRA

Reply via email to