[
https://issues.apache.org/jira/browse/YARN-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14503551#comment-14503551
]
Junping Du commented on YARN-3046:
----------------------------------
Thanks [~sjlee0] for review and comments!
bq. So am I correct in understanding that with this patch it emits events to
the timeline service? Are metrics emitted as well?
That's right. Sorry for missing metrics here. Can we file a separated MR metric
JIRA to track with adjusting current JIRA title a bit? May be we can make our
end-to-end flow work first and shooting MR metric/counters need involve more
work that can happen next?
bq. Why not use HierarchicalTimelineEntity for job entity and task entities? If
hierarchical timeline entities don't work for MR jobs and tasks, I don't know
where it will. Shouldn't we try to use it for the MR job/task/etc.?
I was thinking on this before, and I agree that HierarchicalTimelineEntity is
the best for describing parent-children relationship. The quick reason that we
don't use HierarchicalTimelineEntity here is because HierarchicalTimelineEntity
is an abstract class, and we don't have related Task/Job entity as
implementations so far. Given Task/Job entity is MR specific concept rather
than generic application concept, it seems a little overweight to create the
subclass of HierarchicalTimelineEntity for Task/Job entities? I think
TimelineEntity provide enough flexibility to describe this app-specific
relationship (especially for describing other relationships, like: Tez's DAG),
as we don't expect every application have to add new sub entities of
TimelineEntity. Isn't it?
bq. For that matter, do we need to think about emitting task attempts, or
should we consider as the next step after this?
Nice catch! We addressed task attempt event as task event for now but it should
be a separated type of event which has Task_Attempt_ID as entity ID (point to
Task_ID). Will address this in next patch.
bq. JobHistoryEventHandler.java:775: is that change related with this JIRA, or
is it something different?
That fix a typo of v1 timeline service. I can remove it if you think the change
is not necessary.
> [Event producers] Implement MapReduce AM writing some MR metrics to ATS
> -----------------------------------------------------------------------
>
> Key: YARN-3046
> URL: https://issues.apache.org/jira/browse/YARN-3046
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: timelineserver
> Reporter: Sangjin Lee
> Assignee: Junping Du
> Attachments: YARN-3046-no-test-v2.patch, YARN-3046-no-test.patch,
> YARN-3046-v1-rebase.patch, YARN-3046-v1.patch, YARN-3046-v2.patch,
> YARN-3046-v3.patch, YARN-3046-v4.patch
>
>
> Per design in YARN-2928, select a handful of MR metrics (e.g. HDFS bytes
> written) and have the MR AM write the framework-specific metrics to ATS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
