[
https://issues.apache.org/jira/browse/MAPREDUCE-6424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15259517#comment-15259517
]
Sangjin Lee commented on MAPREDUCE-6424:
----------------------------------------
{quote}
This seems to be a good idea for oozie like flows but just one query, would it
be good for app to publish metrics in SystemEntities ? IIRC YARN-3401 was
raised to block it later on ? so do we need to support some other mechanism to
publish only metrics ?
{quote}
We may need to refine what we do on YARN-3401. YARN-3401 notwithstanding, we
still have a need to support aggregating framework-owned metrics. Consider our
famous example: HDFS bytes written for the job. If we emit it for the MR job
entity, then it would not benefit from aggregation onto the flow and so on. One
can query for the job entities and get the HDFS bytes written for individual
jobs, but there would be no flow aggregation. By adding the metrics on the YARN
application entity, the flow aggregation will kick in naturally and be able to
show it at the flow run level.
If we don't do this, then we need to find a way to transfer metrics from
certain framework-specific entities to the application entities so that they
can be aggregated. I would think that would be considerably more involved.
{quote}
Ok basically was trying to understand for the next jira, may be we can discuss
more there. Query was : Apart from metrics getting published at the end of each
task(M/R) we need to publish metrics when task(M/R) is running or have some way
to share running task's (M/R) metrics to AM(via umbilical proto) and AM publish
aggregated running tasks (M/R) metrics periodically ?
{quote}
I see. Yeah, I haven't given a lot of thoughts on exactly how we might do this.
As you mentioned, we can carry on the discussion in a new JIRA.
> Store MR counters as timeline metrics instead of event
> ------------------------------------------------------
>
> Key: MAPREDUCE-6424
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6424
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Reporter: Junping Du
> Assignee: Naganarasimha G R
> Labels: yarn-2928-1st-milestone
> Attachments: MAPREDUCE-6424-YARN-2928.v1.001.patch
>
>
> In MAPREDUCE-6327, we make map/reduce counters get encoded from
> JobFinishedEvent as timeline events with counters details in JSON format.
> We need to store framework specific counters as metrics in timeline service
> to support query, aggregation, etc.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
