[
https://issues.apache.org/jira/browse/TEZ-1610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14236544#comment-14236544
]
Siddharth Seth commented on TEZ-1610:
-------------------------------------
I think both are useful metrics (when shuffle completed, and when the merger
was closed). I think it's better to rename them though.
Minor:
- The TaskCounter is called MERGE_FINISH_TIME, it's referred to as
MERGE_TIME_TAKEN in Shuffle.
- Make parameters final in Shuffle
Wondering if this should just be something along the lines of
MERGED_INPUT_READY_DELTA (or a better name).
MERGE_TIME_TAKEN can be misleading - since the actual merge is likely to be in
progress while the data is being read from the input (50 in memory segments for
example - which don't trigger any limits will only be merged while being
consumed).
SHUFFLE_TIME_TAKEN - similar. This isn't really absolute shuffle time taken,
since a merge could happen while transferring data which would lead to the
fetchers sitting idle. Also doesn't factor in the time waiting for events - in
case of pre-starting tasks for example.
When per-IO counters are not enabled, and there's multiple Inputs, this will
end up being the value for the latest merge / fetcher which completes, and may
not be as useful.
Need some docs to clarify the behaviour in TaskCounters
> additional task counters for fetchers
> -------------------------------------
>
> Key: TEZ-1610
> URL: https://issues.apache.org/jira/browse/TEZ-1610
> Project: Apache Tez
> Issue Type: Bug
> Reporter: Rajesh Balamohan
> Assignee: Rajesh Balamohan
> Attachments: TEZ-1610.1.patch
>
>
> - ShuffleFinishTime (per source)
> - Merge time (depending on broadcast/scatter-gather shuffle)
> This would be helpful in determining when shuffle started/ended for different
> sources in a task.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
