[
https://issues.apache.org/jira/browse/MAPREDUCE-479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732362#action_12732362
]
Jiaqi Tan commented on MAPREDUCE-479:
-------------------------------------
I would argue that you cannot actually get the information accurately from TT
logs; doing that would require the assumption of clock synchronization which is
a common enough assumption in academia but in practice does not always hold
true. The clienttrace messages are logged on the TT which is serving the map
outputs, and are timestamped with the clock of the TT serving the map outputs.
But the reduce attempts are running on other TTs, and those log messages that
record the running of the reduce attempts are timestamped with the clock of the
TT running the reduce. The only way to get ground truth that reflects causality
is to pass the reduce attempt ID unless you have a totally ordered global clock
in the system. We could get that from the JT, but unfortunately the job history
logs for some reason record 0 times for failed attempts.
What I'm suggesting is that short of putting the reduce attempt ID in the
shuffle protocol, there is no other way of obtaining that information
accurately, any other way is subject to assumptions such as clock
synchronization or assumptions about the scheduling protocol. Also, in previous
versions, the shuffle passed the entire reduce attempt ID in the HTTP request
IIRC, so I'm not seeing why that's an issue now to put it back.
> Add reduce ID to shuffle clienttrace
> ------------------------------------
>
> Key: MAPREDUCE-479
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-479
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.21.0
> Reporter: Jiaqi Tan
> Assignee: Jiaqi Tan
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: HADOOP-6013.patch, MAPREDUCE-479-1.patch,
> MAPREDUCE-479-2.patch, MAPREDUCE-479-3.patch, MAPREDUCE-479.patch
>
>
> Current clienttrace messages from shuffles note only the destination map ID
> but not the source reduce ID. Having both source and destination ID of each
> shuffle enables full tracing of execution.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
