[
https://issues.apache.org/jira/browse/MAPREDUCE-479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12732356#action_12732356
]
Chris Douglas commented on MAPREDUCE-479:
-----------------------------------------
bq. if the system's operation cannot be exposed at all then that greatly
detracts from the usability of the system for users. [...] needing to perform
time-based + host-based disambiguation of multiple attempts of the same task
requires the assumption that the same task will not have multiple attempts
being run on the same host, which is even more opaque than maintaining a
publicly-documented (i.e. with this JIRA) side-effect.
I think you and I differ in our perspectives of users. Even granting the
premise that traceability greatly improves diagnosis of system problems, that
Hadoop will be less usable to those writing map/reduce programs because it's
possible that, someday, someone may write a scheduler that speculatively
executes two instances of the same task on the same machine at the same time...
does not follow. The point we're debating is not whether tracing is good, but
whether including the task attempt ID as a permanent feature of the shuffle
protocol is a good tradeoff for map/reduce.
The information _is_ available in general, but for near-certainty in this
hypothetical situation one needs to examine task logs. That it's easier to
collect from the TT logs isn't a good enough reason to impose this on
maintainers; expecting them to find its purpose "documented" in the bug
tracking system and retain the property in support of possible downstream
processing is not reasonable.
> Add reduce ID to shuffle clienttrace
> ------------------------------------
>
> Key: MAPREDUCE-479
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-479
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 0.21.0
> Reporter: Jiaqi Tan
> Assignee: Jiaqi Tan
> Priority: Minor
> Fix For: 0.21.0
>
> Attachments: HADOOP-6013.patch, MAPREDUCE-479-1.patch,
> MAPREDUCE-479-2.patch, MAPREDUCE-479-3.patch, MAPREDUCE-479.patch
>
>
> Current clienttrace messages from shuffles note only the destination map ID
> but not the source reduce ID. Having both source and destination ID of each
> shuffle enables full tracing of execution.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
