[
https://issues.apache.org/jira/browse/HADOOP-3131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12615720#action_12615720
]
Matei Zaharia commented on HADOOP-3131:
---------------------------------------
I'm not sure it's worth fixing that, because we don't need perfect progress
reporting, just a rough guide to tell whether a task is doing something, and
what rate it's working at. With compression enabled, it would also be
difficult to figure out which spot in the buffer corresponds to which byte in
the compressed file, if we were to use the position in the buffer to figure out
progress.
> enabling BLOCK compression for map outputs breaks the reduce progress counters
> ------------------------------------------------------------------------------
>
> Key: HADOOP-3131
> URL: https://issues.apache.org/jira/browse/HADOOP-3131
> Project: Hadoop Core
> Issue Type: Bug
> Affects Versions: 0.17.0, 0.17.1, 0.17.2, 0.18.0, 0.19.0
> Reporter: Colin Evans
> Assignee: Matei Zaharia
> Fix For: 0.19.0
>
> Attachments: HADOOP-3131-v2.patch, HADOOP-3131-v3.patch,
> HADOOP-3131-v4.patch, HADOOP-3131-v5.patch, merge-progress-trunk.patch,
> merge-progress.patch, Picture 1.png
>
>
> Enabling map output compression and setting the compression type to BLOCK
> causes the progress counters during the reduce to go crazy and report
> progress counts over 100%.
> This is problematic for speculative execution because it thinks the tasks are
> doing fine.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
