[
https://issues.apache.org/jira/browse/TEZ-3237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15281573#comment-15281573
]
Tsuyoshi Ozawa commented on TEZ-3237:
-------------------------------------
[~rajesh.balamohan] make sense to me.
> Corrupted shuffle transfers to disk are not detected during transfer
> --------------------------------------------------------------------
>
> Key: TEZ-3237
> URL: https://issues.apache.org/jira/browse/TEZ-3237
> Project: Apache Tez
> Issue Type: Bug
> Affects Versions: 0.7.0
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Attachments: TEZ-3237.001.patch
>
>
> When a shuffle transfer is larger than the single transfer limit it gets
> written straight to disk during the transfer. Unfortunately there are no
> checksum validations performed during that transfer, so if the data is
> corrupted at the source or during transmit it goes undetected. Only later
> when the task tries to consume the transferred data is the error detected,
> but at that point it's too late to blame the source task for the error.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
