[
https://issues.apache.org/jira/browse/HADOOP-3514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12624036#action_12624036
]
Arun C Murthy commented on HADOOP-3514:
---------------------------------------
Fair enough.
I'd be happy with removing the IOUtils.readFully in ChecksumInputStream.read
for the data - I can understand a IOUtils.readFully for the checksum, but the
ChecksumInputStream should just pass along the data read from the underlying
stream, else it might interact badly with buffering done by the underlying
stream e.g. IFile.Reader tries very carefully to read large chunks from the
filesystem - we shouldn't force a loop on it from ChecksumInputStream.
> Reduce seeks during shuffle, by inline crcs
> -------------------------------------------
>
> Key: HADOOP-3514
> URL: https://issues.apache.org/jira/browse/HADOOP-3514
> Project: Hadoop Core
> Issue Type: Improvement
> Components: mapred
> Affects Versions: 0.18.0
> Reporter: Devaraj Das
> Assignee: Jothi Padmanabhan
> Fix For: 0.19.0
>
> Attachments: hadoop-3514-v1.patch, hadoop-3514-v2.patch,
> hadoop-3514-v3.patch, hadoop-3514-v4.patch, hadoop-3514-v5.patch,
> hadoop-3514-v6.patch, hadoop-3514-v7.patch, hadoop-3514-v8.patch,
> hadoop-3514.patch
>
>
> The number of seeks can be reduced by half in the iFile if we move the crc
> into the iFile rather than having a separate file.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
