[
https://issues.apache.org/jira/browse/MAPREDUCE-4882?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524601#comment-14524601
]
Hadoop QA commented on MAPREDUCE-4882:
--------------------------------------
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | patch | 0m 0s | The patch command could not apply
the patch during dryrun. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12566626/MAPREDUCE-4882.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / f1a152c |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5507/console |
This message was automatically generated.
> Error in estimating the length of the output file in Spill Phase
> ----------------------------------------------------------------
>
> Key: MAPREDUCE-4882
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4882
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Affects Versions: 0.20.2, 1.0.3
> Environment: Any Environment
> Reporter: Lijie Xu
> Assignee: Jerry Chen
> Labels: patch
> Attachments: MAPREDUCE-4882.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> The sortAndSpill() method in MapTask.java has an error in estimating the
> length of the output file.
> The "long size" should be "(bufvoid - bufstart) + bufend" not "(bufvoid -
> bufend) + bufstart" when "bufend < bufstart".
> Here is the original code in MapTask.java.
> private void sortAndSpill() throws IOException, ClassNotFoundException,
> InterruptedException {
> //approximate the length of the output file to be the length of the
> //buffer + header lengths for the partitions
> long size = (bufend >= bufstart
> ? bufend - bufstart
> : (bufvoid - bufend) + bufstart) +
> partitions * APPROX_HEADER_LENGTH;
> FSDataOutputStream out = null;
> ------------------------------------------------------------------------------
> I had a test on "TeraSort". A snippet from mapper's log is as follows:
> MapTask: Spilling map output: record full = true
> MapTask: bufstart = 157286200; bufend = 10485460; bufvoid = 199229440
> MapTask: kvstart = 262142; kvend = 131069; length = 655360
> MapTask: Finished spill 3
> In this occasioin, Spill Bytes should be (199229440 - 157286200) + 10485460 =
> 52428700 (52 MB) because the number of spilled records is 524287 and each
> record costs 100B.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
