[jira] [Commented] (MAPREDUCE-6063) In sortAndSpill of MapTask.java, size is calculated wrongly when bufend bufstart.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121254#comment-14121254 ] Hudson commented on MAPREDUCE-6063: --- FAILURE: Integrated in Hadoop-Yarn-trunk #670 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/670/]) MAPREDUCE-6063. Correct spill size calculation for spills wrapping the circular buffer. Contributed by zhihai xu. (cdouglas: rev a18424e724982d01e0191ca7db713da1d43d1e4d) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java In sortAndSpill of MapTask.java, size is calculated wrongly when bufend bufstart. --- Key: MAPREDUCE-6063 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6063 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1, mrv2 Reporter: zhihai xu Assignee: zhihai xu Fix For: 3.0.0, 2.6.0 Attachments: MAPREDUCE-6063.000.patch, MAPREDUCE-6063.branch-1.patch In sortAndSpill of MapTask.java, size is calculated wrongly when bufend bufstart. we should change (bufvoid - bufend) + bufstart to (bufvoid - bufstart) + bufend. Should change {code} long size = (bufend = bufstart ? bufend - bufstart : (bufvoid - bufend) + bufstart) + partitions * APPROX_HEADER_LENGTH; {code} to: {code} long size = (bufend = bufstart ? bufend - bufstart : (bufvoid - bufstart) + bufend) + partitions * APPROX_HEADER_LENGTH; {code} It is because when wraparound happen (bufend bufstart) , the size should bufvoid - bufstart (bigger one) + bufend(small one). You can find similar code implementation in MapTask.java: {code} mapOutputByteCounter.increment(valend = keystart ? valend - keystart : (bufvoid - keystart) + valend); {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6063) In sortAndSpill of MapTask.java, size is calculated wrongly when bufend bufstart.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121356#comment-14121356 ] Hudson commented on MAPREDUCE-6063: --- FAILURE: Integrated in Hadoop-Hdfs-trunk #1861 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1861/]) MAPREDUCE-6063. Correct spill size calculation for spills wrapping the circular buffer. Contributed by zhihai xu. (cdouglas: rev a18424e724982d01e0191ca7db713da1d43d1e4d) * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java * hadoop-mapreduce-project/CHANGES.txt In sortAndSpill of MapTask.java, size is calculated wrongly when bufend bufstart. --- Key: MAPREDUCE-6063 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6063 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1, mrv2 Reporter: zhihai xu Assignee: zhihai xu Fix For: 3.0.0, 2.6.0 Attachments: MAPREDUCE-6063.000.patch, MAPREDUCE-6063.branch-1.patch In sortAndSpill of MapTask.java, size is calculated wrongly when bufend bufstart. we should change (bufvoid - bufend) + bufstart to (bufvoid - bufstart) + bufend. Should change {code} long size = (bufend = bufstart ? bufend - bufstart : (bufvoid - bufend) + bufstart) + partitions * APPROX_HEADER_LENGTH; {code} to: {code} long size = (bufend = bufstart ? bufend - bufstart : (bufvoid - bufstart) + bufend) + partitions * APPROX_HEADER_LENGTH; {code} It is because when wraparound happen (bufend bufstart) , the size should bufvoid - bufstart (bigger one) + bufend(small one). You can find similar code implementation in MapTask.java: {code} mapOutputByteCounter.increment(valend = keystart ? valend - keystart : (bufvoid - keystart) + valend); {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6063) In sortAndSpill of MapTask.java, size is calculated wrongly when bufend bufstart.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121540#comment-14121540 ] Hudson commented on MAPREDUCE-6063: --- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1886 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1886/]) MAPREDUCE-6063. Correct spill size calculation for spills wrapping the circular buffer. Contributed by zhihai xu. (cdouglas: rev a18424e724982d01e0191ca7db713da1d43d1e4d) * hadoop-mapreduce-project/CHANGES.txt * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/MapTask.java In sortAndSpill of MapTask.java, size is calculated wrongly when bufend bufstart. --- Key: MAPREDUCE-6063 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6063 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1, mrv2 Reporter: zhihai xu Assignee: zhihai xu Fix For: 3.0.0, 2.6.0 Attachments: MAPREDUCE-6063.000.patch, MAPREDUCE-6063.branch-1.patch In sortAndSpill of MapTask.java, size is calculated wrongly when bufend bufstart. we should change (bufvoid - bufend) + bufstart to (bufvoid - bufstart) + bufend. Should change {code} long size = (bufend = bufstart ? bufend - bufstart : (bufvoid - bufend) + bufstart) + partitions * APPROX_HEADER_LENGTH; {code} to: {code} long size = (bufend = bufstart ? bufend - bufstart : (bufvoid - bufstart) + bufend) + partitions * APPROX_HEADER_LENGTH; {code} It is because when wraparound happen (bufend bufstart) , the size should bufvoid - bufstart (bigger one) + bufend(small one). You can find similar code implementation in MapTask.java: {code} mapOutputByteCounter.increment(valend = keystart ? valend - keystart : (bufvoid - keystart) + valend); {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6063) In sortAndSpill of MapTask.java, size is calculated wrongly when bufend bufstart.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14121968#comment-14121968 ] Chris Douglas commented on MAPREDUCE-6063: -- bq. This issue is also in MR1 (branch-1) . I attached a patch MAPREDUCE-6063.branch-1.patch for branch-1. Done. Thanks again for the fix In sortAndSpill of MapTask.java, size is calculated wrongly when bufend bufstart. --- Key: MAPREDUCE-6063 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6063 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1, mrv2 Reporter: zhihai xu Assignee: zhihai xu Fix For: 3.0.0, 2.6.0 Attachments: MAPREDUCE-6063.000.patch, MAPREDUCE-6063.branch-1.patch In sortAndSpill of MapTask.java, size is calculated wrongly when bufend bufstart. we should change (bufvoid - bufend) + bufstart to (bufvoid - bufstart) + bufend. Should change {code} long size = (bufend = bufstart ? bufend - bufstart : (bufvoid - bufend) + bufstart) + partitions * APPROX_HEADER_LENGTH; {code} to: {code} long size = (bufend = bufstart ? bufend - bufstart : (bufvoid - bufstart) + bufend) + partitions * APPROX_HEADER_LENGTH; {code} It is because when wraparound happen (bufend bufstart) , the size should bufvoid - bufstart (bigger one) + bufend(small one). You can find similar code implementation in MapTask.java: {code} mapOutputByteCounter.increment(valend = keystart ? valend - keystart : (bufvoid - keystart) + valend); {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6063) In sortAndSpill of MapTask.java, size is calculated wrongly when bufend bufstart.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14122327#comment-14122327 ] zhihai xu commented on MAPREDUCE-6063: -- Hi [~chris.douglas], thanks to commit this patch to branch-1, it is very nice working with you. In sortAndSpill of MapTask.java, size is calculated wrongly when bufend bufstart. --- Key: MAPREDUCE-6063 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6063 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1, mrv2 Reporter: zhihai xu Assignee: zhihai xu Fix For: 3.0.0, 2.6.0 Attachments: MAPREDUCE-6063.000.patch, MAPREDUCE-6063.branch-1.patch In sortAndSpill of MapTask.java, size is calculated wrongly when bufend bufstart. we should change (bufvoid - bufend) + bufstart to (bufvoid - bufstart) + bufend. Should change {code} long size = (bufend = bufstart ? bufend - bufstart : (bufvoid - bufend) + bufstart) + partitions * APPROX_HEADER_LENGTH; {code} to: {code} long size = (bufend = bufstart ? bufend - bufstart : (bufvoid - bufstart) + bufend) + partitions * APPROX_HEADER_LENGTH; {code} It is because when wraparound happen (bufend bufstart) , the size should bufvoid - bufstart (bigger one) + bufend(small one). You can find similar code implementation in MapTask.java: {code} mapOutputByteCounter.increment(valend = keystart ? valend - keystart : (bufvoid - keystart) + valend); {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6063) In sortAndSpill of MapTask.java, size is calculated wrongly when bufend bufstart.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14120902#comment-14120902 ] zhihai xu commented on MAPREDUCE-6063: -- Hi [~chris.douglas], thanks to commit the patch to trunk and branch-2. This issue is also in MR1 (branch-1) . I attached a patch MAPREDUCE-6063.branch-1.patch for branch-1. Could you commit MAPREDUCE-6063.branch-1.patch to branch-1? thanks zhihai In sortAndSpill of MapTask.java, size is calculated wrongly when bufend bufstart. --- Key: MAPREDUCE-6063 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6063 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1, mrv2 Reporter: zhihai xu Assignee: zhihai xu Fix For: 3.0.0, 2.6.0 Attachments: MAPREDUCE-6063.000.patch, MAPREDUCE-6063.branch-1.patch In sortAndSpill of MapTask.java, size is calculated wrongly when bufend bufstart. we should change (bufvoid - bufend) + bufstart to (bufvoid - bufstart) + bufend. Should change {code} long size = (bufend = bufstart ? bufend - bufstart : (bufvoid - bufend) + bufstart) + partitions * APPROX_HEADER_LENGTH; {code} to: {code} long size = (bufend = bufstart ? bufend - bufstart : (bufvoid - bufstart) + bufend) + partitions * APPROX_HEADER_LENGTH; {code} It is because when wraparound happen (bufend bufstart) , the size should bufvoid - bufstart (bigger one) + bufend(small one). You can find similar code implementation in MapTask.java: {code} mapOutputByteCounter.increment(valend = keystart ? valend - keystart : (bufvoid - keystart) + valend); {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6063) In sortAndSpill of MapTask.java, size is calculated wrongly when bufend bufstart.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14118619#comment-14118619 ] Hadoop QA commented on MAPREDUCE-6063: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12665960/MAPREDUCE-6063.000.patch against trunk revision 6595e92. {color:red}-1 patch{color}. Trunk compilation may be broken. Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4840//console This message is automatically generated. In sortAndSpill of MapTask.java, size is calculated wrongly when bufend bufstart. --- Key: MAPREDUCE-6063 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6063 Project: Hadoop Map/Reduce Issue Type: Bug Components: mrv1, mrv2 Reporter: zhihai xu Assignee: zhihai xu Attachments: MAPREDUCE-6063.000.patch In sortAndSpill of MapTask.java, size is calculated wrongly when bufend bufstart. we should change (bufvoid - bufend) + bufstart to (bufvoid - bufstart) + bufend. Should change {code} long size = (bufend = bufstart ? bufend - bufstart : (bufvoid - bufend) + bufstart) + partitions * APPROX_HEADER_LENGTH; {code} to: {code} long size = (bufend = bufstart ? bufend - bufstart : (bufvoid - bufstart) + bufend) + partitions * APPROX_HEADER_LENGTH; {code} It is because when wraparound happen (bufend bufstart) , the size should bufvoid - bufstart (bigger one) + bufend(small one). You can find similar code implementation in MapTask.java: {code} mapOutputByteCounter.increment(valend = keystart ? valend - keystart : (bufvoid - keystart) + valend); {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)