[
https://issues.apache.org/jira/browse/MAPREDUCE-7402?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17825614#comment-17825614
]
ASF GitHub Bot commented on MAPREDUCE-7402:
-------------------------------------------
KeeProMise commented on code in PR #6622:
URL: https://github.com/apache/hadoop/pull/6622#discussion_r1521225842
##########
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestMerger.java:
##########
@@ -229,6 +232,14 @@ public void testInMemoryAndOnDiskMerger() throws Throwable
{
Assert.assertEquals(0, mergeManager.inMemoryMapOutputs.size());
Assert.assertEquals(0, mergeManager.inMemoryMergedMapOutputs.size());
Assert.assertEquals(0, mergeManager.onDiskMapOutputs.size());
+
+ jobConf.set("mapreduce.task.io.sort.factor", "1");
+ thrown.expectMessage("Invalid value for mapreduce.task.io.sort.factor: 1,"
+
+ " please set it to a number greater than 1");
+ thrown.expect(IllegalArgumentException.class);
+ new MergeManagerImpl<Text, Text>(reduceId2, jobConf, fs, lda,
Reporter.NULL, null,
+ null, null, null, null,
+ null, null, new Progress(), new MROutputFiles());
Review Comment:
done
##########
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/test/java/org/apache/hadoop/mapreduce/task/reduce/TestMerger.java:
##########
@@ -229,6 +232,14 @@ public void testInMemoryAndOnDiskMerger() throws Throwable
{
Assert.assertEquals(0, mergeManager.inMemoryMapOutputs.size());
Assert.assertEquals(0, mergeManager.inMemoryMergedMapOutputs.size());
Assert.assertEquals(0, mergeManager.onDiskMapOutputs.size());
+
+ jobConf.set("mapreduce.task.io.sort.factor", "1");
Review Comment:
done
> fix mapreduce.task.io.sort.factor=1 lead to an infinite loop.
> -------------------------------------------------------------
>
> Key: MAPREDUCE-7402
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7402
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Jian Zhang
> Priority: Minor
> Labels: pull-request-available
> Attachments: screenshot-1.png
>
>
> *Describe：*
> In the method long computeBytesInMerges(int factor, int inMem), when
> factor is set to 1, and the initial numSegments>1, inMem=0, it will cause an
> infinite loop of while. Although setting factor to 1 is meaningless, and it
> will not be done in actual production, we need to remind users to avoid an
> infinite loop due to carelessness.
> !screenshot-1.png|width=600,height=400!
> *How to reproduce:*
> This bug can easily occur by setting
> configuration.set("mapreduce.task.io.sort.factor", "1").
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
