[ https://issues.apache.org/jira/browse/SPARK-20943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16034746#comment-16034746 ]
CanBin Zheng commented on SPARK-20943: -------------------------------------- [~saisai_shao] I got you. But I think it's better to change the description, it has confused me for a long time, maybe someone else has the same puzzle. > Correct BypassMergeSortShuffleWriter's comment > ---------------------------------------------- > > Key: SPARK-20943 > URL: https://issues.apache.org/jira/browse/SPARK-20943 > Project: Spark > Issue Type: Improvement > Components: Documentation, Shuffle > Affects Versions: 2.1.1 > Reporter: CanBin Zheng > Priority: Trivial > Labels: starter > > There are some comments written in BypassMergeSortShuffleWriter.java about > when to select this write path, the three required conditions are described > as follows: > 1. no Ordering is specified, and > 2. no Aggregator is specified, and > 3. the number of partitions is less than > spark.shuffle.sort.bypassMergeThreshold > Obviously, the conditions written are partially wrong and misleading, the > right conditions should be: > 1. map-side combine is false, and > 2. the number of partitions is less than > spark.shuffle.sort.bypassMergeThreshold -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org