[
https://issues.apache.org/jira/browse/SPARK-5081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14324137#comment-14324137
]
Dr. Christian Betz commented on SPARK-5081:
-------------------------------------------
And that's the diff from Spark-1.1.0-CDH to Spark 1.1.0 with Hadoop-2.5.0:
diff Spark-1.1.0-Hadoop-2.5.0.txt Spark-1.1.0-CDH5.2.0.txt
1d0
< /asm/asm/3.1/asm-3.1.jar
22a22
> /com/google/code/gson/gson/2.2.4/gson-2.2.4.jar
24a25
> /com/jamesmurty/utils/java-xmlbuilder/0.4/java-xmlbuilder-0.4.jar
28d28
< /com/sun/jersey/jersey-server/1.9/jersey-server-1.9.jar
40d39
< /commons-daemon/commons-daemon/1.0.13/commons-daemon-1.0.13.jar
42d40
< /commons-el/commons-el/1.0/commons-el-1.0.jar
55d52
< /javax/servlet/jsp/jsp-api/2.1/jsp-api-2.1.jar
62c59
< /net/java/dev/jets3t/jets3t/0.7.1/jets3t-0.7.1.jar
---
> /net/java/dev/jets3t/jets3t/0.9.0/jets3t-0.9.0.jar
72c69
< /org/apache/curator/curator-client/2.4.0/curator-client-2.4.0.jar
---
> /org/apache/curator/curator-client/2.6.0/curator-client-2.6.0.jar
79,94c76,91
< /org/apache/hadoop/hadoop-annotations/2.5.0/hadoop-annotations-2.5.0.jar
< /org/apache/hadoop/hadoop-auth/2.5.0/hadoop-auth-2.5.0.jar
< /org/apache/hadoop/hadoop-client/2.5.0/hadoop-client-2.5.0.jar
< /org/apache/hadoop/hadoop-common/2.5.0/hadoop-common-2.5.0.jar
< /org/apache/hadoop/hadoop-hdfs/2.5.0/hadoop-hdfs-2.5.0.jar
<
/org/apache/hadoop/hadoop-mapreduce-client-app/2.5.0/hadoop-mapreduce-client-app-2.5.0.jar
<
/org/apache/hadoop/hadoop-mapreduce-client-common/2.5.0/hadoop-mapreduce-client-common-2.5.0.jar
<
/org/apache/hadoop/hadoop-mapreduce-client-core/2.5.0/hadoop-mapreduce-client-core-2.5.0.jar
<
/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.5.0/hadoop-mapreduce-client-jobclient-2.5.0.jar
<
/org/apache/hadoop/hadoop-mapreduce-client-shuffle/2.5.0/hadoop-mapreduce-client-shuffle-2.5.0.jar
< /org/apache/hadoop/hadoop-yarn-api/2.5.0/hadoop-yarn-api-2.5.0.jar
< /org/apache/hadoop/hadoop-yarn-client/2.5.0/hadoop-yarn-client-2.5.0.jar
< /org/apache/hadoop/hadoop-yarn-common/2.5.0/hadoop-yarn-common-2.5.0.jar
<
/org/apache/hadoop/hadoop-yarn-server-common/2.5.0/hadoop-yarn-server-common-2.5.0.jar
< /org/apache/httpcomponents/httpclient/4.2.5/httpclient-4.2.5.jar
< /org/apache/httpcomponents/httpcore/4.2.4/httpcore-4.2.4.jar
---
> /org/apache/hadoop/hadoop-annotations/2.5.0-cdh5.2.0/hadoop-annotations-2.5.0-cdh5.2.0.jar
> /org/apache/hadoop/hadoop-auth/2.5.0-cdh5.2.0/hadoop-auth-2.5.0-cdh5.2.0.jar
> /org/apache/hadoop/hadoop-client/2.5.0-cdh5.2.0/hadoop-client-2.5.0-cdh5.2.0.jar
> /org/apache/hadoop/hadoop-common/2.5.0-cdh5.2.0/hadoop-common-2.5.0-cdh5.2.0.jar
> /org/apache/hadoop/hadoop-hdfs/2.5.0-cdh5.2.0/hadoop-hdfs-2.5.0-cdh5.2.0.jar
> /org/apache/hadoop/hadoop-mapreduce-client-app/2.5.0-cdh5.2.0/hadoop-mapreduce-client-app-2.5.0-cdh5.2.0.jar
> /org/apache/hadoop/hadoop-mapreduce-client-common/2.5.0-cdh5.2.0/hadoop-mapreduce-client-common-2.5.0-cdh5.2.0.jar
> /org/apache/hadoop/hadoop-mapreduce-client-core/2.5.0-cdh5.2.0/hadoop-mapreduce-client-core-2.5.0-cdh5.2.0.jar
> /org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.5.0-cdh5.2.0/hadoop-mapreduce-client-jobclient-2.5.0-cdh5.2.0.jar
> /org/apache/hadoop/hadoop-mapreduce-client-shuffle/2.5.0-cdh5.2.0/hadoop-mapreduce-client-shuffle-2.5.0-cdh5.2.0.jar
> /org/apache/hadoop/hadoop-yarn-api/2.5.0-cdh5.2.0/hadoop-yarn-api-2.5.0-cdh5.2.0.jar
> /org/apache/hadoop/hadoop-yarn-client/2.5.0-cdh5.2.0/hadoop-yarn-client-2.5.0-cdh5.2.0.jar
> /org/apache/hadoop/hadoop-yarn-common/2.5.0-cdh5.2.0/hadoop-yarn-common-2.5.0-cdh5.2.0.jar
> /org/apache/hadoop/hadoop-yarn-server-common/2.5.0-cdh5.2.0/hadoop-yarn-server-common-2.5.0-cdh5.2.0.jar
> /org/apache/httpcomponents/httpclient/4.1.2/httpclient-4.1.2.jar
> /org/apache/httpcomponents/httpcore/4.1.2/httpcore-4.1.2.jar
96c93
< /org/apache/spark/spark-core_2.10/1.1.0/spark-core_2.10-1.1.0.jar
---
> /org/apache/spark/spark-core_2.10/1.1.0-cdh5.2.0/spark-core_2.10-1.1.0-cdh5.2.0.jar
111,114c108,111
< /org/codehaus/jackson/jackson-core-asl/1.9.13/jackson-core-asl-1.9.13.jar
< /org/codehaus/jackson/jackson-jaxrs/1.9.13/jackson-jaxrs-1.9.13.jar
< /org/codehaus/jackson/jackson-mapper-asl/1.9.13/jackson-mapper-asl-1.9.13.jar
< /org/codehaus/jackson/jackson-xc/1.9.13/jackson-xc-1.9.13.jar
---
> /org/codehaus/jackson/jackson-core-asl/1.8.8/jackson-core-asl-1.8.8.jar
> /org/codehaus/jackson/jackson-jaxrs/1.8.8/jackson-jaxrs-1.8.8.jar
> /org/codehaus/jackson/jackson-mapper-asl/1.8.8/jackson-mapper-asl-1.8.8.jar
> /org/codehaus/jackson/jackson-xc/1.8.8/jackson-xc-1.8.8.jar
157d153
< /tomcat/jasper-runtime/5.5.23/jasper-runtime-5.5.23.jar
> Shuffle write increases
> -----------------------
>
> Key: SPARK-5081
> URL: https://issues.apache.org/jira/browse/SPARK-5081
> Project: Spark
> Issue Type: Bug
> Components: Shuffle
> Affects Versions: 1.2.0
> Reporter: Kevin Jung
> Priority: Critical
> Attachments: Spark_Debug.pdf
>
>
> The size of shuffle write showing in spark web UI is much different when I
> execute same spark job with same input data in both spark 1.1 and spark 1.2.
> At sortBy stage, the size of shuffle write is 98.1MB in spark 1.1 but 146.9MB
> in spark 1.2.
> I set spark.shuffle.manager option to hash because it's default value is
> changed but spark 1.2 still writes shuffle output more than spark 1.1.
> It can increase disk I/O overhead exponentially as the input file gets bigger
> and it causes the jobs take more time to complete.
> In the case of about 100GB input, for example, the size of shuffle write is
> 39.7GB in spark 1.1 but 91.0GB in spark 1.2.
> spark 1.1
> ||Stage Id||Description||Input||Shuffle Read||Shuffle Write||
> |9|saveAsTextFile| |1169.4KB| |
> |12|combineByKey| |1265.4KB|1275.0KB|
> |6|sortByKey| |1276.5KB| |
> |8|mapPartitions| |91.0MB|1383.1KB|
> |4|apply| |89.4MB| |
> |5|sortBy|155.6MB| |98.1MB|
> |3|sortBy|155.6MB| | |
> |1|collect| |2.1MB| |
> |2|mapValues|155.6MB| |2.2MB|
> |0|first|184.4KB| | |
> spark 1.2
> ||Stage Id||Description||Input||Shuffle Read||Shuffle Write||
> |12|saveAsTextFile| |1170.2KB| |
> |11|combineByKey| |1264.5KB|1275.0KB|
> |8|sortByKey| |1273.6KB| |
> |7|mapPartitions| |134.5MB|1383.1KB|
> |5|zipWithIndex| |132.5MB| |
> |4|sortBy|155.6MB| |146.9MB|
> |3|sortBy|155.6MB| | |
> |2|collect| |2.0MB| |
> |1|mapValues|155.6MB| |2.2MB|
> |0|first|184.4KB| | |
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
