[
https://issues.apache.org/jira/browse/SPARK-5081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14541576#comment-14541576
]
Dr. Christian Betz commented on SPARK-5081:
-------------------------------------------
Hi,
I can now say that this was not a fix!
Here's a single task from that count stage:
||Index ▾||ID||Attempt||Status||Locality Level||Executor ID / Host||Launch
Time||Duration||GC Time||Input Size / Records||Shuffle Read Size /
Records||Shuffle Spill (Memory)||Shuffle Spill (Disk)||Errors||
|11|128|0|SUCCESS|PROCESS_LOCAL|3 / marvin.pmd.local|2015/05/13 09:55:08|3.9
min|15 s|352.3 KB (memory) / 1171|10.0 MB / 854977|2.1 GB|113.5 MB|
As you can see, for an input-size+shuffle-read-size of approx 10.0 MByte, it
produces a ShuffleSpill to Memory of 2.1 GByte, to disk 114 MByte.
Here's the debugString of the RDD that I'm counting in this task:
{noformat}
(20) MapPartitionsRDD[64] at mapValues at NativeMethodAccessorImpl.java:-2
[Disk Memory Deserialized 1x Replicated]
| CachedPartitions: 20; MemorySize: 181.7 MB; TachyonSize: 0.0 B;
DiskSize: 360.7 MB
| MapPartitionsRDD[63] at mapPartitionsToPair at
NativeMethodAccessorImpl.java:-2 [Disk Memory Deserialized 1x Replicated]
| MapPartitionsRDD[62] at mapValues at NativeMethodAccessorImpl.java:-2
[Disk Memory Deserialized 1x Replicated]
| MapPartitionsRDD[61] at leftOuterJoin at NativeMethodAccessorImpl.java:-2
[Disk Memory Deserialized 1x Replicated]
| MapPartitionsRDD[60] at leftOuterJoin at NativeMethodAccessorImpl.java:-2
[Disk Memory Deserialized 1x Replicated]
| MapPartitionsRDD[59] at leftOuterJoin at NativeMethodAccessorImpl.java:-2
[Disk Memory Deserialized 1x Replicated]
| CoGroupedRDD[58] at leftOuterJoin at NativeMethodAccessorImpl.java:-2
[Disk Memory Deserialized 1x Replicated]
| MapPartitionsRDD[57] at mapPartitionsToPair at
NativeMethodAccessorImpl.java:-2 [Disk Memory Deserialized 1x Replicated]
| MapPartitionsRDD[56] at mapPartitionsToPair at
NativeMethodAccessorImpl.java:-2 [Disk Memory Deserialized 1x Replicated]
| MapPartitionsRDD[55] at mapValues at NativeMethodAccessorImpl.java:-2
[Disk Memory Deserialized 1x Replicated]
| MapPartitionsRDD[54] at filter at NativeMethodAccessorImpl.java:-2 [Disk
Memory Deserialized 1x Replicated]
| MapPartitionsRDD[53] at join at NativeMethodAccessorImpl.java:-2 [Disk
Memory Deserialized 1x Replicated]
| MapPartitionsRDD[52] at join at NativeMethodAccessorImpl.java:-2 [Disk
Memory Deserialized 1x Replicated]
| CoGroupedRDD[51] at join at NativeMethodAccessorImpl.java:-2 [Disk Memory
Deserialized 1x Replicated]
| MapPartitionsRDD[50] at mapPartitionsToPair at
NativeMethodAccessorImpl.java:-2 [Disk Memory Deserialized 1x Replicated]
| MapPartitionsRDD[49] at mapValues at NativeMethodAccessorImpl.java:-2
[Disk Memory Deserialized 1x Replicated]
| MapPartitionsRDD[48] at cogroup at core.clj:310 [Disk Memory Deserialized
1x Replicated]
| MapPartitionsRDD[47] at cogroup at core.clj:310 [Disk Memory Deserialized
1x Replicated]
| CoGroupedRDD[46] at cogroup at core.clj:310 [Disk Memory Deserialized 1x
Replicated]
| PartitionerAwareUnionRDD[45] at PartitionerAwareUnionRDD at core.clj:326
[Disk Memory Deserialized 1x Replicated]
| MapPartitionsRDD[42] at filter at NativeMethodAccessorImpl.java:-2 [Disk
Memory Deserialized 1x Replicated]
| MapPartitionsRDD[41] at mapValues at NativeMethodAccessorImpl.java:-2
[Disk Memory Deserialized 1x Replicated]
| ShuffledRDD[31] at partitionBy at core.clj:489 [Disk Memory Deserialized
1x Replicated]
+-(31) MapPartitionsRDD[30] at mapToPair at NativeMethodAccessorImpl.java:-2
[Disk Memory Deserialized 1x Replicated]
| CoalescedRDD[29] at coalesce at NativeMethodAccessorImpl.java:-2 [Disk
Memory Deserialized 1x Replicated]
| MapPartitionsRDD[28] at filter at NativeMethodAccessorImpl.java:-2
[Disk Memory Deserialized 1x Replicated]
| MapPartitionsRDD[27] at map at NativeMethodAccessorImpl.java:-2 [Disk
Memory Deserialized 1x Replicated]
| hdfs://host/path.avro NewHadoopRDD[26] at newAPIHadoopFile at
hadoopAvro.clj:24 [Disk Memory Deserialized 1x Replicated]
| MapPartitionsRDD[44] at filter at NativeMethodAccessorImpl.java:-2 [Disk
Memory Deserialized 1x Replicated]
| MapPartitionsRDD[43] at mapValues at NativeMethodAccessorImpl.java:-2
[Disk Memory Deserialized 1x Replicated]
| ShuffledRDD[40] at partitionBy at core.clj:489 [Disk Memory Deserialized
1x Replicated]
+-(10) MapPartitionsRDD[39] at mapToPair at NativeMethodAccessorImpl.java:-2
[Disk Memory Deserialized 1x Replicated]
| JdbcRDD[38] at JdbcRDD at jdbc.clj:64 [Disk Memory Deserialized 1x
Replicated]
| ShuffledRDD[34] at partitionBy at core.clj:489 [Disk Memory Deserialized
1x Replicated]
+-(10) MapPartitionsRDD[33] at mapToPair at NativeMethodAccessorImpl.java:-2
[Disk Memory Deserialized 1x Replicated]
| JdbcRDD[32] at JdbcRDD at jdbc.clj:64 [Disk Memory Deserialized 1x
Replicated]
| ShuffledRDD[37] at partitionBy at core.clj:489 [Disk Memory Deserialized
1x Replicated]
+-(10) MapPartitionsRDD[36] at mapToPair at NativeMethodAccessorImpl.java:-2
[Disk Memory Deserialized 1x Replicated]
| JdbcRDD[35] at JdbcRDD at jdbc.clj:64 [Disk Memory Deserialized 1x
Replicated]
| ShuffledRDD[3] at partitionBy at core.clj:489 [Disk Memory Deserialized 1x
Replicated]
| CachedPartitions: 20; MemorySize: 657.0 KB; TachyonSize: 0.0 B;
DiskSize: 50.0 KB
+-(1) MapPartitionsRDD[2] at mapToPair at NativeMethodAccessorImpl.java:-2
[Disk Memory Deserialized 1x Replicated]
| MapPartitionsRDD[1] at map at NativeMethodAccessorImpl.java:-2 [Disk
Memory Deserialized 1x Replicated]
| JdbcRDD[0] at JdbcRDD at jdbc.clj:64 [Disk Memory Deserialized 1x
Replicated]
| ShuffledRDD[6] at partitionBy at core.clj:489 [Disk Memory Deserialized 1x
Replicated]
| CachedPartitions: 20; MemorySize: 5.1 MB; TachyonSize: 0.0 B;
DiskSize: 0.0 B
+-(1) MapPartitionsRDD[5] at mapToPair at NativeMethodAccessorImpl.java:-2
[Disk Memory Deserialized 1x Replicated]
| JdbcRDD[4] at JdbcRDD at jdbc.clj:64 [Disk Memory Deserialized 1x
Replicated]
{noformat}
> Shuffle write increases
> -----------------------
>
> Key: SPARK-5081
> URL: https://issues.apache.org/jira/browse/SPARK-5081
> Project: Spark
> Issue Type: Bug
> Components: Shuffle
> Affects Versions: 1.2.0
> Reporter: Kevin Jung
> Priority: Critical
> Attachments: Spark_Debug.pdf, diff.txt
>
>
> The size of shuffle write showing in spark web UI is much different when I
> execute same spark job with same input data in both spark 1.1 and spark 1.2.
> At sortBy stage, the size of shuffle write is 98.1MB in spark 1.1 but 146.9MB
> in spark 1.2.
> I set spark.shuffle.manager option to hash because it's default value is
> changed but spark 1.2 still writes shuffle output more than spark 1.1.
> It can increase disk I/O overhead exponentially as the input file gets bigger
> and it causes the jobs take more time to complete.
> In the case of about 100GB input, for example, the size of shuffle write is
> 39.7GB in spark 1.1 but 91.0GB in spark 1.2.
> spark 1.1
> ||Stage Id||Description||Input||Shuffle Read||Shuffle Write||
> |9|saveAsTextFile| |1169.4KB| |
> |12|combineByKey| |1265.4KB|1275.0KB|
> |6|sortByKey| |1276.5KB| |
> |8|mapPartitions| |91.0MB|1383.1KB|
> |4|apply| |89.4MB| |
> |5|sortBy|155.6MB| |98.1MB|
> |3|sortBy|155.6MB| | |
> |1|collect| |2.1MB| |
> |2|mapValues|155.6MB| |2.2MB|
> |0|first|184.4KB| | |
> spark 1.2
> ||Stage Id||Description||Input||Shuffle Read||Shuffle Write||
> |12|saveAsTextFile| |1170.2KB| |
> |11|combineByKey| |1264.5KB|1275.0KB|
> |8|sortByKey| |1273.6KB| |
> |7|mapPartitions| |134.5MB|1383.1KB|
> |5|zipWithIndex| |132.5MB| |
> |4|sortBy|155.6MB| |146.9MB|
> |3|sortBy|155.6MB| | |
> |2|collect| |2.0MB| |
> |1|mapValues|155.6MB| |2.2MB|
> |0|first|184.4KB| | |
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
