[jira] [Commented] (SPARK-23029) Doc spark.shuffle.file.buffer units are kb when no units specified
[ https://issues.apache.org/jira/browse/SPARK-23029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16326260#comment-16326260 ] Apache Spark commented on SPARK-23029: -- User 'ferdonline' has created a pull request for this issue: https://github.com/apache/spark/pull/20269 > Doc spark.shuffle.file.buffer units are kb when no units specified > -- > > Key: SPARK-23029 > URL: https://issues.apache.org/jira/browse/SPARK-23029 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 2.2.1 >Reporter: Fernando Pereira >Priority: Minor > > When setting the spark.shuffle.file.buffer setting, even to its default > value, shuffles fail. > This appears to affect small to medium size partitions. Strangely the error > message is OutOfMemoryError, but it works with large partitions (at least > >32MB). > {code} > pyspark --conf "spark.shuffle.file.buffer=$((32*1024))" > /gpfs/bbp.cscs.ch/scratch/gss/spykfunc/_sparkenv/lib/python2.7/site-packages/pyspark/bin/spark-submit > pyspark-shell-main --name PySparkShell --conf spark.shuffle.file.buffer=32768 > version 2.2.1 > >>> spark.range(1e7, numPartitions=10).sort("id").write.parquet("a", > >>> mode="overwrite") > [Stage 1:>(0 + 10) / > 10]18/01/10 19:34:21 ERROR Executor: Exception in task 1.0 in stage 1.0 (TID > 11) > java.lang.OutOfMemoryError: Java heap space > at java.io.BufferedOutputStream.(BufferedOutputStream.java:75) > at > org.apache.spark.storage.DiskBlockObjectWriter$ManualCloseBufferedOutputStream$1.(DiskBlockObjectWriter.scala:107) > at > org.apache.spark.storage.DiskBlockObjectWriter.initialize(DiskBlockObjectWriter.scala:108) > at > org.apache.spark.storage.DiskBlockObjectWriter.open(DiskBlockObjectWriter.scala:116) > at > org.apache.spark.storage.DiskBlockObjectWriter.write(DiskBlockObjectWriter.scala:237) > at > org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:151) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53) > at org.apache.spark.scheduler.Task.run(Task.scala:108) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > {code} -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23029) Doc spark.shuffle.file.buffer units are kb when no units specified
[ https://issues.apache.org/jira/browse/SPARK-23029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324402#comment-16324402 ] Fernando Pereira commented on SPARK-23029: -- [~jerryshao] Thanks a lot for the info, now I understand better the mem usage! I still don't understand how come it worked with large partitions and a buffer of 1024*1024 (*1024!) Yep I can do the update in the configurations page, hope to save some people from the same hassle. Cheers > Doc spark.shuffle.file.buffer units are kb when no units specified > -- > > Key: SPARK-23029 > URL: https://issues.apache.org/jira/browse/SPARK-23029 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 2.2.1 >Reporter: Fernando Pereira >Priority: Minor > > When setting the spark.shuffle.file.buffer setting, even to its default > value, shuffles fail. > This appears to affect small to medium size partitions. Strangely the error > message is OutOfMemoryError, but it works with large partitions (at least > >32MB). > {code} > pyspark --conf "spark.shuffle.file.buffer=$((32*1024))" > /gpfs/bbp.cscs.ch/scratch/gss/spykfunc/_sparkenv/lib/python2.7/site-packages/pyspark/bin/spark-submit > pyspark-shell-main --name PySparkShell --conf spark.shuffle.file.buffer=32768 > version 2.2.1 > >>> spark.range(1e7, numPartitions=10).sort("id").write.parquet("a", > >>> mode="overwrite") > [Stage 1:>(0 + 10) / > 10]18/01/10 19:34:21 ERROR Executor: Exception in task 1.0 in stage 1.0 (TID > 11) > java.lang.OutOfMemoryError: Java heap space > at java.io.BufferedOutputStream.(BufferedOutputStream.java:75) > at > org.apache.spark.storage.DiskBlockObjectWriter$ManualCloseBufferedOutputStream$1.(DiskBlockObjectWriter.scala:107) > at > org.apache.spark.storage.DiskBlockObjectWriter.initialize(DiskBlockObjectWriter.scala:108) > at > org.apache.spark.storage.DiskBlockObjectWriter.open(DiskBlockObjectWriter.scala:116) > at > org.apache.spark.storage.DiskBlockObjectWriter.write(DiskBlockObjectWriter.scala:237) > at > org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:151) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96) > at > org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53) > at org.apache.spark.scheduler.Task.run(Task.scala:108) > at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org