Maciej Bryński created SPARK-10538:
--------------------------------------
Summary: java.lang.NegativeArraySizeException during join
Key: SPARK-10538
URL: https://issues.apache.org/jira/browse/SPARK-10538
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 1.5.0
Reporter: Maciej Bryński
Hi,
I've got a problem during joining tables. (in my example 20 of them)
I can observe that during calculation of first partition (on one of consecutive
joins) there is a big shuffle read size (294.7 MB / 146 records) vs on others
partitions (approx. 272.5 KB / 113 record)
After some time there is an exception:
{code}
java.lang.NegativeArraySizeException
at
org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown
Source)
at
org.apache.spark.sql.execution.TungstenProject$$anonfun$3$$anonfun$apply$3.apply(basicOperators.scala:90)
at
org.apache.spark.sql.execution.TungstenProject$$anonfun$3$$anonfun$apply$3.apply(basicOperators.scala:88)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at
org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.insertAll(BypassMergeSortShuffleWriter.java:119)
at
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:73)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:73)
at
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)
at org.apache.spark.scheduler.Task.run(Task.scala:88)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:214)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
I'm running this on 2 nodes cluster (12 cores, 64 GB RAM)
Config:
{code}
spark.driver.memory 10g
spark.executor.extraJavaOptions -XX:-UseGCOverheadLimit -XX:+UseParallelGC
-Dfile.encoding=UTF8
spark.executor.memory 60g
spark.storage.memoryFraction 0.05
spark.shuffle.memoryFraction 0.75
spark.driver.maxResultSize 10g 6
spark.cores.max 24
spark.kryoserializer.buffer.max 1g
spark.default.parallelism 200
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
