Solved:
Call spark-submit with
--driver-memory 512m --driver-java-options
"-Dspark.memory.useLegacyMode=true -Dspark.shuffle.memoryFraction=0.2
-Dspark.storage.memoryFraction=0.6 -Dspark.storage.unrollFraction=0.2"
Thanks to:
https://issues.apache.org/jira/browse/SPARK-14367
--
View this
Hi,
I am trying to get the same memory behavior in Spark 1.6 as I had in Spark
1.3 with default settings.
I set
--driver-java-options "--Dspark.memory.useLegacyMode=true
-Dspark.shuffle.memoryFraction=0.2 -Dspark.storage.memoryFraction=0.6
-Dspark.storage.unrollFraction=0.2"
in Spark 1.6.
But