Did you verify the driver memory in the Executor tab of the WebUI? I think you need `--driver-memory 8g` with spark-shell or spark-submit instead of setting it in spark-defaults.conf.
On Fri, Aug 15, 2014 at 12:41 PM, jerryye <jerr...@gmail.com> wrote: > Setting spark.driver.memory has no effect. It's still hanging trying to > compute result.count when I'm sampling greater than 35% regardless of what > value of spark.driver.memory I'm setting. > > Here's my settings: > export SPARK_JAVA_OPTS="-Xms5g -Xmx10g -XX:MaxPermSize=10g" > export SPARK_MEM=10g > > in conf/spark-defaults: > spark.driver.memory 1500 > spark.serializer org.apache.spark.serializer.KryoSerializer > spark.kryoserializer.buffer.mb 500 > spark.executor.memory 58315m > spark.executor.extraLibraryPath /root/ephemeral-hdfs/lib/native/ > spark.executor.extraClassPath /root/ephemeral-hdfs/conf > > > > -- > View this message in context: > http://apache-spark-developers-list.1001551.n3.nabble.com/spark-akka-frameSize-stalls-job-in-1-1-0-tp7865p7877.html > Sent from the Apache Spark Developers List mailing list archive at Nabble.com. > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org > For additional commands, e-mail: dev-h...@spark.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
