Hi all, We just upgraded our Spark from 1.6.2 to 2.1.0.
Our Spark application is started by spark-submit with config of `--executor-memory 35G` in standalone model, but the actual use of memory up to 65G after a full gc(jmap -histo:live $pid) as follow: test@c6 ~ $ ps aux | grep CoarseGrainedExecutorBackend test 181941 181 34.7 94665384 68836752 ? Sl 09:25 711:21 /home/test/service/jdk/bin/java -cp /home/test/service/hadoop/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar:/home/test/service/hadoop/share/hadoop/common/hadoop-lzo-0.4.20-SNAPSHOT.jar:/home/test/service/spark/conf/:/home/test/service/spark/jars/*:/home/test/service/hadoop/etc/hadoop/ -Xmx35840M -Dspark.driver.port=47781 -XX:+PrintGCDetails -XX:+PrintGCDateStamps -Xloggc:./gc.log -verbose:gc org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url spark://coarsegrainedschedu...@xxx.xxx.xxx.xxx:47781 --executor-id 1 --hostname test-192 --cores 36 --app-id app-20170122092509-0017 --worker-url spark://Worker@test-192:33890 Our Spark jobs are all sql. The exceed memory looks like off-heap memory, but the default value of `spark.memory.offHeap.enabled` is `false`. We didn't find the problem in Spark 1.6.x, what causes this in Spark 2.1.0? Any help is greatly appreicated! Best, Stan -- View this message in context: http://apache-spark-developers-list.1001551.n3.nabble.com/Executors-exceed-maximum-memory-defined-with-executor-memory-in-Spark-2-1-0-tp20697.html Sent from the Apache Spark Developers List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
