Hi All, We have a spark streaming v1.4/java 8 application that slows down and eventually runs out of heap space. The less driver memory, the faster it happens.
Appended is our spark configuration and a snapshot of the of heap taken using jmap on the driver process. The RDDInfo, $colon$colon and [C objects keep growing as we observe. We also tried to use G1GC, but it acts the same. Our dependency graph contains multiple updateStateByKey() calls. For each, we explicitly set the checkpoint interval to 240 seconds. We have our batch interval set to 15 seconds; with no delays at the start of the process. Spark configuration (Spark Driver Memory: 6GB, Spark Executor Memory: 2GB): spark.streaming.minRememberDuration=180s spark.ui.showConsoleProgress=false spark.streaming.receiver.writeAheadLog.enable=true spark.streaming.unpersist=true spark.streaming.stopGracefullyOnShutdown=true spark.streaming.ui.retainedBatches=10 spark.ui.retainedJobs=10 spark.ui.retainedStages=10 spark.worker.ui.retainedExecutors=10 spark.worker.ui.retainedDrivers=10 spark.sql.ui.retainedExecutions=10 spark.serializer=org.apache.spark.serializer.KryoSerializer spark.kryoserializer.buffer.max=128m num #instances #bytes class name ---------------------------------------------- 1: 8828200 565004800 org.apache.spark.storage.RDDInfo 2: 20794893 499077432 scala.collection.immutable.$colon$colon 3: 9646097 459928736 [C 4: 9644398 231465552 java.lang.String 5: 12760625 204170000 java.lang.Integer 6: 21326 111198632 [B 7: 556959 44661232 [Lscala.collection.mutable.HashEntry; 8: 1179788 37753216 java.util.concurrent.ConcurrentHashMap$Node 9: 1169264 37416448 java.util.Hashtable$Entry 10: 552707 30951592 org.apache.spark.scheduler.StageInfo 11: 367107 23084712 [Ljava.lang.Object; 12: 556948 22277920 scala.collection.mutable.HashMap 13: 2787 22145568 [Ljava.util.concurrent.ConcurrentHashMap$Node; 14: 116997 12167688 org.apache.spark.executor.TaskMetrics 15: 360425 8650200 java.util.concurrent.LinkedBlockingQueue$Node 16: 360417 8650008 org.apache.spark.deploy.history.yarn.HandleSparkEvent 17: 8332 8478088 [Ljava.util.Hashtable$Entry; 18: 351061 8425464 scala.collection.mutable.ArrayBuffer 19: 116963 8421336 org.apache.spark.scheduler.TaskInfo 20: 446136 7138176 scala.Some 21: 211968 5087232 io.netty.buffer.PoolThreadCache$MemoryRegionCache$Entry 22: 116963 4678520 org.apache.spark.scheduler.SparkListenerTaskEnd 23: 107679 4307160 org.apache.spark.executor.ShuffleWriteMetrics 24: 72162 4041072 org.apache.spark.executor.ShuffleReadMetrics 25: 117223 3751136 scala.collection.mutable.ListBuffer 26: 81473 3258920 org.apache.spark.executor.InputMetrics 27: 125903 3021672 org.apache.spark.rdd.RDDOperationScope 28: 91455 2926560 java.util.HashMap$Node 29: 89 2917776 [Lscala.concurrent.forkjoin.ForkJoinTask; 30: 116957 2806968 org.apache.spark.scheduler.SparkListenerTaskStart 31: 2122 2188568 [Lorg.apache.spark.scheduler.StageInfo; 32: 16411 1819816 java.lang.Class 33: 87862 1405792 org.apache.spark.scheduler.SparkListenerUnpersistRDD 34: 22915 916600 org.apache.spark.storage.BlockStatus 35: 5887 895568 [Ljava.util.HashMap$Node; 36: 480 855552 [Lio.netty.buffer.PoolThreadCache$MemoryRegionCache$Entry; 37: 7569 834968 [I 38: 9626 770080 org.apache.spark.rdd.MapPartitionsRDD 39: 31748 761952 java.lang.Long -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-heap-space-out-of-memory-tp27050.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
