Just as an offhand guess, are you doing something like updateStateByKey without expiring old keys?
On Fri, Jul 8, 2016 at 2:44 AM, Jörn Franke <jornfra...@gmail.com> wrote: > Memory fragmentation? Quiet common with in-memory systems. > >> On 08 Jul 2016, at 08:56, aasish.kumar <aasish.ku...@avekshaa.com> wrote: >> >> Hello everyone: >> >> I have been facing a problem associated spark streaming memory. >> >> I have been running two Spark Streaming jobs concurrently. The jobs read >> data from Kafka with a batch interval of 1 minute, performs aggregation, and >> sinks the computed data to MongoDB using using stratio-mongodb connector. >> >> I have setup the spark standalone cluster on AWS. My setup is configured as >> follows: I have a four-node cluster. One node as a master, and the rest >> 3-nodes as workers, while each worker has only one executor, with 2-cores >> and 8GB of RAM. >> >> Currently, I am processing seven-hundred thousand JSON events, every minute. >> After running the jobs for 3-4 hours, I have observed that the memory >> consumption keeps growing, exiting one of the jobs. >> >> Despite setting /spark.cleaner.ttl/ for 600 seconds, and having used >> /rdd.unpersist/ method at the end of the job. I am not able to understand >> why the memory consumption keeps growing over time. I am unable solve this >> problem. I would appreciate if someone can help me solve or provide >> redirections as to why this is happening. >> >> Thank you. >> >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Memory-grows-exponentially-tp27308.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> --------------------------------------------------------------------- >> To unsubscribe e-mail: user-unsubscr...@spark.apache.org >> > > --------------------------------------------------------------------- > To unsubscribe e-mail: user-unsubscr...@spark.apache.org > --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
