Re: Unsubscribe

2022-08-10 Thread pengyh 

to unsubscribe: user-unsubscr...@spark.apache.org


Shrikar archak wrote:



unsubscribe


-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Unsubscribe

2022-08-10 Thread Shrikar archak 
unsubscribe

Memory leak while caching in foreachBatch block

2022-08-10 Thread kineret M 
Hi,

We have a structured streaming application, and we face a memory leak while
caching in the foreachBatch block.

We do unpersist every iteration, and we also verify via
"spark.sparkContext.getPersistentRDDs" that we don't have unnecessary
cached data.

We also noted in the profiler that many sparkSession objects are created
while we use cache (vs 2 sparkSession objects while not using cache).

Any idea what can cause this? We use spark 3.1.1