Hi, We have a structured streaming application, and we face a memory leak while caching in the foreachBatch block.
We do unpersist every iteration, and we also verify via "spark.sparkContext.getPersistentRDDs" that we don't have unnecessary cached data. We also noted in the profiler that many sparkSession objects are created while we use cache (vs 2 sparkSession objects while not using cache). Any idea what can cause this? We use spark 3.1.1
