Re: What is Spark context cleaner in structured streaming

Hi Akshay, You may refer the following: https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/spark-service-contextcleaner.html Thanks, Kanchan Data Engineer, IBM On Thu, May 2, 2019 at 5:21 PM Akshay Bhardwaj < akshay.bhardwaj1...@gmail.com> wrote: > Hi All, > > I am using Spark

What is Spark context cleaner in structured streaming

Hi All, I am using Spark structured streaming with spark 2.3 running on Yarn cluster with hadoop 2.7.3 In the driver logs I see numerous lines as below. 2019-05-02 17:14:24.619 ContextCleaner Spark Context Cleaner [INFO] Cleaned accumulator 81492577 2019-05-02 17:14:24.619 ContextCleaner Spark