Yes. For this particular use case the state size could be big but I doubt if there's a leak. Maybe adding more memory would help.
On Thu, Jan 21, 2021 at 5:55 PM Sean Owen <sro...@gmail.com> wrote: > Is your app accumulating a lot of streaming state? that's one reason > something could slow down after a long time. Some memory leak in your app > putting GC/memory pressure on the JVM, etc too. > > On Thu, Jan 21, 2021 at 5:13 AM Eric Beabes <mailinglist...@gmail.com> > wrote: > >> Hello, >> >> My Spark Structured Streaming application was performing well for quite >> some time but all of a sudden from today it has slowed down. I noticed in >> the Spark UI that the 'No. of Active Tasks' is 1 even though 64 Cores are >> available. (Please see the attached image). >> >> I don't believe there's any data skew issue related to partitioning of >> data. What could be the reason for this? Please advise. Thanks. >> >> >> >> --------------------------------------------------------------------- >> To unsubscribe e-mail: user-unsubscr...@spark.apache.org > >
