No. We are already capturing these metrics (e.g. numInputRows, inputRowsPerSecond).
I am talking about "No. of States" in the memory at any given time. On Thu, May 7, 2020 at 4:31 PM Jungtaek Lim <kabhwan.opensou...@gmail.com> wrote: > If you're referring total "entries" in all states in SS job, it's being > provided via StreamingQueryListener. > > > http://spark.apache.org/docs/latest/structured-streaming-programming-guide.html#monitoring-streaming-queries > > Hope this helps. > > On Fri, May 8, 2020 at 3:26 AM Something Something < > mailinglist...@gmail.com> wrote: > >> Is there a way to get the total no. of active states in memory at any >> given point in a Stateful Spark Structured Streaming job? We are thinking >> of using this metric for 'Auto Scaling' our Spark cluster. >> >
