Hi, Just reviewing StateStoreRestoreExec [1] and been wondering how to know whether a state was available for a key. It has numOutputRows metric [2], but that gives the number of aggregations from the child operator only and seems to say nothing about whether state was available for an aggregation.
What do you think about adding numSavedStates metric to StateStoreRestoreExec? Or is there a way to find it out already (perhaps in web UI)? [1] https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala#L186 [2] https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala#L206 Pozdrawiam, Jacek Laskowski ---- https://about.me/JacekLaskowski Spark Structured Streaming (Apache Spark 2.2+) https://bit.ly/spark-structured-streaming Mastering Apache Spark 2 https://bit.ly/mastering-apache-spark Follow me at https://twitter.com/jaceklaskowski --------------------------------------------------------------------- To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
