Github user HeartSaVioR commented on a diff in the pull request:
https://github.com/apache/spark/pull/21469#discussion_r194563959
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala
---
@@ -247,6 +253,14 @@ private[state] class HDFSBackedStateStoreProvider
extends StateStoreProvider wit
private lazy val fm = CheckpointFileManager.create(baseDir, hadoopConf)
private lazy val sparkConf =
Option(SparkEnv.get).map(_.conf).getOrElse(new SparkConf)
+ private lazy val metricProviderLoaderMapSizeBytes:
StateStoreCustomSizeMetric =
+ StateStoreCustomSizeMetric("providerLoadedMapSizeBytes",
+ "estimated size of states cache in provider")
+
+ private lazy val metricProviderLoaderCountOfVersionsInMap:
StateStoreCustomAverageMetric =
--- End diff --
This should be average to show `min, med, max` in SQL metrics UI, as I
pasted capture of UI before. Summing them all doesn't give meaningful value.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
