Github user arunmahadevan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21469#discussion_r194480087
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala
---
@@ -247,6 +253,14 @@ private[state] class HDFSBackedStateStoreProvider
extends StateStoreProvider wit
private lazy val fm = CheckpointFileManager.create(baseDir, hadoopConf)
private lazy val sparkConf =
Option(SparkEnv.get).map(_.conf).getOrElse(new SparkConf)
+ private lazy val metricProviderLoaderMapSizeBytes:
StateStoreCustomSizeMetric =
+ StateStoreCustomSizeMetric("providerLoadedMapSizeBytes",
+ "estimated size of states cache in provider")
+
+ private lazy val metricProviderLoaderCountOfVersionsInMap:
StateStoreCustomAverageMetric =
--- End diff --
Why is "metricProviderLoaderCountOfVersionsInMap" an average metrics? The
other metrics like "numTotalStateRows" and even "providerLoadedMapSizeBytes" is
count metric. Shouldn't this be similar?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
