Re: [PR] [SPARK-51252] [SS] Add instance metrics for last uploaded snapshot version in HDFS State Stores [spark]

Mon, 24 Feb 2025 12:41:31 -0800 


zecookiez commented on code in PR #50030:
URL: https://github.com/apache/spark/pull/50030#discussion_r1968378429


##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/HDFSBackedStateStoreProvider.scala:
##########
@@ -219,7 +219,18 @@ private[sql] class HDFSBackedStateStoreProvider extends 
StateStoreProvider with
         supportedCustomMetrics.find(_.name == name).map(_ -> value)
       } + (metricStateOnCurrentVersionSizeBytes -> 
SizeEstimator.estimate(mapToUpdate))
 
-      StateStoreMetrics(mapToUpdate.size(), 
metricsFromProvider("memoryUsedBytes"), customMetrics)
+      val instanceMetrics = Map(
+        instanceMetricSnapshotLastUpload.withNewId(

Review Comment:
   Good question, it looks like the main issue is this:
   
![image](https://github.com/user-attachments/assets/34ccb451-a7f1-43d4-83c6-769701c053a9)
   
   Seems like state store ID is initialized at a later point in time than the 
metrics, and the default value is null



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to