zecookiez commented on code in PR #49816:
URL: https://github.com/apache/spark/pull/49816#discussion_r1950052021
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBStateStoreProvider.scala:
##########
@@ -328,7 +328,9 @@ private[sql] class RocksDBStateStoreProvider
CUSTOM_METRIC_FLUSH_WRITTEN_BYTES ->
nativeOpsMetrics("totalBytesWrittenByFlush"),
CUSTOM_METRIC_PINNED_BLOCKS_MEM_USAGE ->
rocksDBMetrics.pinnedBlocksMemUsage,
CUSTOM_METRIC_NUM_EXTERNAL_COL_FAMILIES -> internalColFamilyCnt(),
- CUSTOM_METRIC_NUM_INTERNAL_COL_FAMILIES -> externalColFamilyCnt()
+ CUSTOM_METRIC_NUM_INTERNAL_COL_FAMILIES -> externalColFamilyCnt(),
+ CUSTOM_METRIC_SNAPSHOT_LAST_UPLOADED.withPartition(id.partitionId) ->
Review Comment:
Good catch, added additional logic to differentiate this case over
[here](https://github.com/apache/spark/pull/49816/files#diff-da6ad0bc819dce994a16436fa0797bfc8484644b475227a04c2a3eb5927515f7R436)
during intitialization, and two new tests are added to verify join operators
work with these metrics
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
