anishshri-db commented on code in PR #46491:
URL: https://github.com/apache/spark/pull/46491#discussion_r1594861475
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -777,10 +777,19 @@ class RocksDB(
.keys.filter(checkInternalColumnFamilies(_)).size
val numExternalColFamilies = colFamilyNameToHandleMap.keys.size -
numInternalColFamilies
+ // if bounded memory usage is enabled, we share the block cache across all
state providers
+ // running on the same node and account the usage to this single cache. In
this case, its not
+ // possible to provide partition level or query level memory usage.
+ val memoryUsage = if (conf.boundedMemoryUsage) {
+ 0L
Review Comment:
Tried this, but it seems the progress metrics will still convert this to 0
```
"stateOperators" : [ {
"operatorName" : "stateStoreSave",
"numRowsTotal" : 3,
"numRowsUpdated" : 3,
"allUpdatesTimeMs" : 10,
"numRowsRemoved" : 0,
"allRemovalsTimeMs" : 0,
"commitTimeMs" : 211,
"memoryUsedBytes" : 0,
```
It seems the lowest allowed value for the SQLMetric will still be
interpreted as 0.
We seem to do the same thing for reporting `numRowsTotal` if the
`trackingTotalNumRows` flag is disabled as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
