HeartSaVioR commented on code in PR #43961:
URL: https://github.com/apache/spark/pull/43961#discussion_r1431946666
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -604,17 +734,56 @@ class RocksDB(
}
private def getDBProperty(property: String): Long = {
- db.getProperty(property).toLong
+ if (useColumnFamilies) {
+ // get cumulative sum across all available column families
+ assert(!colFamilyNameToHandleMap.isEmpty)
+ colFamilyNameToHandleMap
+ .values
+ .map(handle => db.getProperty(handle, property).toLong)
+ .sum
Review Comment:
Maybe we should have done the check as it should be existing issue if there
is any since we accumulate values from all partitions. But good to double
check, and probably leave a code comment (even in current codebase).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
