shrirangmhalgi commented on code in PR #56598:
URL: https://github.com/apache/spark/pull/56598#discussion_r3439733635
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala:
##########
@@ -1583,6 +1583,18 @@ class RocksDB(
originalEndKey
}
+ // Count the keys that are about to be deleted (gated on
trackTotalNumberOfRows).
+ if (conf.trackTotalNumberOfRows) {
Review Comment:
`deleteRange` in RocksDB is O(1) - it writes a single range tombstone
without touching individual keys. Scanning the full range with an iterator to
count keys makes this O(n). For TTL-based state expiration (the primary
`deleteRange` use case in structured streaming), ranges can contain millions of
expired keys. When `trackTotalNumberOfRows` is enabled (the default), every
`deleteRange` now iterates the full range before deleting it. Worth documenting
this cost tradeoff, or considering RocksDB's approximate-count APIs
(`GetApproximateSizes`) as an alternative for large ranges.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
