Zifei Feng created SPARK-53792:
----------------------------------
Summary: Fix rocksdbPinnedBlocksMemoryUsage when bounded memory
usage is enabled
Key: SPARK-53792
URL: https://issues.apache.org/jira/browse/SPARK-53792
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 4.0.1
Reporter: Zifei Feng
We forgot to fix this to show the correct metric when bounded memory usage is
enabled. Currently, it is collecting data for each RocksDB without accounting
for when all the RocksDB instances on the executor are sharing the same cache,
leading to double counting.
This is where we collect the cache memory metric:
[https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala#L2045]
We are currently using getDBProperty("rocksdb.block-cache-pinned-usage") to
read the pinned blocks for each DB. When the block cache is shared, this is
wrong because:
* {*}Instance-specific vs. global stats{*}: Database properties like
"rocksdb.block-cache-pinned-usage" report on the memory size of entries
requested specifically by that DB instance.
* {*}Double-counting potential{*}: If you query the DB property on both
instances and add them together, you could potentially double-count because a
single block in the shared cache could be used by both DB instances.
*Solution:* Do lrucache.getPinnedUsage
([https://github.com/facebook/rocksdb/blob/v9.8.4/java/src/main/java/org/rocksdb/Cache.java#L33C15-L33C29]
in OSS RocksDB) instead, to get the actual memory size of pinned blocks in the
shared cache. We are querying the cache here instead of the DB.
To estimate the pinnedBlocks used by a single DB instance, we can divide
lrucache.getPinnedUsage() by num_rocksdb_instances_sharing_the_cache. We
already do a similar thing for memoryUsage. See:
[https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDBMemoryManager.scala#L124]
*Original fix:* OSS Spark PR:[
https://github.com/apache/spark/commit/35c299a1e3e373e20ae45d7604df51c83ff1dbe2|https://github.com/apache/spark/commit/35c299a1e3e373e20ae45d7604df51c83ff1dbe2]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
