This is an automated email from the ASF dual-hosted git repository.
kabhwan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 726999f098d9 [SPARK-49575][SS] Add logging for lock release only if
acquiredThreadInfo is not null
726999f098d9 is described below
commit 726999f098d9b5381c401edb064f61f1606e572e
Author: Anish Shrigondekar <[email protected]>
AuthorDate: Tue Sep 10 16:05:48 2024 +0900
[SPARK-49575][SS] Add logging for lock release only if acquiredThreadInfo
is not null
### What changes were proposed in this pull request?
Add logging for lock release only if acquiredThreadInfo is not null
### Why are the changes needed?
Without this, there are a lot of extraneous executor logs of the following
type
```
24/09/05 21:38:19 INFO RocksDB StateStoreId(opId=0,partId=45,name=default):
RocksDB instance was released by null for opType=store_task_completion_listener
24/09/05 21:38:19 INFO RocksDB StateStoreId(opId=0,partId=45,name=default):
RocksDB instance was released by null for opType=store_task_completion_listener
24/09/05 21:38:20 INFO RocksDB
StateStoreId(opId=0,partId=173,name=default): RocksDB instance was released by
null for opType=store_task_completion_listener
24/09/05 21:38:20 INFO RocksDB
StateStoreId(opId=0,partId=173,name=default): RocksDB instance was released by
null for opType=store_task_completion_listener
24/09/05 21:38:21 INFO RocksDB StateStoreId(opId=0,partId=13,name=default):
RocksDB instance was released by null for opType=store_task_completion_listener
24/09/05 21:38:21 INFO RocksDB StateStoreId(opId=0,partId=13,name=default):
RocksDB instance was released by null for opType=store_task_completion_listener
24/09/05 21:38:21 INFO RocksDB
StateStoreId(opId=0,partId=141,name=default): RocksDB instance was released by
null for opType=store_task_completion_listener
24/09/05 21:38:21 INFO RocksDB
StateStoreId(opId=0,partId=141,name=default): RocksDB instance was released by
null for opType=store_task_completion_listener
24/09/05 21:38:25 INFO RocksDB
StateStoreId(opId=0,partId=109,name=default): RocksDB instance was released by
null for opType=store_task_completion_listener
24/09/05 21:38:25 INFO RocksDB
StateStoreId(opId=0,partId=109,name=default): RocksDB instance was released by
null for opType=store_task_completion_listener
```
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Existing unit tests
### Was this patch authored or co-authored using generative AI tooling?
No
Closes #48050 from anishshri-db/task/SPARK-49575.
Authored-by: Anish Shrigondekar <[email protected]>
Signed-off-by: Jungtaek Lim <[email protected]>
---
.../apache/spark/sql/execution/streaming/state/RocksDB.scala | 10 ++++++----
1 file changed, 6 insertions(+), 4 deletions(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala
index 5229c6594350..bbb28ac646ec 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala
@@ -929,10 +929,12 @@ class RocksDB(
* @param opType - operation type releasing the lock
*/
private def release(opType: RocksDBOpType): Unit = acquireLock.synchronized {
- logInfo(log"RocksDB instance was released by ${MDC(LogKeys.THREAD,
acquiredThreadInfo)} " +
- log"for opType=${MDC(LogKeys.OP_TYPE, opType.toString)}")
- acquiredThreadInfo = null
- acquireLock.notifyAll()
+ if (acquiredThreadInfo != null) {
+ logInfo(log"RocksDB instance was released by ${MDC(LogKeys.THREAD,
+ acquiredThreadInfo)} " + log"for opType=${MDC(LogKeys.OP_TYPE,
opType.toString)}")
+ acquiredThreadInfo = null
+ acquireLock.notifyAll()
+ }
}
private def getDBProperty(property: String): Long =
db.getProperty(property).toLong
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
