This is an automated email from the ASF dual-hosted git repository.
kabhwan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 630b1777904 [SPARK-44683][SS] Logging level isn't passed to RocksDB
state store provider correctly
630b1777904 is described below
commit 630b1777904f15c7ac05c3cd61c0006cd692bc93
Author: Siying Dong <siying.d...@databricks.com>
AuthorDate: Tue Aug 8 11:11:56 2023 +0900
[SPARK-44683][SS] Logging level isn't passed to RocksDB state store
provider correctly
### What changes were proposed in this pull request?
The logging level is passed into RocksDB in a correct way.
### Why are the changes needed?
We pass log4j's log level to RocksDB so that RocksDB debug log can go to
log4j. However, we pass in log level after we create the logger. However, the
way it is set isn't effective. This has two impacts: (1) setting DEBUG level
don't make RocksDB generate DEBUG level logs; (2) setting WARN or ERROR level
does prevent INFO level logging, but RocksDB still makes JNI calls to Scala,
which is an unnecessary overhead.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Manually change the log level and observe the log lines in unit tests.
Closes #42354 from siying/rocks_log_level.
Authored-by: Siying Dong <siying.d...@databricks.com>
Signed-off-by: Jungtaek Lim <kabhwan.opensou...@gmail.com>
---
.../org/apache/spark/sql/execution/streaming/state/RocksDB.scala | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala
index d4366fe732b..a2868df9411 100644
---
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala
+++
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/RocksDB.scala
@@ -611,8 +611,11 @@ class RocksDB(
if (log.isWarnEnabled) dbLogLevel = InfoLogLevel.WARN_LEVEL
if (log.isInfoEnabled) dbLogLevel = InfoLogLevel.INFO_LEVEL
if (log.isDebugEnabled) dbLogLevel = InfoLogLevel.DEBUG_LEVEL
- dbOptions.setLogger(dbLogger)
+ dbLogger.setInfoLogLevel(dbLogLevel)
+ // The log level set in dbLogger is effective and the one to dbOptions
isn't applied to
+ // customized logger. We still set it as it might show up in RocksDB
config file or logging.
dbOptions.setInfoLogLevel(dbLogLevel)
+ dbOptions.setLogger(dbLogger)
logInfo(s"Set RocksDB native logging level to $dbLogLevel")
dbLogger
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
