This is an automated email from the ASF dual-hosted git repository.
yangjie01 pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/master by this push:
new 75cd9be439b4 [SPARK-55237][SQL] Suppress annoying messages when
looking up nonexistent DBs
75cd9be439b4 is described below
commit 75cd9be439b43d7657b97260b9024ebb18353179
Author: Cheng Pan <[email protected]>
AuthorDate: Fri Jan 30 11:32:36 2026 +0800
[SPARK-55237][SQL] Suppress annoying messages when looking up nonexistent
DBs
### What changes were proposed in this pull request?
When running `spark-sql` with embedded HMS and default log4j config, I see
a lot of noisy logs.
```
spark-sql (default)> create database test_db;
26/01/27 15:56:33 WARN ObjectStore: Failed to get database test_db,
returning NoSuchObjectException
26/01/27 15:56:33 WARN ObjectStore: Failed to get database test_db,
returning NoSuchObjectException
26/01/27 15:56:33 WARN ObjectStore: Failed to get database test_db,
returning NoSuchObjectException
Time taken: 0.87 seconds
spark-sql (default)>
```
This PR modifies the default log4j config files to suppress those logs.
Note, `ObjectStore` will not be called by the Spark driver when using
remote HMS (typical deployment of production cases)
### Why are the changes needed?
Improve UX on `spark-sql` with embedded HMS and default log4j config.
### Does this PR introduce _any_ user-facing change?
Only affect logs. User would see less noise when running `spark-sql` with
embedded HMS and default log4j config.
### How was this patch tested?
before (see above section) vs after
```
spark-sql (default)> create database test_db2;
Time taken: 0.686 seconds
```
### Was this patch authored or co-authored using generative AI tooling?
No.
Closes #54002 from pan3793/SPARK-55237.
Authored-by: Cheng Pan <[email protected]>
Signed-off-by: yangjie01 <[email protected]>
---
.../main/resources/org/apache/spark/log4j2-defaults.properties | 8 ++++----
.../resources/org/apache/spark/log4j2-json-layout.properties | 8 ++++----
conf/log4j2-json-layout.properties.template | 10 +++++-----
conf/log4j2.properties.template | 10 +++++-----
4 files changed, 18 insertions(+), 18 deletions(-)
diff --git
a/common/utils-java/src/main/resources/org/apache/spark/log4j2-defaults.properties
b/common/utils-java/src/main/resources/org/apache/spark/log4j2-defaults.properties
index 777c5f2b2591..b2b71e56eeb9 100644
---
a/common/utils-java/src/main/resources/org/apache/spark/log4j2-defaults.properties
+++
b/common/utils-java/src/main/resources/org/apache/spark/log4j2-defaults.properties
@@ -41,12 +41,12 @@ logger.repl2.level = info
logger.repl.name = org.apache.spark.repl.Main
logger.repl.level = warn
-# SPARK-9183: Settings to avoid annoying messages when looking up nonexistent
UDFs
-# in SparkSQL with Hive support
+# SPARK-9183, SPARK-55237: Settings to avoid annoying messages when looking up
+# nonexistent DBs/UDFs in spark-sql with Hive support
logger.metastore.name = org.apache.hadoop.hive.metastore.RetryingHMSHandler
logger.metastore.level = fatal
-logger.hive_functionregistry.name =
org.apache.hadoop.hive.ql.exec.FunctionRegistry
-logger.hive_functionregistry.level = error
+logger.metastore2.name = org.apache.hadoop.hive.metastore.ObjectStore
+logger.metastore2.level = error
# Parquet related logging
logger.parquet.name = org.apache.parquet.CorruptStatistics
diff --git
a/common/utils-java/src/main/resources/org/apache/spark/log4j2-json-layout.properties
b/common/utils-java/src/main/resources/org/apache/spark/log4j2-json-layout.properties
index 9be86b650d09..b33088272e6a 100644
---
a/common/utils-java/src/main/resources/org/apache/spark/log4j2-json-layout.properties
+++
b/common/utils-java/src/main/resources/org/apache/spark/log4j2-json-layout.properties
@@ -41,12 +41,12 @@ logger.repl2.level = info
logger.repl.name = org.apache.spark.repl.Main
logger.repl.level = warn
-# SPARK-9183: Settings to avoid annoying messages when looking up nonexistent
UDFs
-# in SparkSQL with Hive support
+# SPARK-9183, SPARK-55237: Settings to avoid annoying messages when looking up
+# nonexistent DBs/UDFs in spark-sql with Hive support
logger.metastore.name = org.apache.hadoop.hive.metastore.RetryingHMSHandler
logger.metastore.level = fatal
-logger.hive_functionregistry.name =
org.apache.hadoop.hive.ql.exec.FunctionRegistry
-logger.hive_functionregistry.level = error
+logger.metastore2.name = org.apache.hadoop.hive.metastore.ObjectStore
+logger.metastore2.level = error
# Parquet related logging
logger.parquet.name = org.apache.parquet.CorruptStatistics
diff --git a/conf/log4j2-json-layout.properties.template
b/conf/log4j2-json-layout.properties.template
index 76499bb6691e..c8ae3173578d 100644
--- a/conf/log4j2-json-layout.properties.template
+++ b/conf/log4j2-json-layout.properties.template
@@ -49,8 +49,8 @@ logger.parquet1.level = error
logger.parquet2.name = parquet
logger.parquet2.level = error
-# SPARK-9183: Settings to avoid annoying messages when looking up nonexistent
UDFs in SparkSQL with Hive support
-logger.RetryingHMSHandler.name =
org.apache.hadoop.hive.metastore.RetryingHMSHandler
-logger.RetryingHMSHandler.level = fatal
-logger.FunctionRegistry.name = org.apache.hadoop.hive.ql.exec.FunctionRegistry
-logger.FunctionRegistry.level = error
+# SPARK-9183, SPARK-55237: Settings to avoid annoying messages when looking up
nonexistent DBs/UDFs in spark-sql with Hive support
+logger.metastore.name = org.apache.hadoop.hive.metastore.RetryingHMSHandler
+logger.metastore.level = fatal
+logger.metastore2.name = org.apache.hadoop.hive.metastore.ObjectStore
+logger.metastore2.level = error
diff --git a/conf/log4j2.properties.template b/conf/log4j2.properties.template
index 011fca58c9b2..0c9dfe208283 100644
--- a/conf/log4j2.properties.template
+++ b/conf/log4j2.properties.template
@@ -55,8 +55,8 @@ logger.parquet1.level = error
logger.parquet2.name = parquet
logger.parquet2.level = error
-# SPARK-9183: Settings to avoid annoying messages when looking up nonexistent
UDFs in SparkSQL with Hive support
-logger.RetryingHMSHandler.name =
org.apache.hadoop.hive.metastore.RetryingHMSHandler
-logger.RetryingHMSHandler.level = fatal
-logger.FunctionRegistry.name = org.apache.hadoop.hive.ql.exec.FunctionRegistry
-logger.FunctionRegistry.level = error
+# SPARK-9183, SPARK-55237: Settings to avoid annoying messages when looking up
nonexistent DBs/UDFs in spark-sql with Hive support
+logger.metastore.name = org.apache.hadoop.hive.metastore.RetryingHMSHandler
+logger.metastore.level = fatal
+logger.metastore2.name = org.apache.hadoop.hive.metastore.ObjectStore
+logger.metastore2.level = error
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
