This is an automated email from the ASF dual-hosted git repository.
chengpan pushed a commit to branch branch-1.7
in repository https://gitbox.apache.org/repos/asf/kyuubi.git
The following commit(s) were added to refs/heads/branch-1.7 by this push:
new ddbd46bec [KYUUBI #4951] Add reload4j support
ddbd46bec is described below
commit ddbd46becec2aeeb784bddc8508acdcc9c8500ed
Author: toshihiko.uchida <[email protected]>
AuthorDate: Thu Jun 15 09:57:35 2023 +0800
[KYUUBI #4951] Add reload4j support
### _Why are the changes needed?_
It resolves https://github.com/apache/kyuubi/issues/4951.
It would be also possible to resolve it by adding log4j 1 slf4j binding
(e.g., adding `slf4j-api-1.7.30.jar` and `slf4j-log4j12-1.7.30`) to the Spark
jars directory, but it would be helpful if Kyuubi could support reload4j to
avoid log4j 1's security vulnerabilities.
### _How was this patch tested?_
- [ ] Add some test cases that check the changes thoroughly including
negative and positive cases if possible
- [x] Add screenshots for manual tests if appropriate
Assume
- we use Spark SQL engine in YARN cluster deploy mode;
- we use Hadoop-free Spark 3.2.4 and provides the classpath of Hadoop in
YARN cluster, which uses reload4j, with `SPARK_DIST_CLASSPATH`.
Before applying the patch, Hive beeline shows operation logs only from
Kyuubi server, as follows.
```
0: jdbc:hive2://some_kyuubi_host> select * from some_database.some_table;
2023-06-14 22:55:35.584 INFO
org.apache.kyuubi.credentials.HadoopCredentialsManager: Send new credentials
with epoch 0 to SQL engine through session 5409e4cb-abe0-457f-b3c8-c3f1c5467c60
2023-06-14 22:55:35.604 INFO
org.apache.kyuubi.credentials.HadoopCredentialsManager: Update session
credentials epoch from -1 to 0
2023-06-14 22:55:35.662 INFO org.apache.kyuubi.operation.ExecuteStatement:
Processing some_user's query[013b8294-b395-4e3d-945e-3594b0603f83]:
PENDING_STATE -> RUNNING_STATE, statement:
select * from some_database.some_table
2023-06-14 22:55:38.520 INFO org.apache.kyuubi.operation.ExecuteStatement:
Query[013b8294-b395-4e3d-945e-3594b0603f83] in FINISHED_STATE
2023-06-14 22:55:38.520 INFO org.apache.kyuubi.operation.ExecuteStatement:
Processing some_user's query[013b8294-b395-4e3d-945e-3594b0603f83]:
RUNNING_STATE -> FINISHED_STATE, time taken: 2.857 seconds
+-----+
| id |
+-----+
| 1 |
+-----+
1 row selected (3.113 seconds)
```
After applying the patch, Hive beeline shows operation logs from both
Kyuubi server and Spark SQL engine, as follows.
```
0: jdbc:hive2://some_kyuubi_host> select * from some_database.some_table;
2023-06-14 23:05:57.424 INFO
org.apache.kyuubi.credentials.HadoopCredentialsManager: Send new credentials
with epoch 0 to SQL engine through session 146aa800-6047-471e-a480-5683197a84cd
2023-06-14 23:05:57.453 INFO
org.apache.kyuubi.credentials.HadoopCredentialsManager: Update session
credentials epoch from -1 to 0
2023-06-14 23:05:57.512 INFO org.apache.kyuubi.operation.ExecuteStatement:
Processing some_user's query[1c209bea-6f62-4854-acdc-a1945867a65d]:
PENDING_STATE -> RUNNING_STATE, statement:
select * from some_database.some_table
23/06/14 23:05:57 INFO operation.ExecuteStatement: Processing some_user's
query[1c209bea-6f62-4854-acdc-a1945867a65d]: PENDING_STATE -> RUNNING_STATE,
statement:
select * from some_database.some_table
23/06/14 23:05:57 INFO operation.ExecuteStatement:
Spark application name:
kyuubi_CONNECTION_SPARK_SQL_some_user_146aa800-6047-471e-a480-5683197a84cd
application ID: application_1686295570868_0065
application web UI: some_yarn_application_url
master: yarn
deploy mode: cluster
version: 3.2.4
Start time: 2023-06-14T23:05:02.490
User: some_user
(omitted)
2023-06-14 23:06:00.285 INFO org.apache.kyuubi.operation.ExecuteStatement:
Query[1c209bea-6f62-4854-acdc-a1945867a65d] in FINISHED_STATE
2023-06-14 23:06:00.286 INFO org.apache.kyuubi.operation.ExecuteStatement:
Processing some_user's query[1c209bea-6f62-4854-acdc-a1945867a65d]:
RUNNING_STATE -> FINISHED_STATE, time taken: 2.774 seconds
+-----+
| id |
+-----+
| 1 |
+-----+
1 row selected (3.156 seconds)
```
- [ ] [Run
test](https://kyuubi.readthedocs.io/en/master/develop_tools/testing.html#running-tests)
locally before make a pull request
Closes #4964 from touchida/issue-4951-reload4j.
Closes #4951
d7444be17 [toshihiko.uchida] [KYUUBI #4951] Add reload4j support
Authored-by: toshihiko.uchida <[email protected]>
Signed-off-by: Cheng Pan <[email protected]>
(cherry picked from commit a9d4abe64a6fa206274a7e78e705c8f251396370)
Signed-off-by: Cheng Pan <[email protected]>
---
kyuubi-common/src/main/scala/org/apache/kyuubi/Logging.scala | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/kyuubi-common/src/main/scala/org/apache/kyuubi/Logging.scala
b/kyuubi-common/src/main/scala/org/apache/kyuubi/Logging.scala
index 1df598132..ebcd6b177 100644
--- a/kyuubi-common/src/main/scala/org/apache/kyuubi/Logging.scala
+++ b/kyuubi-common/src/main/scala/org/apache/kyuubi/Logging.scala
@@ -116,8 +116,9 @@ object Logging {
// This distinguishes the log4j 1.2 binding, currently
// org.slf4j.impl.Log4jLoggerFactory, from the log4j 2.0 binding, currently
// org.apache.logging.slf4j.Log4jLoggerFactory
- "org.slf4j.impl.Log4jLoggerFactory"
- .equals(LoggerFactory.getILoggerFactory.getClass.getName)
+ val binderClass = LoggerFactory.getILoggerFactory.getClass.getName
+ "org.slf4j.impl.Log4jLoggerFactory".equals(
+ binderClass) ||
"org.slf4j.impl.Reload4jLoggerFactory".equals(binderClass)
}
private[kyuubi] def isLog4j2: Boolean = {
