touchida opened a new issue, #4951: URL: https://github.com/apache/kyuubi/issues/4951
### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) ### Search before asking - [X] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues. ### Describe the bug Currently, operation logging supports the slf4j bindings of log4j 1 and log4j 2. - https://github.com/apache/kyuubi/blob/06dd7d3/kyuubi-common/src/main/scala/org/apache/kyuubi/operation/log/LogDivertAppender.scala#L27-L34 In https://issues.apache.org/jira/browse/HADOOP-18088, Hadoop replaced log4j 1 with reload4j. When a Spark SQL engine uses the reload4j slf4j binding from Hadoop, operation logs on the engine are not shown to a client, since `LogDivertAppender` is not initialized, as the following engine log indicates: ``` 23/06/11 16:28:39 WARN log.LogDivertAppender: Unsupported SLF4J binding org.slf4j.impl.Reload4jLoggerFactory ``` In more details, we confirmed this issue happened when - we used Spark SQL engine in YARN cluster deploy mode; - we built Hadoop-free Spark 3.2.4, and provided the classpath of Hadoop in YARN cluster, whose version used reload4j, with `SPARK_DIST_CLASSPATH`. Note - We also confirmed this issue did not happen for Spark 3.2.4 built with Hadoop, since then the Spark jars contained the log4j 1 slf4j binding. - We did not test, but it would not happen for Spark 3.3 and later, since Spark used log4j 2 (https://issues.apache.org/jira/browse/SPARK-37814). ### Affects Version(s) master ### Kyuubi Server Log Output _No response_ ### Kyuubi Engine Log Output ```logtalk 23/06/11 16:28:39 WARN log.LogDivertAppender: Unsupported SLF4J binding org.slf4j.impl.Reload4jLoggerFactory ``` ### Kyuubi Server Configurations _No response_ ### Kyuubi Engine Configurations _No response_ ### Additional context _No response_ ### Are you willing to submit PR? - [X] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix. - [ ] No. I cannot submit a PR at this time. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
