[
https://issues.apache.org/jira/browse/SPARK-44646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17752524#comment-17752524
]
Yu Tian commented on SPARK-44646:
---------------------------------
Hi [~viirya]
Thanks for the suggestion, we spent some time evaluate the log4j-to-slf4j
approach, unfortunately, it seems not working.
Compared to log4j-over-slf4j for log4j 1.x, log4j-to-slf4j is mainly an adapter
for log4j-core, which means we still need the dependency of log4j-core. Since
logback and log4j-core are 2 implementations, slf4j will complain about it.
Below is the diagram we have tried:
!Screenshot 2023-08-09 at 2.40.12 PM.png!
If there are no better solutions, we may need to rewrite the existing logging
logics with log4j 2.x. Thanks.
> Migrate Log4j 2.x in Spark 3.4.1 to Logback
> -------------------------------------------
>
> Key: SPARK-44646
> URL: https://issues.apache.org/jira/browse/SPARK-44646
> Project: Spark
> Issue Type: Brainstorming
> Components: Build
> Affects Versions: 3.4.1
> Reporter: Yu Tian
> Priority: Major
> Attachments: Screenshot 2023-08-09 at 2.40.12 PM.png
>
>
> Hi,
> We are working on the spark 3.4.1 upgrade from spark 3.1.3, in our logging
> system, we are using logback framework, it is working with spark 3.1.3 since
> it is using log4j 1.x. However, when we upgrade spark to 3.4.1, based on the
> [release
> notes|https://spark.apache.org/docs/latest/core-migration-guide.html], spark
> is migrating from log4j 2.x from log4j 1.x, the way we are replacing the
> log4j with logback is causing build failures in spark master start process.
> Error: Unable to initialize main class org.apache.spark.deploy.master.Master
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/logging/log4j/core/Filter
> In our current approach, we are using log4j-over-slf4j to replace the
> log4j-core, it is only applicable to log4j 1.x library. And there is no
> log4j-over-slf4j for log4j 2.x out there yet. (please correct me if I am
> wrong).
> I am also curious that why spark choose to use log4j 2.x instead of using
> SPI, which gives the users less flexibility to choose whatever logger
> implementation they want to use.
> I want to share this issue and see if anyone else has been reported this and
> if there is any work-around or alternative solutions for it. Any suggestions
> are appreciated, thanks.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
