[
https://issues.apache.org/jira/browse/SPARK-14703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15247357#comment-15247357
]
Sean Owen commented on SPARK-14703:
-----------------------------------
To be more specific, I mean leaving log4j in the build so it can link
correctly, but then routing actual calls to another logging backend. That
should "work" but Spark logger manipulation will do nothing. (There's an
outside chance I've missed why just doing that doesn't quite work.)
Why logback BTW? doesn't that have the same problem, for any logging
implementation?
> Spark uses SLF4J, but actually relies quite heavily on Log4J
> ------------------------------------------------------------
>
> Key: SPARK-14703
> URL: https://issues.apache.org/jira/browse/SPARK-14703
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core, YARN
> Affects Versions: 1.6.0
> Environment: 1.6.0-cdh5.7.0, logback 1.1.3, yarn
> Reporter: Matthew Byng-Maddick
> Priority: Minor
> Labels: log4j, logback, logging, slf4j
> Attachments: spark-logback.patch
>
>
> We've built a version of Hadoop CDH-5.7.0 in house with logback as the SLF4J
> provider, in order to send hadoop logs straight to logstash (to handle with
> logstash/elasticsearch), on top of our existing use of the logback backend.
> In trying to start spark-shell I discovered several points where the fact
> that we weren't quite using a real L4J caused the sc not to be created or the
> YARN module not to exist. There are many more places where we should probably
> be wrapping the logging more sensibly, but I have a basic patch that fixes
> some of the worst offenders (at least the ones that stop the sparkContext
> being created properly).
> I'm prepared to accept that this is not a good solution and there probably
> needs to be some sort of better wrapper, perhaps in the Logging.scala class
> which handles this properly.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
