Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/4998#issuecomment-80938223
@tsliwowicz Have a read above, where I'm describing why this is more than
just a build profile. My principal concern is as I say above: if log4j 1.2 and
2 are mutually incompatible, then making a build with this profile means that
_user apps_ become mutually incompatible between this and a "normal" build of
Spark. That seems too surprising to advertise as just another build option like
the others. It's not really a solution.
(Doesn't this leave 1.2 and 2 on the runtime classpath? how does that end
up working when the assembly is created?)
I'm not clear if my suggestion above has been tried: what happens when
log4j 2 and log4j 2's 1.2-to-2 bridge artifacts are built into Spark? This is
_not_ the same as just putting log4j 1.2 and 2 on the classpath. It seems like
log4j 2 is trying to provide this as a way to do both at once. I don't know if
it works, but that *might* be a solution; config format may then be the problem.
This isn't a simple question of upgrading Spark for several reasons. Spark
runs user code and is prone to classpath leakage; Spark's choice unfortunately
affects user apps, unlike user apps. Shading is normally the answer to this,
and some key dependencies have been shaded to get them out of the way; the
"userClassPathFirst" mechanism is also a means to avoid some collisions.
Logging frameworks like log4j are however particularly tricky to deal with:
lots of packages use logging frameworks, and they want to be configured once,
in one file. Shading and/or multiple classloaders interfere with this.
Unless something like the above works, the state of things seems to be: use
slf4j in your app, or use log4j 1.2, or see if you can shade and separately
configure your logging in your app.
My guess is that at Spark 2.x, a lot of dependencies can and will be
updated, so log4j either goes away or moves to 2.x. And/or we figure out a way
to shade it that doesn't break it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
