Github user ejsarge-gr commented on the pull request:
https://github.com/apache/spark/pull/4998#issuecomment-118378860
Would we be able to document this better or re-examine it a little?
The problem is that log4j 1.2.17 is shaded into the assembly jar. Thus, to
override it we
Github user liorchaga closed the pull request at:
https://github.com/apache/spark/pull/4998
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is
Github user liorchaga commented on the pull request:
https://github.com/apache/spark/pull/4998#issuecomment-81544184
I'm withdrawing the pull request.
We do not want to break compatibility to the community.
Instead, we are using SPARK_CLASSPATH to add the log4j 2.x and
Github user liorchaga commented on the pull request:
https://github.com/apache/spark/pull/4998#issuecomment-80944189
@srowen , the 1.2 - 2.x bridge should work (I would verify this today).
But keep in mind it would require migrating log4j configuration to 2.x. Are we
sure we want to
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/4998#issuecomment-80938223
@tsliwowicz Have a read above, where I'm describing why this is more than
just a build profile. My principal concern is as I say above: if log4j 1.2 and
2 are mutually
Github user liorchaga commented on the pull request:
https://github.com/apache/spark/pull/4998#issuecomment-80979001
log4j12-api bridge is working properly.
Our executor code is using log4j 1.2.7, and this jar is included in the
jars provided to sparkContext. The spark
Github user tsliwowicz commented on the pull request:
https://github.com/apache/spark/pull/4998#issuecomment-80878911
@srowen We don't know of an option to run side by side with two log4j
versions. It conflicts on both slf4j and log4j classes. In any case, I believe
it won't create a
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/4998#issuecomment-80947530
Yeah that's what I am hoping might work, to remove log4j 1.2 and replace
with log4j 2 + 1.2-to-2 bridge, and also use the slfj4-to-log4j2 bridge, and
update Spark itself
Github user tsliwowicz commented on the pull request:
https://github.com/apache/spark/pull/4998#issuecomment-78868836
@srowen this is in essence the same as the ability to control hadoop
version only through classpath manipulation. Since there are many flavors, its
being done by
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/4998#issuecomment-78893597
I think the difference is that Hadoop profiles build Spark for different,
but fixed, deployment contexts. This is trying to accommodate two different
types of _user_ app.
Github user AmplabJenkins commented on the pull request:
https://github.com/apache/spark/pull/4998#issuecomment-78476242
Can one of the admins verify this patch?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/4998#issuecomment-78500551
Does it actually cause a problem? (I wouldn't be surprised, but I'm just
asking.) Can you use slf4j, or do you need log4j 2? can you use the log4j 2 -
log4j 1.2 adapter?
Github user srowen commented on a diff in the pull request:
https://github.com/apache/spark/pull/4998#discussion_r26318113
--- Diff: pom.xml ---
@@ -1706,6 +1706,43 @@
jline.groupidjline/jline.groupid
/properties
/profile
+profile
+
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/4998#issuecomment-78498110
No, we do not want a new profile for this. What problem are you trying to
solve? Spark uses SLF4J, not a particular logging framework. However it
activates log4j 1 by
Github user liorchaga commented on the pull request:
https://github.com/apache/spark/pull/4998#issuecomment-78502617
The problem is having both log4j-over-slf4j and slf4j-log4j12 in the
classpath. To get it working with log4j 1.2 we need to exclude slf4j-log4j12.
Also, having both
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/4998#issuecomment-78513631
Hm, the bridge I was thinking of is for 1.2 - 2, not the other way around.
I was thinking of
http://logging.apache.org/log4j/log4j-2.1/log4j-to-slf4j/index.html as
Github user liorchaga commented on the pull request:
https://github.com/apache/spark/pull/4998#issuecomment-78525274
On second thought, correct me if I'm wrong, in such case spark users would
have to migrate their log config.
---
If your project is set up for it, you can reply to
Github user srowen commented on the pull request:
https://github.com/apache/spark/pull/4998#issuecomment-78534488
@liorchaga I was thinking that Spark would just use log4j 2, and include
the log4j 1.2 bridge, so that no matter what, user apps would see the classes
they need. I don't
Github user liorchaga commented on the pull request:
https://github.com/apache/spark/pull/4998#issuecomment-78523952
This would require all existing spark deployments to add the bridge to the
classpath (or to rebuild the distribution with a profile that includes this
dependency),
Github user liorchaga commented on the pull request:
https://github.com/apache/spark/pull/4998#issuecomment-78499766
We are using log4j 2.x in our executor code. I tried adding log4j 2.x
dependencies using spark.executor.extraClassPath, but then we have both log4j
1.2 and 2.x in our
Github user liorchaga commented on the pull request:
https://github.com/apache/spark/pull/4998#issuecomment-78515735
Tried it with extraClassPath. I provided through SPARK_EXECUTOR_OPTS the
log4j.configurationFile property for Log4j 2.x.
In the log config, I declared two file
GitHub user liorchaga opened a pull request:
https://github.com/apache/spark/pull/4998
[SPARK-6305] add log4j2 profile and prevent log4j 1.2 shading
* log4j and slf4j-log4j12 compile dependencies in submodules removed
* log4j2 profile added with log4h12 and slf4j-log4j12
22 matches
Mail list logo
