[
https://issues.apache.org/jira/browse/SPARK-4997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14261285#comment-14261285
]
Josh Rosen commented on SPARK-4997:
-----------------------------------
I've noticed this issue on {{spark-ec2}} because that's generally the only test
environment that I personally use, but it might be a more broad issue. In
SPARK-2913 (the EC2 issue), the problem seems to be that the Hadoop directory
is placed ahead of Spark on the classpath. My proposal was to just force
Spark's file to appear at the head of the classpath, but I think that someone
voiced a concern that this could break users' ability to use a custom
log4j.properties file from their application JAR (or something like that).
At a minimum, we should improve the documentation on the configuration page to
discuss classpath ordering issues, since now it's just a fairly simplistic
paragraph that just says "edit this file and it will work" and that's not the
whole story (in fact, there are a bunch of StackOverflow questions from users
who are confused about why the Spark log4j.properties file didn't take effect).
> Check if Spark's conf needs to be put ahead of Hadoop's (for log4j purposes)
> ----------------------------------------------------------------------------
>
> Key: SPARK-4997
> URL: https://issues.apache.org/jira/browse/SPARK-4997
> Project: Spark
> Issue Type: Bug
> Components: EC2
> Affects Versions: 1.2.0
> Reporter: Nicholas Chammas
> Priority: Minor
>
> As originally proposed in this PR: https://github.com/mesos/spark-ec2/pull/55
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
