[
https://issues.apache.org/jira/browse/SPARK-2913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14104366#comment-14104366
]
Josh Rosen commented on SPARK-2913:
-----------------------------------
I've noticed that the Spark EC2 AMI adds ephemeral HDFS's configuration
directory to {{spark.executor.extraClassPath}}; since this is placed at the
head of the classpath, the executors end up using that log4j.properties file
instead of Spark's.
> Spark's log4j.properties should always appear ahead of Hadoop's on classpath
> ----------------------------------------------------------------------------
>
> Key: SPARK-2913
> URL: https://issues.apache.org/jira/browse/SPARK-2913
> Project: Spark
> Issue Type: Bug
> Affects Versions: 1.0.0, 1.0.2, 1.1.0
> Reporter: Josh Rosen
> Assignee: Josh Rosen
> Fix For: 1.1.0
>
>
> In the current {{compute-classpath}} scripts, the Hadoop conf directory may
> appear before Spark's conf directory in the computed classpath. This leads
> to Hadoop's log4j.properties being used instead of Spark's, preventing users
> from easily changing Spark's logging settings.
> To fix this, we should add a new classpath entry for Spark's log4j.properties
> file.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
