Re: How do i get a spark instance to use my log4j properties

[Demi Ben-Ari]

Hi Steve,
I wrote a blog post on the configuration of Spark that we've used,
including the log4j.properties:
http://progexc.blogspot.co.il/2014/12/spark-configuration-mess-solved.html
(What we did was to distribute the relevant *log4j.properties* file to all
of the slaves to the same location)

Hope this helps, Good luck!

Quoting from the text in the post:
---
The right way to pass the parameter is through the property:
“*spark.driver.extraJavaOptions*” and “*spark.executor.extraJavaOptions*”:
I’ve passed both the log4J configurations property and the parameter that I
needed for the configurations. (To the Driver I was able to pass only the
log4j configuration).
For example (was written in properties file passed in spark-submit with “
*—properties-file*”):
*“*
spark.driver.extraJavaOptions
–Dlog4j.configuration=file:///spark/conf/log4j.properties -
spark.executor.extraJavaOptions
–Dlog4j.configuration=file:///spark/conf/log4j.properties
-Dapplication.properties.file=hdfs:///some/path/on/hdfs/app.properties
spark.application.properties.file hdfs:///some/path/on/hdfs/app.properties
*“*

On Tue, Apr 12, 2016 at 7:51 PM, Steve Lewis  wrote:

> Ok I am stymied. I have tried everything I can think of to get spark to
> use my own version of
>
> log4j.properties
>
> In the launcher code - I launch a local instance from a Java application
>
> I say -Dlog4j.configuration=conf/log4j.properties
>
> where conf/log4j.properties is user.dir - no luck
>
> Spark always starts saying
>
> Using Spark's default log4j profile:
> org/apache/spark/log4j-defaults.properties
>
> I have a directory conf with my log4j.properties there but it seems to be
> ignored
>
> I use maven and an VERY RELUCTANT to edit the spark jars
>
> I know this point has been discussed here before but I do not see a clean
> answer
>
>
>
>
>
>
>


-- 
Best regards,
Demi Ben-Ari 
Entrepreneur @ Stealth Mode Startup
Twitter: @demibenari 

How do i get a spark instance to use my log4j properties

[Steve Lewis]

Ok I am stymied. I have tried everything I can think of to get spark to use
my own version of

log4j.properties

In the launcher code - I launch a local instance from a Java application

I say -Dlog4j.configuration=conf/log4j.properties

where conf/log4j.properties is user.dir - no luck

Spark always starts saying

Using Spark's default log4j profile:
org/apache/spark/log4j-defaults.properties

I have a directory conf with my log4j.properties there but it seems to be
ignored

I use maven and an VERY RELUCTANT to edit the spark jars

I know this point has been discussed here before but I do not see a clean
answer