Re: log4j pains

2016-03-10 Thread Tristan Nixon 
Hmmm… that should be right.

> On Mar 10, 2016, at 11:26 AM, Ashic Mahtab  wrote:
> 
> src/main/resources/log4j.properties
> 
> Subject: Re: log4j pains
> From: st...@memeticlabs.org
> Date: Thu, 10 Mar 2016 11:08:46 -0600
> CC: user@spark.apache.org
> To: as...@live.com
> 
> Where in the jar is the log4j.properties file?
> 
> On Mar 10, 2016, at 9:40 AM, Ashic Mahtab  <mailto:as...@live.com>> wrote:
> 
> 1. Fat jar with logging dependencies included. log4j.properties in fat jar. 
> Spark doesn't pick up the properties file, so uses its defaults.

RE: log4j pains

2016-03-10 Thread Ashic Mahtab 
src/main/resources/log4j.properties

Subject: Re: log4j pains
From: st...@memeticlabs.org
Date: Thu, 10 Mar 2016 11:08:46 -0600
CC: user@spark.apache.org
To: as...@live.com

Where in the jar is the log4j.properties file?
On Mar 10, 2016, at 9:40 AM, Ashic Mahtab  wrote:1. Fat jar 
with logging dependencies included. log4j.properties in fat jar. Spark doesn't 
pick up the properties file, so uses its defaults.

Re: log4j pains

2016-03-10 Thread Tristan Nixon 
Where in the jar is the log4j.properties file?

> On Mar 10, 2016, at 9:40 AM, Ashic Mahtab  wrote:
> 
> 1. Fat jar with logging dependencies included. log4j.properties in fat jar. 
> Spark doesn't pick up the properties file, so uses its defaults.

log4j pains

2016-03-10 Thread Ashic Mahtab 
Hello,I'm trying to use a custom log4j appender, with things specified in a 
log4j.properties file. Very little seems to work in this regard. Here's what 
I've tried:
1. Fat jar with logging dependencies included. log4j.properties in fat jar. 
Spark doesn't pick up the properties file, so uses its defaults.
2. Fat jar with logging dependencies included. Submitted the properties file 
like this:
 spark-submit  --files ./log4j.properties   --conf 
'spark.executor.extraJavaOptions=-Dlog4j.configuration=log4j.properties' 
fat.jarJust uses defaults. Doesn't use the submitted file.
3. Fat jar with logging dependencies included. Submitted the properties file 
like this:
 spark-submit  --files ./log4j.properties   --conf 
'spark.executor.extraJavaOptions=-Dlog4j.configuration=file:log4j.properties' 
fat.jarCrashes. Says log4j.properties file not found.
4. Fat jar with logging dependencies included. Submitted the properties file 
like this:
 spark-submit  --files ./log4j.properties   --conf 
'spark.executor.extraJavaOptions=-Dlog4j.configuration=file:/localfolder/log4j.properties'
 fat.jarCrashes. Says it could not find the appender stuff (JSonEventLayoutV1). 
That's something that's provided in the fat jar.
5. Fat jar with logging dependencies included. Submitted the properties file 
and logging jars like this: spark-submit  --files ./log4j.properties 
--jars 'comma,separated,list'   --conf 
'spark.executor.extraJavaOptions=-Dlog4j.configuration=file:/localfolder/log4j.properties'
 fat.jarCrashes. Says it could not find the appender stuff (JSonEventLayoutV1). 
That's something that's provided in the fat jar.

It seems only including the jars in extra class path is working. There seems to 
be a somewhat related issue here: 
https://issues.apache.org/jira/browse/SPARK-9826 , but that seems to have been 
resolved in 1.4.1. I'm using 1.5.2. Is this a regression? Or am I missing 
something?
Any help will be appreciated.
Thanks,Ashic.