Apologies, outlook for mac is ridiculous. Copy and paste the original below:
-
I’m running into a strange issue with trying to use a custom Log4j layout for
Spark (1.6.1) on YARN (CDH). The layout is:
https://github.com/michaeltandy/log4j-json
If I use a log4j.properties file (supplied with --files) with:
log4j.appender.consolejson=org.apache.log4j.ConsoleAppender
log4j.appender.consolejson.target=System.err
log4j.appender.consolejson.layout=uk.me.mjt.log4jjson.SimpleJsonLayout
And supply the log4j-json.1.0.jar with ‘--jars’ to spark-submit, the driver and
executors throw an exception right at the start of the log file:
log4j:ERROR Could not instantiate class [uk.me.mjt.log4jjson.SimpleJsonLayout].
java.lang.ClassNotFoundException: uk.me.mjt.log4jjson.SimpleJsonLayout
However, a simple spark job that does something like:
sc.parallelize(List(1,2,3)).foreach(i =>
{Class.forName("uk.me.mjt.log4jjson.SimpleJsonLayout")})
Doesn’t throw an error. So the class is being loaded, but just not in time for
Log4j to use it.
I've tried a few different options trying to get it to work (including it in
the YARN application classpath, spark executor classpaths etc) and they all
produce the same results. The only thing that seems to work is building a
custom spark-assembly with the maven dependency included in core/pom.xml. This
way, the layout is included in the spark assembly jar, and I get the JSON log
output desired.
Is there a classloading issue on Log4j when using --jars? I can't imagine why
it works with bundling in spark-assembly, but doesn't work with --jars.
From: Ted Yu
Date: Friday, 22 April 2016 at 14:55
To: Andrew Rowson
Cc: "user@spark.apache.org"
Subject: Re: Custom Log4j layout on YARN = ClassNotFoundException
There is not much in the body of email.
Can you elaborate what issue you encountered ?
Thanks
On Fri, Apr 22, 2016 at 2:27 AM, Rowson, Andrew G. (TR Technology & Ops)
wrote:
This e-mail is for the sole use of the intended recipient and contains
information that may be privileged and/or confidential. If you are not an
intended recipient, please notify the sender by return e-mail and delete this
e-mail and any attachments. Certain required legal entity disclosures can be
accessed on our website.<http://site.thomsonreuters.com/site/disclosures/>
-- Forwarded message --
From: "Rowson, Andrew G. (TR Technology & Ops)"
To: "user@spark.apache.org"
Cc:
Date: Fri, 22 Apr 2016 10:27:53 +0100
Subject: Custom Log4j layout on YARN = ClassNotFoundException
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
smime.p7s
Description: S/MIME cryptographic signature
