Jars distributed using --jars are not added to the system classpath, so log4j cannot see them.
To work around that, you need to manually add the *name* jar to the driver executor classpaths: spark.driver.extraClassPath=some.jar spark.executor.extraClassPath=some.jar In client mode you should use spark.yarn.dist.jars instead of --jars, and change the driver classpath above to point to the local copy of the jar. On Wed, Aug 9, 2017 at 2:52 PM, Mikhailau, Alex <alex.mikhai...@mlb.com> wrote: > I have log4j json layout jars added via spark-submit on EMR > > > > /usr/lib/spark/bin/spark-submit --deploy-mode cluster --master yarn --jars > /home/hadoop/lib/jsonevent-layout-1.7.jar,/home/hadoop/lib/json-smart-1.1.1.jar > --driver-java-options "-XX:+AlwaysPreTouch -XX:MaxPermSize=6G" --class > com.mlbam.emr.XXX s3://xxx/aa/jars/ spark-job-assembly-1.4.1-SNAPSHOT.jar > ActionOnFailure=CONTINUE > > > > > > this is the process running on the executor: > > > > /usr/lib/jvm/java-1.8.0/bin/java -server -Xmx8192m -XX:+AlwaysPreTouch > -XX:MaxPermSize=6G > -Djava.io.tmpdir=/mnt/yarn/usercache/hadoop/appcache/application_1502310393755_0003/container_1502310393755_0003_01_000005/tmp > -Dspark.driver.port=32869 -Dspark.history.ui.port=18080 -Dspark.ui.port=0 > -Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/containers/application_1502310393755_0003/container_1502310393755_0003_01_000005 > -XX:OnOutOfMemoryError=kill %p > org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url > spark://CoarseGrainedScheduler@10.202.138.158:32869 --executor-id 3 > --hostname ip-10-202-138-98.mlbam.qa.us-east-1.bamgrid.net --cores 8 > --app-id application_1502310393755_0003 --user-class-path > file:/mnt/yarn/usercache/hadoop/appcache/application_1502310393755_0003/container_1502310393755_0003_01_000005/__app__.jar > --user-class-path > file:/mnt/yarn/usercache/hadoop/appcache/application_1502310393755_0003/container_1502310393755_0003_01_000005/jsonevent-layout-1.7.jar > --user-class-path > file:/mnt/yarn/usercache/hadoop/appcache/application_1502310393755_0003/container_1502310393755_0003_01_000005/json-smart-1.1.1.jar > > > > I see that jsonevent-layout-1.7.jar is passed as –user-class-path to the job > (see the above process), yet, I see the following log exception in my > stderr: > > > > log4j:ERROR Could not instantiate class > [net.logstash.log4j.JSONEventLayoutV1]. > > java.lang.ClassNotFoundException: net.logstash.log4j.JSONEventLayoutV1 > > > > > > Am I doing something wrong? > > > > Thank you, > > > > Alex -- Marcelo --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org
