Thanks, Marcelo. Will give it a shot tomorrow.

-Alex

On 8/9/17, 5:59 PM, "Marcelo Vanzin" <van...@cloudera.com> wrote:

    Jars distributed using --jars are not added to the system classpath,
    so log4j cannot see them.
    
    To work around that, you need to manually add the *name* jar to the
    driver executor classpaths:
    
    spark.driver.extraClassPath=some.jar
    spark.executor.extraClassPath=some.jar
    
    In client mode you should use spark.yarn.dist.jars instead of --jars,
    and change the driver classpath above to point to the local copy of
    the jar.
    
    
    On Wed, Aug 9, 2017 at 2:52 PM, Mikhailau, Alex <alex.mikhai...@mlb.com> 
wrote:
    > I have log4j json layout jars added via spark-submit on EMR
    >
    >
    >
    > /usr/lib/spark/bin/spark-submit --deploy-mode cluster --master yarn --jars
    > 
/home/hadoop/lib/jsonevent-layout-1.7.jar,/home/hadoop/lib/json-smart-1.1.1.jar
    > --driver-java-options "-XX:+AlwaysPreTouch -XX:MaxPermSize=6G" --class
    > com.mlbam.emr.XXX  s3://xxx/aa/jars/ spark-job-assembly-1.4.1-SNAPSHOT.jar
    > ActionOnFailure=CONTINUE
    >
    >
    >
    >
    >
    > this is the process running on the executor:
    >
    >
    >
    > /usr/lib/jvm/java-1.8.0/bin/java -server -Xmx8192m -XX:+AlwaysPreTouch
    > -XX:MaxPermSize=6G
    > 
-Djava.io.tmpdir=/mnt/yarn/usercache/hadoop/appcache/application_1502310393755_0003/container_1502310393755_0003_01_000005/tmp
    > -Dspark.driver.port=32869 -Dspark.history.ui.port=18080 -Dspark.ui.port=0
    > 
-Dspark.yarn.app.container.log.dir=/var/log/hadoop-yarn/containers/application_1502310393755_0003/container_1502310393755_0003_01_000005
    > -XX:OnOutOfMemoryError=kill %p
    > org.apache.spark.executor.CoarseGrainedExecutorBackend --driver-url
    > spark://CoarseGrainedScheduler@10.202.138.158:32869 --executor-id 3
    > --hostname ip-10-202-138-98.mlbam.qa.us-east-1.bamgrid.net --cores 8
    > --app-id application_1502310393755_0003 --user-class-path
    > 
file:/mnt/yarn/usercache/hadoop/appcache/application_1502310393755_0003/container_1502310393755_0003_01_000005/__app__.jar
    > --user-class-path
    > 
file:/mnt/yarn/usercache/hadoop/appcache/application_1502310393755_0003/container_1502310393755_0003_01_000005/jsonevent-layout-1.7.jar
    > --user-class-path
    > 
file:/mnt/yarn/usercache/hadoop/appcache/application_1502310393755_0003/container_1502310393755_0003_01_000005/json-smart-1.1.1.jar
    >
    >
    >
    > I see that jsonevent-layout-1.7.jar is passed as –user-class-path to the 
job
    > (see the above process), yet, I see the following log exception in my
    > stderr:
    >
    >
    >
    > log4j:ERROR Could not instantiate class
    > [net.logstash.log4j.JSONEventLayoutV1].
    >
    > java.lang.ClassNotFoundException: net.logstash.log4j.JSONEventLayoutV1
    >
    >
    >
    >
    >
    > Am I doing something wrong?
    >
    >
    >
    > Thank you,
    >
    >
    >
    > Alex
    
    
    
    -- 
    Marcelo
    


---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Reply via email to