I resolve this issue finally by adding --conf spark.executor.extraClassPath= snakeyaml-1.10.jar
2015-10-16 22:57 GMT+08:00 Rex Xiong <bycha...@gmail.com>: > Hi folks, > > In my spark application, executor task depends on snakeyaml-1.10.jar > I build it with Maven and it works fine: > spark-submit --master local --jars d:\snakeyaml-1.10.jar ........... > > But when I try to run it in yarn, I have issue, it seems spark executor > cannot find the jar file: > spark-submit --master yarn-cluster --jars hdfs://....../snakeyaml-1.10.jar > .......... > > java.lang.NoSuchMethodError: > org.yaml.snakeyaml.Yaml.<init>(Lorg/yaml/snakeyaml/constructor/BaseConstructor;)V > > I check one executor container folder, snakeyaml-1.10.jar has been > successfully downloaded, and in spark driver page, in environment tab, > spark.yarn.secondary.jars > also contains snakeyaml-1.10.jar > > I have no idea why it doesn't work. Could some one help to take a look? > > Thanks > >
