I am trying to use LivyScalaClient 0.7.1 with HDP 3.1.1. My Spark Application
is packaged into a big self-contained jar file that does not have dependency on
any other jar files, and the jar file is put into spark.yarn.archive directory.
The job fails in Yarn with the following exception,
21/08/22 11:41:07 INFO JobWrapper: Failed to run job
4aafeaad-b331-4f55-9cf5-5976f542a99c
org.apache.livy.shaded.kryo.kryo.KryoException: Unable to find class:
org.apache.livy.scalaapi.LivyScalaClient$$anon$1
at
org.apache.livy.shaded.kryo.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:138)
at
org.apache.livy.shaded.kryo.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:115)
at org.apache.livy.shaded.kryo.kryo.Kryo.readClass(Kryo.java:656)
at
org.apache.livy.shaded.kryo.kryo.Kryo.readClassAndObject(Kryo.java:767)
at
org.apache.livy.client.common.Serializer.deserialize(Serializer.java:63)
at org.apache.livy.rsc.driver.BypassJob.call(BypassJob.java:39)
at org.apache.livy.rsc.driver.BypassJob.call(BypassJob.java:27)
at org.apache.livy.rsc.driver.JobWrapper.call(JobWrapper.java:64)
at
org.apache.livy.rsc.driver.BypassJobWrapper.call(BypassJobWrapper.java:45)
at
org.apache.livy.rsc.driver.BypassJobWrapper.call(BypassJobWrapper.java:27)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException:
org.apache.livy.scalaapi.LivyScalaClient$$anon$1
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:348)
at
org.apache.livy.shaded.kryo.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:136)
... 13 more
I am sure the Spark application jar file contains
org.apache.livy.scalaapi.LivyScalaClient.
Thanks,
Feng
