Hi, I am new to spark (and Java + Scala also) and still trying to setup the standalone mode spark with livy job manager. I am not able to run the examples given in the livy example directory for any of the languages. I am sure I am missing something here, but not able to figure out what's missing. The examples are working properly if run from Spark direcoty.
- I am using Ubuntu 16.04.3. - Other Installed versions : - 1. Spark Version : 2.2.1 - 2. Java Version : - openjdk version "1.8.0_162" - OpenJDK Runtime Environment (build 1.8.0_162-8u162-b12-0ubuntu0.16.04.2-b12) - OpenJDK 64-Bit Server VM (build 25.162-b12, mixed mode) - 3. Livy Version : 0.6.0-incubating-SNAPSHOT *Error in Java Example : * 1) setting java classpaths : JAVA_CP=<path>/spark-core_<version>.jar:<path>/livy-api-0.6.0-incubating-SNAPSHOT.jar:<path>/livy-client-http-0.6.0-incubating-SNAPSHOT.jar:<path>/livy-examples-0.6.0-incubating-SNAPSHOT.jar:<path>/slf4j-1.7.25/slf4j-api-1.7.25.jar:<path>/slf4j-1.7.25/slf4j-simple-1.7.25.jar 2) running example from livy home direcoty : java -cp $JAVA_CP org.apache.livy.examples.PiApp http://localhost:8998 2 3) Error : Uploading livy-example jar to the SparkContext... Exception in thread "main" java.util.concurrent.ExecutionException: java.io.IOException: Bad Request: "requirement failed: Local path /root/.livy-sessions/f2169a4c-41eb-4b1d-8b77-c9eaf853b949/livy-examples-0.6.0-incubating-SNAPSHOT.jar cannot be added to user sessions." at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at org.apache.livy.examples.PiApp.main(PiApp.java:87) Caused by: java.io.IOException: Bad Request: "requirement failed: Local path /root/.livy-sessions/f2169a4c-41eb-4b1d-8b77-c9eaf853b949/livy-examples-0.6.0-incubating-SNAPSHOT.jar cannot be added to user sessions." at org.apache.livy.client.http.LivyConnection.sendRequest(LivyConnection.java:229) at org.apache.livy.client.http.LivyConnection.post(LivyConnection.java:192) at org.apache.livy.client.http.HttpClient$2.call(HttpClient.java:152) at org.apache.livy.client.http.HttpClient$2.call(HttpClient.java:149) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 4) I have added the path /root/.livy-sessions/ in the whitelist config param but that did not solve the error. *Error in Python example : * 1) python ./examples/src/main/python/pi_app.py http://localhost:8998 2 2) Error in the execution : Exception: org.apache.livy.shaded.kryo.kryo.KryoException: Encountered unregistered class ID: 254 org.apache.livy.shaded.kryo.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:119) org.apache.livy.shaded.kryo.kryo.Kryo.readClass(Kryo.java:656) org.apache.livy.shaded.kryo.kryo.Kryo.readClassAndObject(Kryo.java:767) org.apache.livy.client.common.Serializer.deserialize(Serializer.java:63) org.apache.livy.rsc.driver.BypassJob.call(BypassJob.java:39) org.apache.livy.rsc.driver.BypassJob.call(BypassJob.java:27) org.apache.livy.rsc.driver.JobWrapper.call(JobWrapper.java:57) org.apache.livy.rsc.driver.BypassJobWrapper.call(BypassJobWrapper.java:42) org.apache.livy.rsc.driver.BypassJobWrapper.call(BypassJobWrapper.java:27) java.util.concurrent.FutureTask.run(FutureTask.java:266) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:748) Can anybody please tell me what is missing here in both the examples? -- Thanks & Regards, Ruchir