Hello,
I'm new in Livy, and I have this problem.
When I run a batch job like this:
curl -X POST --data '{"file": "/user/spark/program.jar", "className":
"my.program"}' -H "Content-Type: application/json" LIVY-HOST:8998/batches
The program keeps runing for ever.
Searching in the logs I found the reason:
17/07/25 14:21:46 ERROR yarn.ApplicationMaster: User class threw exception:
java.lang.OutOfMemoryError: PermGen space
So in order to solve this issue I increased the executor and driver memory:
"driverMemory":"15g","executorMemory":"15g". But I still seen this error.
So I tried running this job directly using spark-submit and it worked
perfectly, no memory error.
spark-submit --class my.program --deploy-mode cluster --conf spark.master=yarn
hdfs://HDFS_NN:8020/user/spark/program.jar
I'm using Livy 0.4.0-SNAPSHOT on Spark 2.1.0.
Joaquín Silva | Pentagon Security & AKAINIX
Av. Kennedy 4.700, Piso 10, Of. 1002, Edificio New Century, Vitacura | Código
Postal (ZIP Code) 7561127
Cel: (56-9) 6304 2498
