Hi
I have tried to run my Spark Application using oozie on Hadoop cluster using Yarn but After some time my Oozie launcher MAPREDUCE application show Error : java.lang.OutOfMemoryError: PermGen space. ERROR [sparkDriver-akka.actor.default-dispatcher-35] akka.actor.ActorSystemImpl: Uncaught fatal error from thread [sparkDriver-akka.actor.default-dispatcher-24] shutting down ActorSystem [sparkDriver] java.lang.OutOfMemoryError: PermGen space at sun.misc.Unsafe.defineClass(Native Method) at sun.reflect.ClassDefiner.defineClass(ClassDefiner.java:63) at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:399) at sun.reflect.MethodAccessorGenerator$1.run(MethodAccessorGenerator.java:396) at java.security.AccessController.doPrivileged(Native Method) at sun.reflect.MethodAccessorGenerator.generate(MethodAccessorGenerator.java:395) at sun.reflect.MethodAccessorGenerator.generateSerializationConstructor(MethodAccessorGenerator.java:113) at sun.reflect.ReflectionFactory.newConstructorForSerialization(ReflectionFactory.java:331) at java.io.ObjectStreamClass.getSerializableConstructor(ObjectStreamClass.java:1376) at java.io.ObjectStreamClass.access$1500(ObjectStreamClass.java:72) at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:493) at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:468) at java.security.AccessController.doPrivileged(Native Method) at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:468) at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:365) at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:602) at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990) at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798) at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) at akka.serialization.JavaSerializer$$anonfun$1.apply(Serializer.scala:136) at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57) at akka.serialization.JavaSerializer.fromBinary(Serializer.scala:136) at akka.serialization.Serialization$$anonfun$deserialize$1.apply(Serialization.scala:104) at scala.util.Try$.apply(Try.scala:161) at akka.serialization.Serialization.deserialize(Serialization.scala:98) at akka.remote.MessageSerializer$.deserialize(MessageSerializer.scala:23) my workflow.xml is : <workflow-app xmlns='uri:oozie:workflow:0.5' name='FaceDetection'> <start to='FaceDetection' /> <action name="FaceDetection"> <spark xmlns="uri:oozie:spark-action:0.1"> <job-tracker>ivcp-m01:8032</job-tracker> <name-node>hdfs://ivcp-m01:8020</name-node> master>yarn-client</master> <mode>client</mode> <name>Face Detection</name> <class>com.streaming.facedetection.streaming_facedetection</class> <jar>hdfs://ivcp-m01 :8020/user/master/oozie/lib/face_detection_streaming-assembly-3.0.jar</jar> <spark-opts>--num-executors 6 --conf spark.executor.extraLibraryPath=/opt/cloudera/parcels/CDH-5.5.0-1.cdh5.5.0.p0.8/lib/hadoop/lib/native --conf spark.authenticate=false --conf spark.dynamicAllocation.enabled=true --conf spark.dynamicAllocation.executorIdleTimeout=60 --conf spark.dynamicAllocation.minExecutors=0 --conf spark.dynamicAllocation.schedulerBacklogTimeout=1 --conf spark.eventLog.dir=hdfs://ivcp-m01:8020/user/spark/applicationHistory --conf spark.eventLog.enabled=true --conf spark.serializer=org.apache.spark.serializer.KryoSerializer --conf spark.shuffle.service.enabled=true --conf spark.shuffle.service.port=7337 --conf spark.yarn.historyServer.address=http://ivcp-m01:18088 --conf spark.yarn.jar=local:/opt/cloudera/parcels/CDH-5.5.2-1.cdh5.5.2.p0.4/lib/spark/lib/spark-assembly.jar</spark-opts> <arg>ivcp-m01:9092</arg> <arg>a-0</arg> <arg>output</arg> </spark> <ok to="end"/> <error to="fail"/> </action> <kill name="fail"> <message>Workflow failed, error message[${wf:errorMessage(wf:lastErrorNode())}] </message> </kill> <end name='end' /> </workflow-app> my job.properties : jobOutput=/user/master/output oozie.wf.application.path=/user/master/oozie/ oozie.use.system.libpath=true how can I solve it? Regards Prateek
