RE: ClassNotFoundException while unmarshalling a remote RDD on Spark 1.5.1
Ok, it just seems to be an issue with the syntax of the spark-submit command. It should be : spark-submit --queue default \ --class com.my.Launcher \ --deploy-mode cluster \ --master yarn-cluster \ --driver-java-options "-Dfile.encoding=UTF-8" \ --jars /home/user/hibernate-validator-5.2.2.Final.jar \ --driver-class-path hibernate-validator-5.2.2.Final.jar \ --conf "spark.executor.extraClassPath=hibernate -validator-5.2.2.Final.jar" \ /home/user/uberjar-job.jar I also have to add some others jars, like jboss-logging to meet the needs of hibernate-validator. De : PICARD Damien (EXT) AssuResPriSmsAts Envoyé : lundi 11 septembre 2017 08:53 À : 'user@spark.apache.org' Objet : ClassNotFoundException while unmarshalling a remote RDD on Spark 1.5.1 Hi ! I'm facing a Classloader problem using Spark 1.5.1 I use javax.validation and hibernate validation annotations on some of my beans : @NotBlank @Valid private String attribute1 ; @Valid private String attribute2 ; When Spark tries to unmarshall these beans (after a remote RDD), I get the ClassNotFoundException : 17/09/07 09:19:25 INFO storage.BlockManager: Found block rdd_8_1 remotely 17/09/07 09:19:25 ERROR executor.Executor: Exception in task 3.0 in stage 2.0 (TID 6) java.lang.ClassNotFoundException: org.hibernate.validator.constraints.NotBlank at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at java.io.ObjectInputStream.resolveProxyClass(ObjectInputStream.java:700) at java.io.ObjectInputStream.readProxyDesc(ObjectInputStream.java:1566) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1781) ... Indeed, it means that the annotation class is not found, because it is not in the classpath. Why ? I don't know, because I make a uber JAR that contains this class. I suppose that at the time the job tries to unmarshall the RDD, the uber jar is not loaded. So, I try to add the hibernate JAR to the class loader manually, using this spark-submit command : spark-submit --queue default \ --class com.my.Launcher \ --deploy-mode cluster \ --master yarn-cluster \ --driver-java-options "-Dfile.encoding=UTF-8" \ --jars /home/user/hibernate-validator-5.2.2.Final.jar \ --driver-class-path /home/user/hibernate-validator-5.2.2.Final.jar \ --conf "spark.executor.extraClassPath=/home/user/hibernate-validator-5.2.2.Final.jar" \ /home/user/uberjar-job.jar Without effects. So, is there a way to add this class to the classloader ? Thank you in advance. Damien = Ce message et toutes les pieces jointes (ci-apres le "message") sont confidentiels et susceptibles de contenir des informations couvertes par le secret professionnel. Ce message est etabli a l'intention exclusive de ses destinataires. Toute utilisation ou diffusion non autorisee interdite. Tout message electronique est susceptible d'alteration. La SOCIETE GENERALE et ses filiales declinent toute responsabilite au titre de ce message s'il a ete altere, deforme falsifie. = This message and any attachments (the "message") are confidential, intended solely for the addresses, and may contain legally privileged information. Any unauthorized use or dissemination is prohibited. E-mails are susceptible to alteration. Neither SOCIETE GENERALE nor any of its subsidiaries or affiliates shall be liable for the message if altered, changed or falsified. =
ClassNotFoundException while unmarshalling a remote RDD on Spark 1.5.1
Hi ! I'm facing a Classloader problem using Spark 1.5.1 I use javax.validation and hibernate validation annotations on some of my beans : @NotBlank @Valid private String attribute1 ; @Valid private String attribute2 ; When Spark tries to unmarshall these beans (after a remote RDD), I get the ClassNotFoundException : 17/09/07 09:19:25 INFO storage.BlockManager: Found block rdd_8_1 remotely 17/09/07 09:19:25 ERROR executor.Executor: Exception in task 3.0 in stage 2.0 (TID 6) java.lang.ClassNotFoundException: org.hibernate.validator.constraints.NotBlank at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at java.io.ObjectInputStream.resolveProxyClass(ObjectInputStream.java:700) at java.io.ObjectInputStream.readProxyDesc(ObjectInputStream.java:1566) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1781) ... Indeed, it means that the annotation class is not found, because it is not in the classpath. Why ? I don't know, because I make a uber JAR that contains this class. I suppose that at the time the job tries to unmarshall the RDD, the uber jar is not loaded. So, I try to add the hibernate JAR to the class loader manually, using this spark-submit command : spark-submit --queue default \ --class com.my.Launcher \ --deploy-mode cluster \ --master yarn-cluster \ --driver-java-options "-Dfile.encoding=UTF-8" \ --jars /home/user/hibernate-validator-5.2.2.Final.jar \ --driver-class-path /home/user/hibernate-validator-5.2.2.Final.jar \ --conf "spark.executor.extraClassPath=/home/user/hibernate-validator-5.2.2.Final.jar" \ /home/user/uberjar-job.jar Without effects. So, is there a way to add this class to the classloader ? Thank you in advance. Damien = Ce message et toutes les pieces jointes (ci-apres le "message") sont confidentiels et susceptibles de contenir des informations couvertes par le secret professionnel. Ce message est etabli a l'intention exclusive de ses destinataires. Toute utilisation ou diffusion non autorisee interdite. Tout message electronique est susceptible d'alteration. La SOCIETE GENERALE et ses filiales declinent toute responsabilite au titre de ce message s'il a ete altere, deforme falsifie. = This message and any attachments (the "message") are confidential, intended solely for the addresses, and may contain legally privileged information. Any unauthorized use or dissemination is prohibited. E-mails are susceptible to alteration. Neither SOCIETE GENERALE nor any of its subsidiaries or affiliates shall be liable for the message if altered, changed or falsified. =
[no subject]
Hi ! I'm facing a Classloader problem using Spark 1.5.1 I use javax.validation and hibernate validation annotations on some of my beans : @NotBlank @Valid private String attribute1 ; @Valid private String attribute2 ; When Spark tries to unmarshall these beans (after a remote RDD), I get the ClassNotFoundException : 17/09/07 09:19:25 INFO storage.BlockManager: Found block rdd_8_1 remotely 17/09/07 09:19:25 ERROR executor.Executor: Exception in task 3.0 in stage 2.0 (TID 6) java.lang.ClassNotFoundException: org.hibernate.validator.constraints.NotBlank at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) at java.io.ObjectInputStream.resolveProxyClass(ObjectInputStream.java:700) at java.io.ObjectInputStream.readProxyDesc(ObjectInputStream.java:1566) at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1518) at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1781) ... Indeed, it means that the annotation class is not found, because it is not in the classpath. Why ? I don't know, because I make a uber JAR that contains this class. I suppose that at the time the job tries to unmarshall the RDD, the uber jar is not loaded. So, I try to add the hibernate JAR to the class loader manually, using this spark-submit command : spark-submit --queue default \ --class com.my.Launcher \ --deploy-mode cluster \ --master yarn-cluster \ --driver-java-options "-Dfile.encoding=UTF-8" \ --jars /home/user/hibernate-validator-5.2.2.Final.jar \ --driver-class-path /home/user/hibernate-validator-5.2.2.Final.jar \ --conf "spark.executor.extraClassPath=/home/user/hibernate-validator-5.2.2.Final.jar" \ /home/user/uberjar-job.jar Without effects. So, is there a way to add this class to the classloader ? Thank you in advance. Damien = Ce message et toutes les pieces jointes (ci-apres le "message") sont confidentiels et susceptibles de contenir des informations couvertes par le secret professionnel. Ce message est etabli a l'intention exclusive de ses destinataires. Toute utilisation ou diffusion non autorisee interdite. Tout message electronique est susceptible d'alteration. La SOCIETE GENERALE et ses filiales declinent toute responsabilite au titre de ce message s'il a ete altere, deforme falsifie. = This message and any attachments (the "message") are confidential, intended solely for the addresses, and may contain legally privileged information. Any unauthorized use or dissemination is prohibited. E-mails are susceptible to alteration. Neither SOCIETE GENERALE nor any of its subsidiaries or affiliates shall be liable for the message if altered, changed or falsified. =