Re: Invalid signature file digest for Manifest main attributes with spark job built using maven
Sean,
Thanks. That worked.
Kevin
On Mon, Sep 15, 2014 at 3:37 PM, Sean Owen wrote:
> This is more of a Java / Maven issue than Spark per se. I would use
> the shade plugin to remove signature files in your final META-INF/
> dir. As Spark does, in its :
>
>
>
> *:*
>
> org/datanucleus/**
> META-INF/*.SF
> META-INF/*.DSA
> META-INF/*.RSA
>
>
>
>
> On Mon, Sep 15, 2014 at 11:33 PM, kpeng1 wrote:
> > Hi All,
> >
> > I am trying to submit a spark job that I have built in maven using the
> > following command:
> > /usr/bin/spark-submit --deploy-mode client --class com.spark.TheMain
> > --master local[1] /home/cloudera/myjar.jar 100
> >
> > But I seem to be getting the following error:
> > Exception in thread "main" java.lang.SecurityException: Invalid signature
> > file digest for Manifest main attributes
> > at
> >
> sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVerifier.java:286)
> > at
> >
> sun.security.util.SignatureFileVerifier.process(SignatureFileVerifier.java:239)
> > at java.util.jar.JarVerifier.processEntry(JarVerifier.java:307)
> > at java.util.jar.JarVerifier.update(JarVerifier.java:218)
> > at java.util.jar.JarFile.initializeVerifier(JarFile.java:345)
> > at java.util.jar.JarFile.getInputStream(JarFile.java:412)
> > at
> sun.misc.URLClassPath$JarLoader$2.getInputStream(URLClassPath.java:775)
> > at sun.misc.Resource.cachedInputStream(Resource.java:77)
> > at sun.misc.Resource.getByteBuffer(Resource.java:160)
> > at java.net.URLClassLoader.defineClass(URLClassLoader.java:436)
> > at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> > at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> > at java.security.AccessController.doPrivileged(Native Method)
> > at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> > at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> > at java.lang.Class.forName0(Native Method)
> > at java.lang.Class.forName(Class.java:270)
> > at
> org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:289)
> > at
> org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)
> > at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> >
> >
> > Here is the pom file I am using to build the jar:
> > http://maven.apache.org/POM/4.0.0";
> > xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
> > xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> > http://maven.apache.org/maven-v4_0_0.xsd";>
> > 4.0.0
> > com.spark
> > myjar
> > 0.0.1-SNAPSHOT
> > ${project.artifactId}
> > My wonderfull scala app
> > 2010
> >
> >
> > My License
> > http://
> > repo
> >
> >
> >
> >
> > cdh5.1.0
> > 1.6
> > 1.6
> > UTF-8
> > 2.10
> > 2.10.4
> >
> >
> >
> >
> > scala-tools.org
> > Scala-tools Maven2 Repository
> > https://oss.sonatype.org/content/repositories/snapshots/
>
> >
> >
> > maven-hadoop
> > Hadoop Releases
> >
> > https://repository.cloudera.com/content/repositories/releases/
>
> >
> >
> > cloudera-repos
> > Cloudera Repos
> > https://repository.cloudera.com/artifactory/cloudera-repos/
>
> >
> >
> >
> >
> > scala-tools.org
> > Scala-tools Maven2 Repository
> > https://oss.sonatype.org/content/repositories/snapshots/
>
> >
> >
> >
> >
> >
> > org.scala-lang
> > scala-library
> > ${scala.version}
> >
> >
> > org.apache.spark
> > spark-core_2.10
> > 1.0.0-${cdh.version}
> >
> >
> > org.apache.spark
> > spark-tools_2.10
> > 1.0.0-${cdh.version}
> >
> >
> > org.apache.spark
> > spark-streaming-flume_2.10
> > 1.0.0-${cdh.version}
> >
> >
> > org.apache.spark
> > spark-streaming_2.10
> > 1.0.0-${cdh.version}
> >
> >
> > org.apache.flume
> > flume-ng-sdk
> > 1.5.0-${cdh.version}
> >
> >
> >
> > io.netty
> > netty
> >
> >
> >
> >
> > org.apache.flume
> > flume-ng-core
> > 1.5.0-${cdh.version}
> >
> >
> >
> > io.netty
> > netty
> >
> >
> >
> >
> > org.apache.hbase
> > hbase-client
> > 0.98.1-${cdh.version}
> >
> >
> >
> > io.netty
> > netty
> >
> >
> >
> >
> > org.apache.hadoop
> > hadoop-client
> > 2.3.0-${cdh.version}
> >
> >
> >
> >
> >
> > junit
> > junit
> > 4
Re: Invalid signature file digest for Manifest main attributes with spark job built using maven
This is more of a Java / Maven issue than Spark per se. I would use
the shade plugin to remove signature files in your final META-INF/
dir. As Spark does, in its :
*:*
org/datanucleus/**
META-INF/*.SF
META-INF/*.DSA
META-INF/*.RSA
On Mon, Sep 15, 2014 at 11:33 PM, kpeng1 wrote:
> Hi All,
>
> I am trying to submit a spark job that I have built in maven using the
> following command:
> /usr/bin/spark-submit --deploy-mode client --class com.spark.TheMain
> --master local[1] /home/cloudera/myjar.jar 100
>
> But I seem to be getting the following error:
> Exception in thread "main" java.lang.SecurityException: Invalid signature
> file digest for Manifest main attributes
> at
> sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVerifier.java:286)
> at
> sun.security.util.SignatureFileVerifier.process(SignatureFileVerifier.java:239)
> at java.util.jar.JarVerifier.processEntry(JarVerifier.java:307)
> at java.util.jar.JarVerifier.update(JarVerifier.java:218)
> at java.util.jar.JarFile.initializeVerifier(JarFile.java:345)
> at java.util.jar.JarFile.getInputStream(JarFile.java:412)
> at
> sun.misc.URLClassPath$JarLoader$2.getInputStream(URLClassPath.java:775)
> at sun.misc.Resource.cachedInputStream(Resource.java:77)
> at sun.misc.Resource.getByteBuffer(Resource.java:160)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:436)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:270)
> at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:289)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
>
> Here is the pom file I am using to build the jar:
> http://maven.apache.org/POM/4.0.0";
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/maven-v4_0_0.xsd";>
> 4.0.0
> com.spark
> myjar
> 0.0.1-SNAPSHOT
> ${project.artifactId}
> My wonderfull scala app
> 2010
>
>
> My License
> http://
> repo
>
>
>
>
> cdh5.1.0
> 1.6
> 1.6
> UTF-8
> 2.10
> 2.10.4
>
>
>
>
> scala-tools.org
> Scala-tools Maven2 Repository
> https://oss.sonatype.org/content/repositories/snapshots/
>
>
> maven-hadoop
> Hadoop Releases
>
> https://repository.cloudera.com/content/repositories/releases/
>
>
> cloudera-repos
> Cloudera Repos
> https://repository.cloudera.com/artifactory/cloudera-repos/
>
>
>
>
> scala-tools.org
> Scala-tools Maven2 Repository
> https://oss.sonatype.org/content/repositories/snapshots/
>
>
>
>
>
> org.scala-lang
> scala-library
> ${scala.version}
>
>
> org.apache.spark
> spark-core_2.10
> 1.0.0-${cdh.version}
>
>
> org.apache.spark
> spark-tools_2.10
> 1.0.0-${cdh.version}
>
>
> org.apache.spark
> spark-streaming-flume_2.10
> 1.0.0-${cdh.version}
>
>
> org.apache.spark
> spark-streaming_2.10
> 1.0.0-${cdh.version}
>
>
> org.apache.flume
> flume-ng-sdk
> 1.5.0-${cdh.version}
>
>
>
> io.netty
> netty
>
>
>
>
> org.apache.flume
> flume-ng-core
> 1.5.0-${cdh.version}
>
>
>
> io.netty
> netty
>
>
>
>
> org.apache.hbase
> hbase-client
> 0.98.1-${cdh.version}
>
>
>
> io.netty
> netty
>
>
>
>
> org.apache.hadoop
> hadoop-client
> 2.3.0-${cdh.version}
>
>
>
>
>
> junit
> junit
> 4.11
> test
>
>
> org.specs2
> specs2_${scala.tools.version}
> 1.13
> test
>
>
> org.scalatest
> scalatest_${scala.tools.version}
> 2.0.M6-SNAP8
> test
>
>
>
>
> src/main/scala
> src/test/scala
>
>
>
> net.alchim31.maven
> scala-maven-plugin
> 3.1.3
>
>
>
> compile
> testCompile
>
