This is more of a Java / Maven issue than Spark per se. I would use
the shade plugin to remove signature files in your final META-INF/
dir. As Spark does, in its <configuration>:
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>org/datanucleus/**</exclude>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
On Mon, Sep 15, 2014 at 11:33 PM, kpeng1 <kpe...@gmail.com> wrote:
> Hi All,
>
> I am trying to submit a spark job that I have built in maven using the
> following command:
> /usr/bin/spark-submit --deploy-mode client --class com.spark.TheMain
> --master local[1] /home/cloudera/myjar.jar 100
>
> But I seem to be getting the following error:
> Exception in thread "main" java.lang.SecurityException: Invalid signature
> file digest for Manifest main attributes
> at
> sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVerifier.java:286)
> at
> sun.security.util.SignatureFileVerifier.process(SignatureFileVerifier.java:239)
> at java.util.jar.JarVerifier.processEntry(JarVerifier.java:307)
> at java.util.jar.JarVerifier.update(JarVerifier.java:218)
> at java.util.jar.JarFile.initializeVerifier(JarFile.java:345)
> at java.util.jar.JarFile.getInputStream(JarFile.java:412)
> at
> sun.misc.URLClassPath$JarLoader$2.getInputStream(URLClassPath.java:775)
> at sun.misc.Resource.cachedInputStream(Resource.java:77)
> at sun.misc.Resource.getByteBuffer(Resource.java:160)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:436)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at java.lang.Class.forName0(Native Method)
> at java.lang.Class.forName(Class.java:270)
> at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:289)
> at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)
> at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
>
>
> Here is the pom file I am using to build the jar:
> <project xmlns="http://maven.apache.org/POM/4.0.0";
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
> xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/maven-v4_0_0.xsd";>
> <modelVersion>4.0.0</modelVersion>
> <groupId>com.spark</groupId>
> <artifactId>myjar</artifactId>
> <version>0.0.1-SNAPSHOT</version>
> <name>${project.artifactId}</name>
> <description>My wonderfull scala app</description>
> <inceptionYear>2010</inceptionYear>
> <licenses>
> <license>
> <name>My License</name>
> <url>http://....</url>
> <distribution>repo</distribution>
> </license>
> </licenses>
>
> <properties>
> <cdh.version>cdh5.1.0</cdh.version>
> <maven.compiler.source>1.6</maven.compiler.source>
> <maven.compiler.target>1.6</maven.compiler.target>
> <encoding>UTF-8</encoding>
> <scala.tools.version>2.10</scala.tools.version>
> <scala.version>2.10.4</scala.version>
> </properties>
>
> <repositories>
> <repository>
> <id>scala-tools.org</id>
> <name>Scala-tools Maven2 Repository</name>
> <url>https://oss.sonatype.org/content/repositories/snapshots/</url>
> </repository>
> <repository>
> <id>maven-hadoop</id>
> <name>Hadoop Releases</name>
>
> <url>https://repository.cloudera.com/content/repositories/releases/</url>
> </repository>
> <repository>
> <id>cloudera-repos</id>
> <name>Cloudera Repos</name>
> <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
> </repository>
> </repositories>
> <pluginRepositories>
> <pluginRepository>
> <id>scala-tools.org</id>
> <name>Scala-tools Maven2 Repository</name>
> <url>https://oss.sonatype.org/content/repositories/snapshots/</url>
> </pluginRepository>
> </pluginRepositories>
>
> <dependencies>
> <dependency>
> <groupId>org.scala-lang</groupId>
> <artifactId>scala-library</artifactId>
> <version>${scala.version}</version>
> </dependency>
> <dependency>
> <groupId>org.apache.spark</groupId>
> <artifactId>spark-core_2.10</artifactId>
> <version>1.0.0-${cdh.version}</version>
> </dependency>
> <dependency>
> <groupId>org.apache.spark</groupId>
> <artifactId>spark-tools_2.10</artifactId>
> <version>1.0.0-${cdh.version}</version>
> </dependency>
> <dependency>
> <groupId>org.apache.spark</groupId>
> <artifactId>spark-streaming-flume_2.10</artifactId>
> <version>1.0.0-${cdh.version}</version>
> </dependency>
> <dependency>
> <groupId>org.apache.spark</groupId>
> <artifactId>spark-streaming_2.10</artifactId>
> <version>1.0.0-${cdh.version}</version>
> </dependency>
> <dependency>
> <groupId>org.apache.flume</groupId>
> <artifactId>flume-ng-sdk</artifactId>
> <version>1.5.0-${cdh.version}</version>
>
> <exclusions>
> <exclusion>
> <groupId>io.netty</groupId>
> <artifactId>netty</artifactId>
> </exclusion>
> </exclusions>
> </dependency>
> <dependency>
> <groupId>org.apache.flume</groupId>
> <artifactId>flume-ng-core</artifactId>
> <version>1.5.0-${cdh.version}</version>
>
> <exclusions>
> <exclusion>
> <groupId>io.netty</groupId>
> <artifactId>netty</artifactId>
> </exclusion>
> </exclusions>
> </dependency>
> <dependency>
> <groupId>org.apache.hbase</groupId>
> <artifactId>hbase-client</artifactId>
> <version>0.98.1-${cdh.version}</version>
>
> <exclusions>
> <exclusion>
> <groupId>io.netty</groupId>
> <artifactId>netty</artifactId>
> </exclusion>
> </exclusions>
> </dependency>
> <dependency>
> <groupId>org.apache.hadoop</groupId>
> <artifactId>hadoop-client</artifactId>
> <version>2.3.0-${cdh.version}</version>
>
> </dependency>
>
>
> <dependency>
> <groupId>junit</groupId>
> <artifactId>junit</artifactId>
> <version>4.11</version>
> <scope>test</scope>
> </dependency>
> <dependency>
> <groupId>org.specs2</groupId>
> <artifactId>specs2_${scala.tools.version}</artifactId>
> <version>1.13</version>
> <scope>test</scope>
> </dependency>
> <dependency>
> <groupId>org.scalatest</groupId>
> <artifactId>scalatest_${scala.tools.version}</artifactId>
> <version>2.0.M6-SNAP8</version>
> <scope>test</scope>
> </dependency>
> </dependencies>
>
> <build>
> <sourceDirectory>src/main/scala</sourceDirectory>
> <testSourceDirectory>src/test/scala</testSourceDirectory>
> <plugins>
> <plugin>
>
> <groupId>net.alchim31.maven</groupId>
> <artifactId>scala-maven-plugin</artifactId>
> <version>3.1.3</version>
> <executions>
> <execution>
> <goals>
> <goal>compile</goal>
> <goal>testCompile</goal>
> </goals>
> <configuration>
> <args>
> <arg>-make:transitive</arg>
> <arg>-dependencyfile</arg>
> <arg>${project.build.directory}/.scala_dependencies</arg>
> </args>
> </configuration>
> </execution>
> </executions>
> </plugin>
> <plugin>
> <groupId>org.apache.maven.plugins</groupId>
> <artifactId>maven-surefire-plugin</artifactId>
> <version>2.13</version>
> <configuration>
> <useFile>false</useFile>
> <disableXmlReport>true</disableXmlReport>
>
>
> <includes>
> <include>**/*Test.*</include>
> <include>**/*Suite.*</include>
> </includes>
> </configuration>
> </plugin>
> <plugin>
> <groupId>org.apache.maven.plugins</groupId>
> <artifactId>maven-shade-plugin</artifactId>
> <version>2.2</version>
> <configuration>
>
> <transformers>
> <transformer
> implementation="org.apache..plugins.shade.resource.ApacheLicenseResourceTransformer">
> </transformer>
> <transformer
> implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
> <resource>reference.conf</resource>
> </transformer>
> </transformers>
> </configuration>
> <executions>
> <execution>
> <phase>package</phase>
> <goals>
> <goal>shade</goal>
> </goals>
> </execution>
> </executions>
> </plugin>
> </plugins>
> </build>
> </project>
>
> Was wondering what I am missing, since the jar built and a MANIFEST.MF file
> is created, not sure why it is looking for a signature file digest, am I
> missing some exclusion clause maybe?
>
> I saw a thread based on SBT and tried using the --deploy-mode client option;
> even though, my understanding is by default it would use that mode anyway.
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Invalid-signature-file-digest-for-Manifest-main-attributes-with-spark-job-built-using-maven-tp14299.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
