I don't believe the Kinesis asl should be provided. I used mergeStrategy 
successfully to produce an "uber jar."

Fyi, I've been having trouble consuming data out of Kinesis with Spark with no 
success :( 
Would be curious to know if you got it working.

Vadim

> On Apr 13, 2015, at 9:36 PM, Mike Trienis <[email protected]> wrote:
> 
> Hi All,
> 
> I have having trouble building a fat jar file through sbt-assembly. 
> 
> [warn] Merging 'META-INF/NOTICE.txt' with strategy 'rename'
> [warn] Merging 'META-INF/NOTICE' with strategy 'rename'
> [warn] Merging 'META-INF/LICENSE.txt' with strategy 'rename'
> [warn] Merging 'META-INF/LICENSE' with strategy 'rename'
> [warn] Merging 'META-INF/MANIFEST.MF' with strategy 'discard'
> [warn] Merging 
> 'META-INF/maven/com.thoughtworks.paranamer/paranamer/pom.properties' with 
> strategy 'discard'
> [warn] Merging 'META-INF/maven/com.thoughtworks.paranamer/paranamer/pom.xml' 
> with strategy 'discard'
> [warn] Merging 'META-INF/maven/commons-dbcp/commons-dbcp/pom.properties' with 
> strategy 'discard'
> [warn] Merging 'META-INF/maven/commons-dbcp/commons-dbcp/pom.xml' with 
> strategy 'discard'
> [warn] Merging 'META-INF/maven/commons-pool/commons-pool/pom.properties' with 
> strategy 'discard'
> [warn] Merging 'META-INF/maven/commons-pool/commons-pool/pom.xml' with 
> strategy 'discard'
> [warn] Merging 'META-INF/maven/joda-time/joda-time/pom.properties' with 
> strategy 'discard'
> [warn] Merging 'META-INF/maven/joda-time/joda-time/pom.xml' with strategy 
> 'discard'
> [warn] Merging 'META-INF/maven/log4j/log4j/pom.properties' with strategy 
> 'discard'
> [warn] Merging 'META-INF/maven/log4j/log4j/pom.xml' with strategy 'discard'
> [warn] Merging 'META-INF/maven/org.joda/joda-convert/pom.properties' with 
> strategy 'discard'
> [warn] Merging 'META-INF/maven/org.joda/joda-convert/pom.xml' with strategy 
> 'discard'
> [warn] Merging 'META-INF/maven/org.slf4j/slf4j-api/pom.properties' with 
> strategy 'discard'
> [warn] Merging 'META-INF/maven/org.slf4j/slf4j-api/pom.xml' with strategy 
> 'discard'
> [warn] Merging 'META-INF/maven/org.slf4j/slf4j-log4j12/pom.properties' with 
> strategy 'discard'
> [warn] Merging 'META-INF/maven/org.slf4j/slf4j-log4j12/pom.xml' with strategy 
> 'discard'
> [warn] Merging 'META-INF/services/java.sql.Driver' with strategy 
> 'filterDistinctLines'
> [warn] Merging 'rootdoc.txt' with strategy 'concat'
> [warn] Strategy 'concat' was applied to a file
> [warn] Strategy 'discard' was applied to 17 files
> [warn] Strategy 'filterDistinctLines' was applied to a file
> [warn] Strategy 'rename' was applied to 4 files
> 
> When submitting the spark application through the command
> 
> sh /usr/lib/spark/bin/spark-submit -class com.xxx.ExampleClassName 
> target/scala-2.10/xxxx-snapshot.jar
> 
> I end up the the following error, 
> 
> Exception in thread "main" java.lang.NoClassDefFoundError: 
> org/joda/time/format/DateTimeFormat
>       at com.amazonaws.auth.AWS4Signer.<clinit>(AWS4Signer.java:44)
>       at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>       at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>       at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>       at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>       at java.lang.Class.newInstance(Class.java:379)
>       at com.amazonaws.auth.SignerFactory.createSigner(SignerFactory.java:119)
>       at 
> com.amazonaws.auth.SignerFactory.lookupAndCreateSigner(SignerFactory.java:105)
>       at com.amazonaws.auth.SignerFactory.getSigner(SignerFactory.java:78)
>       at 
> com.amazonaws.AmazonWebServiceClient.computeSignerByServiceRegion(AmazonWebServiceClient.java:307)
>       at 
> com.amazonaws.AmazonWebServiceClient.computeSignerByURI(AmazonWebServiceClient.java:280)
>       at 
> com.amazonaws.AmazonWebServiceClient.setEndpoint(AmazonWebServiceClient.java:160)
>       at 
> com.amazonaws.services.kinesis.AmazonKinesisClient.setEndpoint(AmazonKinesisClient.java:2102)
>       at 
> com.amazonaws.services.kinesis.AmazonKinesisClient.init(AmazonKinesisClient.java:216)
>       at 
> com.amazonaws.services.kinesis.AmazonKinesisClient.<init>(AmazonKinesisClient.java:202)
>       at 
> com.amazonaws.services.kinesis.AmazonKinesisClient.<init>(AmazonKinesisClient.java:175)
>       at 
> com.amazonaws.services.kinesis.AmazonKinesisClient.<init>(AmazonKinesisClient.java:155)
>       at com.quickstatsengine.aws.AwsProvider$.<init>(AwsProvider.scala:20)
>       at com.quickstatsengine.aws.AwsProvider$.<clinit>(AwsProvider.scala)
> 
> The snippet from my build.sbt file is: 
> 
>         "org.apache.spark" %% "spark-core" % "1.2.0" % "provided",
>         "org.apache.spark" %% "spark-streaming" % "1.2.0" % "provided",
>         "com.datastax.spark" %% "spark-cassandra-connector" % "1.2.0-alpha1" 
> % "provided",
>         "org.apache.spark" %% "spark-streaming-kinesis-asl" % "1.2.0" % 
> "provided",
> 
> And the error is originating from:
> 
> val kinesisClient = new AmazonKinesisClient(new 
> DefaultAWSCredentialsProviderChain())
> 
> Am I correct to set spark-streaming-kinesis-asl as a provided dependency? 
> Also, is there a merge strategy I need apply? 
> 
> Any help would be appreciated, Mike. 
> 
> 

Reply via email to