Thanks Mike. I was having trouble on EC2.
> On Apr 13, 2015, at 10:25 PM, Mike Trienis <mike.trie...@orcsol.com> wrote: > > Thanks Vadim, I can certainly consume data from a Kinesis stream when running > locally. I'm currently in the processes of extending my work to a proper > cluster (i.e. using a spark-submit job via uber jar). Feel free to add me to > gmail chat and maybe we can help each other. > >> On Mon, Apr 13, 2015 at 6:46 PM, Vadim Bichutskiy >> <vadim.bichuts...@gmail.com> wrote: >> I don't believe the Kinesis asl should be provided. I used mergeStrategy >> successfully to produce an "uber jar." >> >> Fyi, I've been having trouble consuming data out of Kinesis with Spark with >> no success :( >> Would be curious to know if you got it working. >> >> Vadim >> >>> On Apr 13, 2015, at 9:36 PM, Mike Trienis <mike.trie...@orcsol.com> wrote: >>> >>> Hi All, >>> >>> I have having trouble building a fat jar file through sbt-assembly. >>> >>> [warn] Merging 'META-INF/NOTICE.txt' with strategy 'rename' >>> [warn] Merging 'META-INF/NOTICE' with strategy 'rename' >>> [warn] Merging 'META-INF/LICENSE.txt' with strategy 'rename' >>> [warn] Merging 'META-INF/LICENSE' with strategy 'rename' >>> [warn] Merging 'META-INF/MANIFEST.MF' with strategy 'discard' >>> [warn] Merging >>> 'META-INF/maven/com.thoughtworks.paranamer/paranamer/pom.properties' with >>> strategy 'discard' >>> [warn] Merging >>> 'META-INF/maven/com.thoughtworks.paranamer/paranamer/pom.xml' with strategy >>> 'discard' >>> [warn] Merging 'META-INF/maven/commons-dbcp/commons-dbcp/pom.properties' >>> with strategy 'discard' >>> [warn] Merging 'META-INF/maven/commons-dbcp/commons-dbcp/pom.xml' with >>> strategy 'discard' >>> [warn] Merging 'META-INF/maven/commons-pool/commons-pool/pom.properties' >>> with strategy 'discard' >>> [warn] Merging 'META-INF/maven/commons-pool/commons-pool/pom.xml' with >>> strategy 'discard' >>> [warn] Merging 'META-INF/maven/joda-time/joda-time/pom.properties' with >>> strategy 'discard' >>> [warn] Merging 'META-INF/maven/joda-time/joda-time/pom.xml' with strategy >>> 'discard' >>> [warn] Merging 'META-INF/maven/log4j/log4j/pom.properties' with strategy >>> 'discard' >>> [warn] Merging 'META-INF/maven/log4j/log4j/pom.xml' with strategy 'discard' >>> [warn] Merging 'META-INF/maven/org.joda/joda-convert/pom.properties' with >>> strategy 'discard' >>> [warn] Merging 'META-INF/maven/org.joda/joda-convert/pom.xml' with strategy >>> 'discard' >>> [warn] Merging 'META-INF/maven/org.slf4j/slf4j-api/pom.properties' with >>> strategy 'discard' >>> [warn] Merging 'META-INF/maven/org.slf4j/slf4j-api/pom.xml' with strategy >>> 'discard' >>> [warn] Merging 'META-INF/maven/org.slf4j/slf4j-log4j12/pom.properties' with >>> strategy 'discard' >>> [warn] Merging 'META-INF/maven/org.slf4j/slf4j-log4j12/pom.xml' with >>> strategy 'discard' >>> [warn] Merging 'META-INF/services/java.sql.Driver' with strategy >>> 'filterDistinctLines' >>> [warn] Merging 'rootdoc.txt' with strategy 'concat' >>> [warn] Strategy 'concat' was applied to a file >>> [warn] Strategy 'discard' was applied to 17 files >>> [warn] Strategy 'filterDistinctLines' was applied to a file >>> [warn] Strategy 'rename' was applied to 4 files >>> >>> When submitting the spark application through the command >>> >>> sh /usr/lib/spark/bin/spark-submit -class com.xxx.ExampleClassName >>> target/scala-2.10/xxxx-snapshot.jar >>> >>> I end up the the following error, >>> >>> Exception in thread "main" java.lang.NoClassDefFoundError: >>> org/joda/time/format/DateTimeFormat >>> at com.amazonaws.auth.AWS4Signer.<clinit>(AWS4Signer.java:44) >>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) >>> at >>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) >>> at >>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) >>> at java.lang.reflect.Constructor.newInstance(Constructor.java:526) >>> at java.lang.Class.newInstance(Class.java:379) >>> at com.amazonaws.auth.SignerFactory.createSigner(SignerFactory.java:119) >>> at >>> com.amazonaws.auth.SignerFactory.lookupAndCreateSigner(SignerFactory.java:105) >>> at com.amazonaws.auth.SignerFactory.getSigner(SignerFactory.java:78) >>> at >>> com.amazonaws.AmazonWebServiceClient.computeSignerByServiceRegion(AmazonWebServiceClient.java:307) >>> at >>> com.amazonaws.AmazonWebServiceClient.computeSignerByURI(AmazonWebServiceClient.java:280) >>> at >>> com.amazonaws.AmazonWebServiceClient.setEndpoint(AmazonWebServiceClient.java:160) >>> at >>> com.amazonaws.services.kinesis.AmazonKinesisClient.setEndpoint(AmazonKinesisClient.java:2102) >>> at >>> com.amazonaws.services.kinesis.AmazonKinesisClient.init(AmazonKinesisClient.java:216) >>> at >>> com.amazonaws.services.kinesis.AmazonKinesisClient.<init>(AmazonKinesisClient.java:202) >>> at >>> com.amazonaws.services.kinesis.AmazonKinesisClient.<init>(AmazonKinesisClient.java:175) >>> at >>> com.amazonaws.services.kinesis.AmazonKinesisClient.<init>(AmazonKinesisClient.java:155) >>> at com.quickstatsengine.aws.AwsProvider$.<init>(AwsProvider.scala:20) >>> at com.quickstatsengine.aws.AwsProvider$.<clinit>(AwsProvider.scala) >>> >>> The snippet from my build.sbt file is: >>> >>> "org.apache.spark" %% "spark-core" % "1.2.0" % "provided", >>> "org.apache.spark" %% "spark-streaming" % "1.2.0" % "provided", >>> "com.datastax.spark" %% "spark-cassandra-connector" % >>> "1.2.0-alpha1" % "provided", >>> "org.apache.spark" %% "spark-streaming-kinesis-asl" % "1.2.0" % >>> "provided", >>> >>> And the error is originating from: >>> >>> val kinesisClient = new AmazonKinesisClient(new >>> DefaultAWSCredentialsProviderChain()) >>> >>> Am I correct to set spark-streaming-kinesis-asl as a provided dependency? >>> Also, is there a merge strategy I need apply? >>> >>> Any help would be appreciated, Mike. >
