Ah excellent! I will be sure to check if we need to update our documentation based on your feedback :)
Cheers, Helena On Oct 28, 2014, at 3:03 PM, Harold Nguyen <har...@nexgate.com> wrote: > Hi Helena, > > It's great to e-meet you! I've actually been following along your blogs and > talks trying to get this to work. > > I just solved it, and you were absolutely correct. I've been using > 1.1.0-alpha3 as my dependency, but my assembly is the 1.2.0-SNAPSHOT. > > Thanks for looking through all my other threads to piece together the problem. > > So the solution is that I just included the > spark-cassandra-connector-assembly-1.2.0-SNAPSHOT straight into a "lib" > folder, removed the line from 1.1.0-alpha3 line from libraryDepenencies, and > re-packaged it. > > It all works great now, and I couldn't be happier. I had to piece together 6 > different forums and sites to get that working (being absolutely new to Spark > and Scala and sbt). I'll write a blog post on how to get this working later, > in case it can help someone. > > I really appreciate the help! > > Harold > > On Tue, Oct 28, 2014 at 11:55 AM, Helena Edelson > <helena.edel...@datastax.com> wrote: > Hi Harold, > It seems like, based on your previous post, you are using one version of the > connector as a dependency yet building the assembly jar from master? You were > using 1.1.0-alpha3 (you can upgrade to alpha4, beta coming this week) yet > your assembly is spark-cassandra-connector-assembly-1.2.0-SNAPSHOT. > > libraryDependencies ++= Seq( > "org.apache.spark" % "spark-streaming_2.10" % "1.1.0", > "com.datastax.spark" %% "spark-cassandra-connector" % "1.1.0-alpha3" > withSources() withJavadoc(), > "org.apache.spark" %% "spark-sql" % "1.1.0" > ) > > - Helena > > On Oct 28, 2014, at 2:08 PM, Harold Nguyen <har...@nexgate.com> wrote: > > > Hi all, > > > > The following works fine when submitting dependency jars through > > Spark-Shell: > > > > ./bin/spark-shell --master spark://ip-172-31-38-112:7077 --jars > > /home/ubuntu/spark-cassandra-connector/spark-cassandra-connector/spark-cassandra-connector/target/scala-2.10/spark-cassandra-connector-assembly-1.2.0-SNAPSHOT.jar > > > > But not through spark-submit: > > > > ./bin/spark-submit --class > > "org.apache.spark.examples.streaming.CassandraSave" --master > > spark://ip-172-31-38-112:7077 > > streaming-test/target/scala-2.10/simple-streaming_2.10-1.0.jar --jars > > local:///home/ubuntu/spark-cassandra-connector/spark-cassandra-connector/target/scala-2.10/spark-cassandra-connector-assembly-1.2.0-SNAPSHOT.jar > > > > Am I issuing the spark-submit command incorrectly ? Each of the workers has > > that built jar in their respective directories > > (spark-cassandra-connector-assembly-1.2.0-SNAPSHOT.jar) > > > > Thanks, > > > > Harold > >
