Saving RDD into cassandra keyspace.
Hi,
I am beginner to spark , I want save the word and its count to cassandra
keyspace, I wrote the following code
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
import com.datastax.spark.connector._
object SparkWordCount {
def main(args: Array[String]) {
val sc = new SparkContext(new SparkConf().setAppName(Spark Count))
val tokenized = sc.textFile(args(0)).flatMap(_.split( ))
val wordCounts = tokenized.map((_, 1)).reduceByKey(_ + _)
wordCounts.saveToCassandra(sparkdata, words, SomeColumns(word,
count));
}
and did spark-submit. The code doesn't work ( may be some very basic error
because I am new to it).I know there is datastax cassandra connector but how to
make connection?
What all things I am missing in my code?
Thanks
DISCLAIMER: This message is proprietary to Aricent and is intended solely for
the use of the individual to whom it is addressed. It may contain privileged or
confidential information and should not be circulated or used for any purpose
other than for what it is intended. If you have received this message in error,
please notify the originator immediately. If you are not the intended
recipient, you are notified that you are strictly prohibited from using,
copying, altering, or disclosing the contents of this message. Aricent accepts
no responsibility for loss or damage arising from the use of the information
transmitted by this email including damage from virus.
Re: Saving RDD into cassandra keyspace.
I would strongly encourage you to read the docs at, they are very useful in
getting up and running:
https://github.com/datastax/spark-cassandra-connector/blob/master/doc/0_quick_start.md
For your use case shown above, you will need to ensure that you include the
appropriate version of the spark-cassandra-connectore assembly jar when you
submit the job. The version you use should correspond to the version of
Spark you are running. In addition, you will want to ensure that you set
the spark.cassandra.connection.host as shown below, prior to creating the
SparkContext.
val conf = new SparkConf(true)
.set(spark.cassandra.connection.host, 127.0.0.1)
HTH
-Todd
On Fri, Jul 10, 2015 at 5:24 AM, Prateek . prat...@aricent.com wrote:
Hi,
I am beginner to spark , I want save the word and its count to cassandra
keyspace, I wrote the following code
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
import com.datastax.spark.connector._
object SparkWordCount {
def main(args: Array[String]) {
val sc = new SparkContext(new SparkConf().setAppName(Spark Count))
val tokenized = sc.textFile(args(0)).flatMap(_.split( ))
val wordCounts = tokenized.map((_, 1)).reduceByKey(_ + _)
wordCounts.saveToCassandra(sparkdata, words, SomeColumns(word,
count));
}
and did spark-submit. The code doesn’t work ( may be some very basic error
because I am new to it).I know there is datastax cassandra connector but
how to make connection?
What all things I am missing in my code?
Thanks
DISCLAIMER: This message is proprietary to Aricent and is intended
solely for the use of the individual to whom it is addressed. It may
contain privileged or confidential information and should not be circulated
or used for any purpose other than for what it is intended. If you have
received this message in error, please notify the originator immediately.
If you are not the intended recipient, you are notified that you are
strictly prohibited from using, copying, altering, or disclosing the
contents of this message. Aricent accepts no responsibility for loss or
damage arising from the use of the information transmitted by this email
including damage from virus.
RE: Saving RDD into cassandra keyspace.
Hi,
Thanks Todd..the link is really helpful to get started. ☺
-Prateek
From: Todd Nist [mailto:tsind...@gmail.com]
Sent: Friday, July 10, 2015 4:43 PM
To: Prateek .
Cc: user@spark.apache.org
Subject: Re: Saving RDD into cassandra keyspace.
I would strongly encourage you to read the docs at, they are very useful in
getting up and running:
https://github.com/datastax/spark-cassandra-connector/blob/master/doc/0_quick_start.md
For your use case shown above, you will need to ensure that you include the
appropriate version of the spark-cassandra-connectore assembly jar when you
submit the job. The version you use should correspond to the version of Spark
you are running. In addition, you will want to ensure that you set the
spark.cassandra.connection.host as shown below, prior to creating the
SparkContext.
val conf = new SparkConf(true)
.set(spark.cassandra.connection.host, 127.0.0.1)
HTH
-Todd
On Fri, Jul 10, 2015 at 5:24 AM, Prateek .
prat...@aricent.commailto:prat...@aricent.com wrote:
Hi,
I am beginner to spark , I want save the word and its count to cassandra
keyspace, I wrote the following code
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.SparkConf
import com.datastax.spark.connector._
object SparkWordCount {
def main(args: Array[String]) {
val sc = new SparkContext(new SparkConf().setAppName(Spark Count))
val tokenized = sc.textFile(args(0)).flatMap(_.split( ))
val wordCounts = tokenized.map((_, 1)).reduceByKey(_ + _)
wordCounts.saveToCassandra(sparkdata, words, SomeColumns(word,
count));
}
and did spark-submit. The code doesn’t work ( may be some very basic error
because I am new to it).I know there is datastax cassandra connector but how to
make connection?
What all things I am missing in my code?
Thanks
DISCLAIMER: This message is proprietary to Aricent and is intended solely for
the use of the individual to whom it is addressed. It may contain privileged or
confidential information and should not be circulated or used for any purpose
other than for what it is intended. If you have received this message in error,
please notify the originator immediately. If you are not the intended
recipient, you are notified that you are strictly prohibited from using,
copying, altering, or disclosing the contents of this message. Aricent accepts
no responsibility for loss or damage arising from the use of the information
transmitted by this email including damage from virus.
DISCLAIMER: This message is proprietary to Aricent and is intended solely for
the use of the individual to whom it is addressed. It may contain privileged or
confidential information and should not be circulated or used for any purpose
other than for what it is intended. If you have received this message in error,
please notify the originator immediately. If you are not the intended
recipient, you are notified that you are strictly prohibited from using,
copying, altering, or disclosing the contents of this message. Aricent accepts
no responsibility for loss or damage arising from the use of the information
transmitted by this email including damage from virus.
