Thanks Ashish and Jean.
I am using scala API, so I used the case class
case class Coordinate(id: String, ax: Double, ay: Double, az: Double, oa:
Double, ob: Double, og:Double)
def mapToCoordinate(jsonMap: Map[String,Any]): Coordinate = {
//map the coordinate
}
val lines = KafkaUtils.createStream(ssc, zkQuorum, group, topicMap).map(_._2)
val jsonf =
lines.map(JSON.parseFull(_)).map(_.get.asInstanceOf[scala.collection.immutable.Map[String,
Any]])
val coordinate = jsonf.map(m => mapToCoordinate(m))
coordinate.saveToCassandra("android_data", "coordinate", SomeColumns("id",
"ax","ay", "az", "oa", "ob", "og"))
// I am doing something wrong here Casssandra schema is getting "Some" type of
value
Some(9e5ccb3d5e4e421392eb98978a6b368e) | Some(0.8437097133160075) |
Some(0.5656238331780492) | Some(0.5235250642853548)
I am not able to figure out how to map the Dstream[Coordinate] to columns in
schema .
Thank You
Prateek
-----Original Message-----
From: Jean-Baptiste Onofré [mailto:j...@nanthrax.net]
Sent: Monday, October 05, 2015 7:58 PM
To: user@spark.apache.org
Subject: Re: DStream Transformation to save JSON in Cassandra 2.1
Hi Prateek,
I see two ways:
- using Cassandra CQL to adapt the RDD in the DStream to Cassandra
- using a Cassandra converter
You have a couple of code snippet in the examples. Let me know if you need a
code sample.
Regards
JB
On 10/05/2015 04:14 PM, Prateek . wrote:
> Hi,
>
> I am beginner in Spark , this is sample data I get from Kafka stream:
>
> {"id":
> "9f5ccb3d5f4f421392fb98978a6b368f","coordinate":{"ax":"1.20","ay":"3.8
> 0","az":"9.90","oa":"8.03","ob":"8.8","og":"9.97"}}
>
> val lines = KafkaUtils.createStream(ssc, zkQuorum, group,
> topicMap).map(_._2)
> val jsonf =
> lines.map(JSON.parseFull(_)).map(_.get.asInstanceOf[scala.collection.i
> mmutable.Map[String, Any]])
>
> I am getting a, DSTream[Map[String,Any]]. I need to store each
> coordinate values in the below Cassandra schema
>
> CREATE TABLE iotdata.coordinate (
> id text PRIMARY KEY, ax double, ay double, az double, oa double,
> ob double, oz double
> )
>
> For this what transformations I need to apply before I execute
> saveToCassandra().
>
> Thank You,
> Prateek
>
>
> "DISCLAIMER: This message is proprietary to Aricent and is intended solely
> for the use of the individual to whom it is addressed. It may contain
> privileged or confidential information and should not be circulated or used
> for any purpose other than for what it is intended. If you have received this
> message in error, please notify the originator immediately. If you are not
> the intended recipient, you are notified that you are strictly prohibited
> from using, copying, altering, or disclosing the contents of this message.
> Aricent accepts no responsibility for loss or damage arising from the use of
> the information transmitted by this email including damage from virus."
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For
> additional commands, e-mail: user-h...@spark.apache.org
>
--
Jean-Baptiste Onofré
jbono...@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional
commands, e-mail: user-h...@spark.apache.org
"DISCLAIMER: This message is proprietary to Aricent and is intended solely for
the use of the individual to whom it is addressed. It may contain privileged or
confidential information and should not be circulated or used for any purpose
other than for what it is intended. If you have received this message in error,
please notify the originator immediately. If you are not the intended
recipient, you are notified that you are strictly prohibited from using,
copying, altering, or disclosing the contents of this message. Aricent accepts
no responsibility for loss or damage arising from the use of the information
transmitted by this email including damage from virus."
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
