, September 29, 2015 at 5:09 PM
To: Daniel Haviv, user
Subject: RE: Converting a DStream to schemaRDD
Something like:
dstream.foreachRDD { rdd =>
val df = sqlContext.read.json(rdd)
df.select(…)
}
https://spark.apache.org/docs/latest/streaming-programming-guide.html#output-operati
Something like:
dstream.foreachRDD { rdd =>
val df = sqlContext.read.json(rdd)
df.select(…)
}
https://spark.apache.org/docs/latest/streaming-programming-guide.html#output-operations-on-dstreams
Might be the place to start, it’ll convert each batch of dstream into an RDD
then let you work
