Hi,
case class Person(name: String, age: Int)
val lines = ssc.textFileStream("blah blah")
val sqc = new SQLContext(sc);
lines.foreachRDD(rdd=>{
rdd.map(_.split(",")).map(p => Persons(p(0),
p(1).trim.toInt)).registerAsTable("data")
val teenagers = sqc.sql("SELECT * FROM data")
teenagers.saveAsParquetFile("people.parquet")
})
You can also try insertInto API instead of registerAsTable..but havnt used
it myself..
also you need to dynamically change parquet file name for every dstream...
Mayur Rustagi
Ph: +1 (760) 203 3257
http://www.sigmoidanalytics.com
@mayur_rustagi <https://twitter.com/mayur_rustagi>
On Wed, Aug 20, 2014 at 1:01 AM, salemi <alireza.sal...@udo.edu> wrote:
> Thank you but how do you convert the stream to parquet file?
>
> Ali
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Re-spark-reading-hfds-files-every-5-minutes-tp12359p12401.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>
