Yes, check out
mapWithState:https://databricks.com/blog/2016/02/01/faster-stateful-stream-processing-in-apache-spark-streaming.html
_____________________________
From: Nikhil Goyal <[email protected]>
Sent: Monday, May 23, 2016 23:28
Subject: Timed aggregation in Spark
To: <[email protected]>
Hi all,
I want to aggregate my data for 5-10 min and then flush the aggregated data to
some database like vertica. updateStateByKey is not exactly helpful in this
scenario as I can't flush all the records at once, neither can I clear the
state. I wanted to know if anyone else has faced a similar issue and how did
they handle it.
ThanksNikhil
