May be you can make use of the Window operations
https://spark.apache.org/docs/1.2.0/streaming-programming-guide.html#window-operations,
Also another approach would be to keep your incoming data in
Hbase/Redis/Cassandra kind of database and then whenever you need to
average it, you just query the database and average it.
Thanks
Best Regards
On Thu, May 28, 2015 at 1:22 AM, David Webber david.web...@gmail.com
wrote:
Hi All,
I'm new to Spark and I'd like some help understanding if a particular use
case would be a good fit for Spark Streaming.
I have an imaginary stream of sensor data consisting of integers 1-10.
Every time the sensor reads 10 I'd like to average all the numbers that
were received since the last 10
example input: 10 5 8 4 6 2 1 2 8 8 8 1 6 9 1 3 10 1 3 10 ...
desired output: 4.8, 2.0
I'm confused about what happens if sensor readings fall into different
RDDs.
RDD1: 10 5 8 4 6 2 1 2 8 8 8
RDD2: 1 6 9 1 3 10 1 3 10
output: ???, 2.0
My imaginary sensor doesn't read at fixed time intervals, so breaking the
stream into RDDs by time interval won't ensure the data is packaged
properly. Additionally, multiple sensors are writing to the same stream
(though I think flatMap can parse the origin stream into streams for
individual sensors, correct?).
My best guess for processing goes like
1) flatMap() to break out individual sensor streams
2) Custom parser to accumulate input data until 10 is found, then create
a
new output RDD for each sensor and data grouping
3) average the values from step 2
I would greatly appreciate pointers to some specific documentation or
examples if you have seen something like this before.
Thanks,
David
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/RDD-boundaries-and-triggering-processing-using-tags-in-the-data-tp23060.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
