The 2 operations can't be used inside one another.
If you need something like an all time average then you need to keep a tuple
(sum, count) to which you add all the new values that come in every batch. The
average is then just a map on the state DStream.
Makes sense? have I guessed your use case?
Sent from my iPhone
> On 24 Sep 2015, at 19:47, swetha wrote:
>
> Hi,
>
> How to use reduceByKey inside updateStateByKey? Suppose I have a bunch of
> keys for which I need to do sum and average inside the updateStateByKey by
> joining with old state. How do I accomplish that?
>
>
> Thanks,
> Swetha
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/reduceByKey-inside-updateStateByKey-in-Spark-Streaming-tp24808.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> -
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
