After comparing with previous code, I got it work by making the return a Some
instead of Tuple2. Perhaps some day I will understand this.
spr wrote
--code
val updateDnsCount = (values: Seq[(Int, Time)], state: Option[(Int,
Time)]) = {
val currentCount = if
My understanding is that the reason you have an Option is so you could filter
out tuples when None is returned. This way your state data won't grow forever.
-Original Message-
From: spr [mailto:s...@yarcdata.com]
Sent: November-12-14 2:25 PM
To: u...@spark.incubator.apache.org
Subject:
Adrian, do you know if this is documented somewhere? I was also under the
impression that setting a key's value to None would cause the key to be
discarded (without any explicit filtering on the user's part) but can not
find any official documentation to that effect
On Wed, Nov 12, 2014 at 2:43
You are correct; the filtering I’m talking about is done implicitly. You don’t
have to do it yourself. Spark will do it for you and remove those entries from
the state collection.
From: Yana Kadiyska [mailto:yana.kadiy...@gmail.com]
Sent: November-12-14 3:50 PM
To: Adrian Mocanu
Cc: spr;
I'm missing something simpler (I think). That is, why do I need a Some instead
of Tuple2? Because a Some might or might not be there, but a Tuple2 must be
there? Or something like that?
From: Adrian Mocanu
amoc...@verticalscope.commailto:amoc...@verticalscope.com
You are correct; the
I am creating a workflow; I have an existing call to updateStateByKey that
works fine, but when I created a second use where the key is a Tuple2, it's
now failing with the dreaded overloaded method value updateStateByKey with
alternatives ... cannot be applied to ... Comparing the two uses I'm
