So, one portion of our Spark streaming application requires some
state. Our application takes a bunch of application events (i.e.
user_session_started, user_session_ended, etc..), and calculates out
metrics from these, and writes them to a serving layer (see: Lambda
Architecture). Two related events can be ingested into the streaming
context moments apart, or time inderminate. Given this, and the fact
that our normal windows pump data out every 500-1 ms, with a step
of 500ms, you might end up with two related pieces of data across two
windows. In order to work around this, we go ahead and do
updateStateByKey to persist state, as opposed to persisting key
intermediate state in some external system, as building a system to
handle the complexities of (concurrent, idempotent) updates, as well
as ensure scalability is non-trivial.
The questions I have around this, is even in a highly-partitionable
dataset, what's the upper scalability limits with stateful dstreams?
If I have a dataset, starting at around 10-million keys, growing at
that rate monthly, what are the complexities within? Most of the data
is cold. I realize that I can remove data from the stateful dstream,
by sending (key, null) to it, but there is not necessarily an easy way
of knowing when the last update is coming in (unless there is some way
in spark of saying, Wait N windows, and send this tuple or Wait
until all keys in the upstream Dstreams smaller than M are processed
before sending such a tuple. Additionally, given that my data is
partitionable by datetime, does it make sense to have a custom
datetime partitioner, and just persist the dstream to disk, to ensure
that its RDDs are only pulled off of disk (into memory) occasionally?
What's the cost of having a bunch of relatively large, stateful RDDs
around on disk? Does Spark have to load / deserialize the entire RDD
to update one key?
