I hope it is the right forum.
I am looking for some information of what to expect from
StructuredStreaming in its next releases to help me choose when / where to
start using it more seriously (or where to invest in workarounds and where
to wait). I couldn't find a good place where such planning discussed for
2.1 (like, for example ML and SPARK-15581).
I'm aware of the 2.0 documented limits (
like no support for multiple aggregations levels, joins are strictly to a
static dataset (no SCD or stream-stream) etc, limited sources / sinks (like
no sink for interactive queries) etc etc
I'm also aware of some changes that have landed in master, like the new
Kafka 0.10 source (and its on-going improvements) in SPARK-15406, the
metrics in SPARK-17731, and some improvements for the file source.
If I remember correctly, the discussion on Spark release cadence concluded
with a preference to a four-month cycles, with likely code freeze pretty
soon (end of October). So I believe the scope for 2.1 should likely quite
clear to some, and that 2.2 planning should likely be starting about now.
Any visibility / sharing will be highly appreciated!
thanks in advance,
Co-Founder & CTO | Equalum
Mobile: +972-54-7801286 | Email: ofir.ma...@equalum.io