Hi dev,
I'd like to propose the deprecation of DStream in Spark 3.4, in favor of
promoting Structured Streaming.
(Sorry for the late proposal, if we don't make the change in 3.4, we will
have to wait for another 6 months.)
We have been focusing on Structured Streaming for years (across multiple
major and minor versions), and during the time we haven't made any
improvements for DStream. Furthermore, recently we updated the DStream doc
to explicitly say DStream is a legacy project.
https://spark.apache.org/docs/latest/streaming-programming-guide.html#note
The baseline of deprecation is that we don't see a particular use case
which only DStream solves. This is a different story with GraphX and MLLIB,
as we don't have replacements for that.
The proposal does not mean we will remove the API soon, as the Spark
project has been making deprecation against public API. I don't intend to
propose the target version for removal. The goal is to guide users to
refrain from constructing a new workload with DStream. We might want to go
with this in future, but it would require a new discussion thread at that
time.
What do you think?
Thanks,
Jungtaek Lim (HeartSaVioR)
