+1 On Thu, Jan 12, 2023 at 7:46 PM Hyukjin Kwon <gurwls...@gmail.com> wrote:
> +1 > > On Fri, 13 Jan 2023 at 08:51, Jungtaek Lim <kabhwan.opensou...@gmail.com> > wrote: > >> bump for more visibility. >> >> On Wed, Jan 11, 2023 at 12:20 PM Jungtaek Lim < >> kabhwan.opensou...@gmail.com> wrote: >> >>> Hi dev, >>> >>> I'd like to propose the deprecation of DStream in Spark 3.4, in favor of >>> promoting Structured Streaming. >>> (Sorry for the late proposal, if we don't make the change in 3.4, we >>> will have to wait for another 6 months.) >>> >>> We have been focusing on Structured Streaming for years (across multiple >>> major and minor versions), and during the time we haven't made any >>> improvements for DStream. Furthermore, recently we updated the DStream doc >>> to explicitly say DStream is a legacy project. >>> >>> https://spark.apache.org/docs/latest/streaming-programming-guide.html#note >>> >>> The baseline of deprecation is that we don't see a particular use case >>> which only DStream solves. This is a different story with GraphX and MLLIB, >>> as we don't have replacements for that. >>> >>> The proposal does not mean we will remove the API soon, as the Spark >>> project has been making deprecation against public API. I don't intend to >>> propose the target version for removal. The goal is to guide users to >>> refrain from constructing a new workload with DStream. We might want to go >>> with this in future, but it would require a new discussion thread at that >>> time. >>> >>> What do you think? >>> >>> Thanks, >>> Jungtaek Lim (HeartSaVioR) >>> >>
