+1 for the proposal (guiding only without any code change). Thanks, Dongjoon.
On Thu, Jan 12, 2023 at 9:33 PM Shixiong Zhu <zsxw...@gmail.com> wrote: > +1 > > > On Thu, Jan 12, 2023 at 5:08 PM Tathagata Das <tathagata.das1...@gmail.com> > wrote: > >> +1 >> >> On Thu, Jan 12, 2023 at 7:46 PM Hyukjin Kwon <gurwls...@gmail.com> wrote: >> >>> +1 >>> >>> On Fri, 13 Jan 2023 at 08:51, Jungtaek Lim <kabhwan.opensou...@gmail.com> >>> wrote: >>> >>>> bump for more visibility. >>>> >>>> On Wed, Jan 11, 2023 at 12:20 PM Jungtaek Lim < >>>> kabhwan.opensou...@gmail.com> wrote: >>>> >>>>> Hi dev, >>>>> >>>>> I'd like to propose the deprecation of DStream in Spark 3.4, in favor >>>>> of promoting Structured Streaming. >>>>> (Sorry for the late proposal, if we don't make the change in 3.4, we >>>>> will have to wait for another 6 months.) >>>>> >>>>> We have been focusing on Structured Streaming for years (across >>>>> multiple major and minor versions), and during the time we haven't made >>>>> any >>>>> improvements for DStream. Furthermore, recently we updated the DStream doc >>>>> to explicitly say DStream is a legacy project. >>>>> >>>>> https://spark.apache.org/docs/latest/streaming-programming-guide.html#note >>>>> >>>>> The baseline of deprecation is that we don't see a particular use case >>>>> which only DStream solves. This is a different story with GraphX and >>>>> MLLIB, >>>>> as we don't have replacements for that. >>>>> >>>>> The proposal does not mean we will remove the API soon, as the Spark >>>>> project has been making deprecation against public API. I don't intend to >>>>> propose the target version for removal. The goal is to guide users to >>>>> refrain from constructing a new workload with DStream. We might want to go >>>>> with this in future, but it would require a new discussion thread at that >>>>> time. >>>>> >>>>> What do you think? >>>>> >>>>> Thanks, >>>>> Jungtaek Lim (HeartSaVioR) >>>>> >>>>
