Re: [DISCUSS] Deprecate DStream in 3.4
+1 On Thu, Jan 12, 2023 at 10:39 PM Jungtaek Lim wrote: > > Yes, exactly. I'm sorry to bring confusion - should have clarified action > items on the proposal. > > On Fri, Jan 13, 2023 at 3:31 PM Dongjoon Hyun wrote: >> >> Then, could you elaborate `the proposed code change` specifically? >> Maybe, usual deprecation warning logs and annotation on the API? >> >> >> On Thu, Jan 12, 2023 at 10:05 PM Jungtaek Lim >> wrote: >>> >>> Maybe I need to clarify - my proposal is "explicitly" deprecating it, which >>> incurs code change for sure. Guidance on the Spark website is done already >>> as I mentioned - we updated the DStream doc page to mention that DStream is >>> a "legacy" project and users should move to SS. I don't feel this is >>> sufficient to refrain users from using it, hence initiating this proposal. >>> >>> Sorry to make confusion. I just wanted to make sure the goal of the >>> proposal is not "removing" the API. The discussion on the removal of API >>> doesn't tend to go well, so I wanted to make sure I don't mean that. >>> >>> On Fri, Jan 13, 2023 at 2:46 PM Dongjoon Hyun >>> wrote: +1 for the proposal (guiding only without any code change). Thanks, Dongjoon. On Thu, Jan 12, 2023 at 9:33 PM Shixiong Zhu wrote: > > +1 > > > On Thu, Jan 12, 2023 at 5:08 PM Tathagata Das > wrote: >> >> +1 >> >> On Thu, Jan 12, 2023 at 7:46 PM Hyukjin Kwon wrote: >>> >>> +1 >>> >>> On Fri, 13 Jan 2023 at 08:51, Jungtaek Lim >>> wrote: bump for more visibility. On Wed, Jan 11, 2023 at 12:20 PM Jungtaek Lim wrote: > > Hi dev, > > I'd like to propose the deprecation of DStream in Spark 3.4, in favor > of promoting Structured Streaming. > (Sorry for the late proposal, if we don't make the change in 3.4, we > will have to wait for another 6 months.) > > We have been focusing on Structured Streaming for years (across > multiple major and minor versions), and during the time we haven't > made any improvements for DStream. Furthermore, recently we updated > the DStream doc to explicitly say DStream is a legacy project. > https://spark.apache.org/docs/latest/streaming-programming-guide.html#note > > The baseline of deprecation is that we don't see a particular use > case which only DStream solves. This is a different story with GraphX > and MLLIB, as we don't have replacements for that. > > The proposal does not mean we will remove the API soon, as the Spark > project has been making deprecation against public API. I don't > intend to propose the target version for removal. The goal is to > guide users to refrain from constructing a new workload with DStream. > We might want to go with this in future, but it would require a new > discussion thread at that time. > > What do you think? > > Thanks, > Jungtaek Lim (HeartSaVioR) - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
Re: [DISCUSS] Deprecate DStream in 3.4
Yes, exactly. I'm sorry to bring confusion - should have clarified action items on the proposal. On Fri, Jan 13, 2023 at 3:31 PM Dongjoon Hyun wrote: > Then, could you elaborate `the proposed code change` specifically? > Maybe, usual deprecation warning logs and annotation on the API? > > > On Thu, Jan 12, 2023 at 10:05 PM Jungtaek Lim < > kabhwan.opensou...@gmail.com> wrote: > >> Maybe I need to clarify - my proposal is "explicitly" deprecating it, >> which incurs code change for sure. Guidance on the Spark website is done >> already as I mentioned - we updated the DStream doc page to mention that >> DStream is a "legacy" project and users should move to SS. I don't feel >> this is sufficient to refrain users from using it, hence initiating >> this proposal. >> >> Sorry to make confusion. I just wanted to make sure the goal of the >> proposal is not "removing" the API. The discussion on the removal of API >> doesn't tend to go well, so I wanted to make sure I don't mean that. >> >> On Fri, Jan 13, 2023 at 2:46 PM Dongjoon Hyun >> wrote: >> >>> +1 for the proposal (guiding only without any code change). >>> >>> Thanks, >>> Dongjoon. >>> >>> On Thu, Jan 12, 2023 at 9:33 PM Shixiong Zhu wrote: >>> +1 On Thu, Jan 12, 2023 at 5:08 PM Tathagata Das < tathagata.das1...@gmail.com> wrote: > +1 > > On Thu, Jan 12, 2023 at 7:46 PM Hyukjin Kwon > wrote: > >> +1 >> >> On Fri, 13 Jan 2023 at 08:51, Jungtaek Lim < >> kabhwan.opensou...@gmail.com> wrote: >> >>> bump for more visibility. >>> >>> On Wed, Jan 11, 2023 at 12:20 PM Jungtaek Lim < >>> kabhwan.opensou...@gmail.com> wrote: >>> Hi dev, I'd like to propose the deprecation of DStream in Spark 3.4, in favor of promoting Structured Streaming. (Sorry for the late proposal, if we don't make the change in 3.4, we will have to wait for another 6 months.) We have been focusing on Structured Streaming for years (across multiple major and minor versions), and during the time we haven't made any improvements for DStream. Furthermore, recently we updated the DStream doc to explicitly say DStream is a legacy project. https://spark.apache.org/docs/latest/streaming-programming-guide.html#note The baseline of deprecation is that we don't see a particular use case which only DStream solves. This is a different story with GraphX and MLLIB, as we don't have replacements for that. The proposal does not mean we will remove the API soon, as the Spark project has been making deprecation against public API. I don't intend to propose the target version for removal. The goal is to guide users to refrain from constructing a new workload with DStream. We might want to go with this in future, but it would require a new discussion thread at that time. What do you think? Thanks, Jungtaek Lim (HeartSaVioR) >>>
Re: [DISCUSS] Deprecate DStream in 3.4
Then, could you elaborate `the proposed code change` specifically? Maybe, usual deprecation warning logs and annotation on the API? On Thu, Jan 12, 2023 at 10:05 PM Jungtaek Lim wrote: > Maybe I need to clarify - my proposal is "explicitly" deprecating it, > which incurs code change for sure. Guidance on the Spark website is done > already as I mentioned - we updated the DStream doc page to mention that > DStream is a "legacy" project and users should move to SS. I don't feel > this is sufficient to refrain users from using it, hence initiating > this proposal. > > Sorry to make confusion. I just wanted to make sure the goal of the > proposal is not "removing" the API. The discussion on the removal of API > doesn't tend to go well, so I wanted to make sure I don't mean that. > > On Fri, Jan 13, 2023 at 2:46 PM Dongjoon Hyun > wrote: > >> +1 for the proposal (guiding only without any code change). >> >> Thanks, >> Dongjoon. >> >> On Thu, Jan 12, 2023 at 9:33 PM Shixiong Zhu wrote: >> >>> +1 >>> >>> >>> On Thu, Jan 12, 2023 at 5:08 PM Tathagata Das < >>> tathagata.das1...@gmail.com> wrote: >>> +1 On Thu, Jan 12, 2023 at 7:46 PM Hyukjin Kwon wrote: > +1 > > On Fri, 13 Jan 2023 at 08:51, Jungtaek Lim < > kabhwan.opensou...@gmail.com> wrote: > >> bump for more visibility. >> >> On Wed, Jan 11, 2023 at 12:20 PM Jungtaek Lim < >> kabhwan.opensou...@gmail.com> wrote: >> >>> Hi dev, >>> >>> I'd like to propose the deprecation of DStream in Spark 3.4, in >>> favor of promoting Structured Streaming. >>> (Sorry for the late proposal, if we don't make the change in 3.4, we >>> will have to wait for another 6 months.) >>> >>> We have been focusing on Structured Streaming for years (across >>> multiple major and minor versions), and during the time we haven't made >>> any >>> improvements for DStream. Furthermore, recently we updated the DStream >>> doc >>> to explicitly say DStream is a legacy project. >>> >>> https://spark.apache.org/docs/latest/streaming-programming-guide.html#note >>> >>> The baseline of deprecation is that we don't see a particular use >>> case which only DStream solves. This is a different story with GraphX >>> and >>> MLLIB, as we don't have replacements for that. >>> >>> The proposal does not mean we will remove the API soon, as the Spark >>> project has been making deprecation against public API. I don't intend >>> to >>> propose the target version for removal. The goal is to guide users to >>> refrain from constructing a new workload with DStream. We might want to >>> go >>> with this in future, but it would require a new discussion thread at >>> that >>> time. >>> >>> What do you think? >>> >>> Thanks, >>> Jungtaek Lim (HeartSaVioR) >>> >>
Re: [DISCUSS] Deprecate DStream in 3.4
There might be possible terminology differences, so let me elaborate the action item from the proposal explicitly: - Add "deprecation" annotation to the user-facing public API in streaming directory (DStream) - Write a release note to explicitly mention the deprecation. (Maybe promote again that they are encouraged to move to SS.) This is not an action item from the proposal: - Add (tentative) target version to remove the API on the deprecation message. Hope this makes the proposal crystally clear. On Fri, Jan 13, 2023 at 3:05 PM Jungtaek Lim wrote: > Maybe I need to clarify - my proposal is "explicitly" deprecating it, > which incurs code change for sure. Guidance on the Spark website is done > already as I mentioned - we updated the DStream doc page to mention that > DStream is a "legacy" project and users should move to SS. I don't feel > this is sufficient to refrain users from using it, hence initiating > this proposal. > > Sorry to make confusion. I just wanted to make sure the goal of the > proposal is not "removing" the API. The discussion on the removal of API > doesn't tend to go well, so I wanted to make sure I don't mean that. > > On Fri, Jan 13, 2023 at 2:46 PM Dongjoon Hyun > wrote: > >> +1 for the proposal (guiding only without any code change). >> >> Thanks, >> Dongjoon. >> >> On Thu, Jan 12, 2023 at 9:33 PM Shixiong Zhu wrote: >> >>> +1 >>> >>> >>> On Thu, Jan 12, 2023 at 5:08 PM Tathagata Das < >>> tathagata.das1...@gmail.com> wrote: >>> +1 On Thu, Jan 12, 2023 at 7:46 PM Hyukjin Kwon wrote: > +1 > > On Fri, 13 Jan 2023 at 08:51, Jungtaek Lim < > kabhwan.opensou...@gmail.com> wrote: > >> bump for more visibility. >> >> On Wed, Jan 11, 2023 at 12:20 PM Jungtaek Lim < >> kabhwan.opensou...@gmail.com> wrote: >> >>> Hi dev, >>> >>> I'd like to propose the deprecation of DStream in Spark 3.4, in >>> favor of promoting Structured Streaming. >>> (Sorry for the late proposal, if we don't make the change in 3.4, we >>> will have to wait for another 6 months.) >>> >>> We have been focusing on Structured Streaming for years (across >>> multiple major and minor versions), and during the time we haven't made >>> any >>> improvements for DStream. Furthermore, recently we updated the DStream >>> doc >>> to explicitly say DStream is a legacy project. >>> >>> https://spark.apache.org/docs/latest/streaming-programming-guide.html#note >>> >>> The baseline of deprecation is that we don't see a particular use >>> case which only DStream solves. This is a different story with GraphX >>> and >>> MLLIB, as we don't have replacements for that. >>> >>> The proposal does not mean we will remove the API soon, as the Spark >>> project has been making deprecation against public API. I don't intend >>> to >>> propose the target version for removal. The goal is to guide users to >>> refrain from constructing a new workload with DStream. We might want to >>> go >>> with this in future, but it would require a new discussion thread at >>> that >>> time. >>> >>> What do you think? >>> >>> Thanks, >>> Jungtaek Lim (HeartSaVioR) >>> >>
Re: [DISCUSS] Deprecate DStream in 3.4
Maybe I need to clarify - my proposal is "explicitly" deprecating it, which incurs code change for sure. Guidance on the Spark website is done already as I mentioned - we updated the DStream doc page to mention that DStream is a "legacy" project and users should move to SS. I don't feel this is sufficient to refrain users from using it, hence initiating this proposal. Sorry to make confusion. I just wanted to make sure the goal of the proposal is not "removing" the API. The discussion on the removal of API doesn't tend to go well, so I wanted to make sure I don't mean that. On Fri, Jan 13, 2023 at 2:46 PM Dongjoon Hyun wrote: > +1 for the proposal (guiding only without any code change). > > Thanks, > Dongjoon. > > On Thu, Jan 12, 2023 at 9:33 PM Shixiong Zhu wrote: > >> +1 >> >> >> On Thu, Jan 12, 2023 at 5:08 PM Tathagata Das < >> tathagata.das1...@gmail.com> wrote: >> >>> +1 >>> >>> On Thu, Jan 12, 2023 at 7:46 PM Hyukjin Kwon >>> wrote: >>> +1 On Fri, 13 Jan 2023 at 08:51, Jungtaek Lim < kabhwan.opensou...@gmail.com> wrote: > bump for more visibility. > > On Wed, Jan 11, 2023 at 12:20 PM Jungtaek Lim < > kabhwan.opensou...@gmail.com> wrote: > >> Hi dev, >> >> I'd like to propose the deprecation of DStream in Spark 3.4, in favor >> of promoting Structured Streaming. >> (Sorry for the late proposal, if we don't make the change in 3.4, we >> will have to wait for another 6 months.) >> >> We have been focusing on Structured Streaming for years (across >> multiple major and minor versions), and during the time we haven't made >> any >> improvements for DStream. Furthermore, recently we updated the DStream >> doc >> to explicitly say DStream is a legacy project. >> >> https://spark.apache.org/docs/latest/streaming-programming-guide.html#note >> >> The baseline of deprecation is that we don't see a particular use >> case which only DStream solves. This is a different story with GraphX and >> MLLIB, as we don't have replacements for that. >> >> The proposal does not mean we will remove the API soon, as the Spark >> project has been making deprecation against public API. I don't intend to >> propose the target version for removal. The goal is to guide users to >> refrain from constructing a new workload with DStream. We might want to >> go >> with this in future, but it would require a new discussion thread at that >> time. >> >> What do you think? >> >> Thanks, >> Jungtaek Lim (HeartSaVioR) >> >
Re: [DISCUSS] Deprecate DStream in 3.4
+1 On Thu, Jan 12, 2023 at 9:46 PM, Dongjoon Hyun < dongjoon.h...@gmail.com > wrote: > > +1 for the proposal (guiding only without any code change). > > > Thanks, > Dongjoon. > > On Thu, Jan 12, 2023 at 9:33 PM Shixiong Zhu < zsxwing@ gmail. com ( > zsxw...@gmail.com ) > wrote: > > >> +1 >> >> >> >> On Thu, Jan 12, 2023 at 5:08 PM Tathagata Das < tathagata. das1565@ gmail. >> com ( tathagata.das1...@gmail.com ) > wrote: >> >> >>> +1 >>> >>> >>> On Thu, Jan 12, 2023 at 7:46 PM Hyukjin Kwon < gurwls223@ gmail. com ( >>> gurwls...@gmail.com ) > wrote: >>> >>> +1 On Fri, 13 Jan 2023 at 08:51, Jungtaek Lim < kabhwan. opensource@ gmail. com ( kabhwan.opensou...@gmail.com ) > wrote: > bump for more visibility. > > On Wed, Jan 11, 2023 at 12:20 PM Jungtaek Lim < kabhwan. opensource@ > gmail. > com ( kabhwan.opensou...@gmail.com ) > wrote: > > >> Hi dev, >> >> >> I'd like to propose the deprecation of DStream in Spark 3.4, in favor of >> promoting Structured Streaming. >> (Sorry for the late proposal, if we don't make the change in 3.4, we will >> have to wait for another 6 months.) >> >> >> We have been focusing on Structured Streaming for years (across multiple >> major and minor versions), and during the time we haven't made any >> improvements for DStream. Furthermore, recently we updated the DStream >> doc >> to explicitly say DStream is a legacy project. >> https:/ / spark. apache. org/ docs/ latest/ streaming-programming-guide. >> html#note >> ( >> https://spark.apache.org/docs/latest/streaming-programming-guide.html#note >> ) >> >> >> >> The baseline of deprecation is that we don't see a particular use case >> which only DStream solves. This is a different story with GraphX and >> MLLIB, as we don't have replacements for that. >> >> >> The proposal does not mean we will remove the API soon, as the Spark >> project has been making deprecation against public API. I don't intend to >> propose the target version for removal. The goal is to guide users to >> refrain from constructing a new workload with DStream. We might want to >> go >> with this in future, but it would require a new discussion thread at that >> time. >> >> >> What do you think? >> >> >> Thanks, >> Jungtaek Lim (HeartSaVioR) >> > > >>> >>> >> >> > > smime.p7s Description: S/MIME Cryptographic Signature
Re: [DISCUSS] Deprecate DStream in 3.4
+1 for the proposal (guiding only without any code change). Thanks, Dongjoon. On Thu, Jan 12, 2023 at 9:33 PM Shixiong Zhu wrote: > +1 > > > On Thu, Jan 12, 2023 at 5:08 PM Tathagata Das > wrote: > >> +1 >> >> On Thu, Jan 12, 2023 at 7:46 PM Hyukjin Kwon wrote: >> >>> +1 >>> >>> On Fri, 13 Jan 2023 at 08:51, Jungtaek Lim >>> wrote: >>> bump for more visibility. On Wed, Jan 11, 2023 at 12:20 PM Jungtaek Lim < kabhwan.opensou...@gmail.com> wrote: > Hi dev, > > I'd like to propose the deprecation of DStream in Spark 3.4, in favor > of promoting Structured Streaming. > (Sorry for the late proposal, if we don't make the change in 3.4, we > will have to wait for another 6 months.) > > We have been focusing on Structured Streaming for years (across > multiple major and minor versions), and during the time we haven't made > any > improvements for DStream. Furthermore, recently we updated the DStream doc > to explicitly say DStream is a legacy project. > > https://spark.apache.org/docs/latest/streaming-programming-guide.html#note > > The baseline of deprecation is that we don't see a particular use case > which only DStream solves. This is a different story with GraphX and > MLLIB, > as we don't have replacements for that. > > The proposal does not mean we will remove the API soon, as the Spark > project has been making deprecation against public API. I don't intend to > propose the target version for removal. The goal is to guide users to > refrain from constructing a new workload with DStream. We might want to go > with this in future, but it would require a new discussion thread at that > time. > > What do you think? > > Thanks, > Jungtaek Lim (HeartSaVioR) >
Re: [DISCUSS] Deprecate DStream in 3.4
+1 On Thu, Jan 12, 2023 at 5:08 PM Tathagata Das wrote: > +1 > > On Thu, Jan 12, 2023 at 7:46 PM Hyukjin Kwon wrote: > >> +1 >> >> On Fri, 13 Jan 2023 at 08:51, Jungtaek Lim >> wrote: >> >>> bump for more visibility. >>> >>> On Wed, Jan 11, 2023 at 12:20 PM Jungtaek Lim < >>> kabhwan.opensou...@gmail.com> wrote: >>> Hi dev, I'd like to propose the deprecation of DStream in Spark 3.4, in favor of promoting Structured Streaming. (Sorry for the late proposal, if we don't make the change in 3.4, we will have to wait for another 6 months.) We have been focusing on Structured Streaming for years (across multiple major and minor versions), and during the time we haven't made any improvements for DStream. Furthermore, recently we updated the DStream doc to explicitly say DStream is a legacy project. https://spark.apache.org/docs/latest/streaming-programming-guide.html#note The baseline of deprecation is that we don't see a particular use case which only DStream solves. This is a different story with GraphX and MLLIB, as we don't have replacements for that. The proposal does not mean we will remove the API soon, as the Spark project has been making deprecation against public API. I don't intend to propose the target version for removal. The goal is to guide users to refrain from constructing a new workload with DStream. We might want to go with this in future, but it would require a new discussion thread at that time. What do you think? Thanks, Jungtaek Lim (HeartSaVioR) >>>
Re: [DISCUSS] Deprecate DStream in 3.4
+1 On Thu, Jan 12, 2023 at 7:46 PM Hyukjin Kwon wrote: > +1 > > On Fri, 13 Jan 2023 at 08:51, Jungtaek Lim > wrote: > >> bump for more visibility. >> >> On Wed, Jan 11, 2023 at 12:20 PM Jungtaek Lim < >> kabhwan.opensou...@gmail.com> wrote: >> >>> Hi dev, >>> >>> I'd like to propose the deprecation of DStream in Spark 3.4, in favor of >>> promoting Structured Streaming. >>> (Sorry for the late proposal, if we don't make the change in 3.4, we >>> will have to wait for another 6 months.) >>> >>> We have been focusing on Structured Streaming for years (across multiple >>> major and minor versions), and during the time we haven't made any >>> improvements for DStream. Furthermore, recently we updated the DStream doc >>> to explicitly say DStream is a legacy project. >>> >>> https://spark.apache.org/docs/latest/streaming-programming-guide.html#note >>> >>> The baseline of deprecation is that we don't see a particular use case >>> which only DStream solves. This is a different story with GraphX and MLLIB, >>> as we don't have replacements for that. >>> >>> The proposal does not mean we will remove the API soon, as the Spark >>> project has been making deprecation against public API. I don't intend to >>> propose the target version for removal. The goal is to guide users to >>> refrain from constructing a new workload with DStream. We might want to go >>> with this in future, but it would require a new discussion thread at that >>> time. >>> >>> What do you think? >>> >>> Thanks, >>> Jungtaek Lim (HeartSaVioR) >>> >>
Re: [DISCUSS] Deprecate DStream in 3.4
+1 On Fri, 13 Jan 2023 at 08:51, Jungtaek Lim wrote: > bump for more visibility. > > On Wed, Jan 11, 2023 at 12:20 PM Jungtaek Lim < > kabhwan.opensou...@gmail.com> wrote: > >> Hi dev, >> >> I'd like to propose the deprecation of DStream in Spark 3.4, in favor of >> promoting Structured Streaming. >> (Sorry for the late proposal, if we don't make the change in 3.4, we will >> have to wait for another 6 months.) >> >> We have been focusing on Structured Streaming for years (across multiple >> major and minor versions), and during the time we haven't made any >> improvements for DStream. Furthermore, recently we updated the DStream doc >> to explicitly say DStream is a legacy project. >> https://spark.apache.org/docs/latest/streaming-programming-guide.html#note >> >> The baseline of deprecation is that we don't see a particular use case >> which only DStream solves. This is a different story with GraphX and MLLIB, >> as we don't have replacements for that. >> >> The proposal does not mean we will remove the API soon, as the Spark >> project has been making deprecation against public API. I don't intend to >> propose the target version for removal. The goal is to guide users to >> refrain from constructing a new workload with DStream. We might want to go >> with this in future, but it would require a new discussion thread at that >> time. >> >> What do you think? >> >> Thanks, >> Jungtaek Lim (HeartSaVioR) >> >
Re: [DISCUSS] Deprecate DStream in 3.4
bump for more visibility. On Wed, Jan 11, 2023 at 12:20 PM Jungtaek Lim wrote: > Hi dev, > > I'd like to propose the deprecation of DStream in Spark 3.4, in favor of > promoting Structured Streaming. > (Sorry for the late proposal, if we don't make the change in 3.4, we will > have to wait for another 6 months.) > > We have been focusing on Structured Streaming for years (across multiple > major and minor versions), and during the time we haven't made any > improvements for DStream. Furthermore, recently we updated the DStream doc > to explicitly say DStream is a legacy project. > https://spark.apache.org/docs/latest/streaming-programming-guide.html#note > > The baseline of deprecation is that we don't see a particular use case > which only DStream solves. This is a different story with GraphX and MLLIB, > as we don't have replacements for that. > > The proposal does not mean we will remove the API soon, as the Spark > project has been making deprecation against public API. I don't intend to > propose the target version for removal. The goal is to guide users to > refrain from constructing a new workload with DStream. We might want to go > with this in future, but it would require a new discussion thread at that > time. > > What do you think? > > Thanks, > Jungtaek Lim (HeartSaVioR) >
