Thanks everyone for the feedback! Given that we get positive feedback without major concerns, I will initiate the vote thread soon. Please make a vote in that thread as well.
Thanks again! On Tue, Jan 9, 2024 at 7:44 AM Bhuwan Sahni <bhuwan.sa...@databricks.com.invalid> wrote: > +1 on the newer APIs. I believe these APIs provide a much powerful > mechanism for the user to perform arbitrary state management in Structured > Streaming queries. > > Thanks > Bhuwan Sahni > > On Mon, Jan 8, 2024 at 10:07 AM L. C. Hsieh <vii...@gmail.com> wrote: > >> +1 >> >> I left some comments in the SPIP doc and got replies quickly. The new >> API looks good and more comprehensive. I think it will help Spark >> Structured Streaming to be more useful in more complicated streaming >> use cases. >> >> On Fri, Jan 5, 2024 at 8:15 PM Burak Yavuz <brk...@gmail.com> wrote: >> > >> > I'm also a +1 on the newer APIs. We had a lot of learnings from using >> flatMapGroupsWithState and I believe that we can make the APIs a lot easier >> to use. >> > >> > On Wed, Nov 29, 2023 at 6:43 PM Anish Shrigondekar >> <anish.shrigonde...@databricks.com.invalid> wrote: >> >> >> >> Hi dev, >> >> >> >> Addressed the comments that Jungtaek had on the doc. Bumping the >> thread once again to see if other folks have any feedback on the proposal. >> >> >> >> Thanks, >> >> Anish >> >> >> >> On Mon, Nov 27, 2023 at 8:15 PM Jungtaek Lim < >> kabhwan.opensou...@gmail.com> wrote: >> >>> >> >>> Kindly bump for better reach after the long holiday. Please kindly >> review the proposal which opens the chance to address complex use cases of >> streaming. Thanks! >> >>> >> >>> On Thu, Nov 23, 2023 at 8:19 AM Jungtaek Lim < >> kabhwan.opensou...@gmail.com> wrote: >> >>>> >> >>>> Thanks Anish for proposing SPIP and initiating this thread! I >> believe this SPIP will help a bunch of complex use cases on streaming. >> >>>> >> >>>> dev@: We are coincidentally initiating this discussion in >> thanksgiving holidays. We understand people in the US may not have time to >> review the SPIP, and we plan to bump this thread in early next week. We are >> open for any feedback from non-US during the holiday. We can either address >> feedback altogether after the holiday (Anish is in the US) or I can answer >> if the feedback is more about the question. Thanks! >> >>>> >> >>>> On Thu, Nov 23, 2023 at 5:27 AM Anish Shrigondekar < >> anish.shrigonde...@databricks.com> wrote: >> >>>>> >> >>>>> Hi dev, >> >>>>> >> >>>>> I would like to start a discussion on "Structured Streaming - >> Arbitrary State API v2". This proposal aims to address a bunch of >> limitations we see today using mapGroupsWithState/flatMapGroupsWithState >> operator. The detailed set of limitations is described in the SPIP doc. >> >>>>> >> >>>>> We propose to support various features such as multiple state >> variables (flexible data modeling), composite types, enhanced timer >> functionality, support for chaining operators after new operator, handling >> initial state along with state data source, schema evolution etc This will >> allow users to write more powerful streaming state management logic >> primarily used in operational use-cases. Other built-in stateful operators >> could also benefit from such changes in the future. >> >>>>> >> >>>>> JIRA: https://issues.apache.org/jira/browse/SPARK-45939 >> >>>>> SPIP: >> https://docs.google.com/document/d/1QtC5qd4WQEia9kl1Qv74WE0TiXYy3x6zeTykygwPWig/edit?usp=sharing >> >>>>> Design Doc: >> https://docs.google.com/document/d/1QjZmNZ-fHBeeCYKninySDIoOEWfX6EmqXs2lK097u9o/edit?usp=sharing >> >>>>> >> >>>>> cc - @Jungtaek Lim who has graciously agreed to be the shepherd >> for this project >> >>>>> >> >>>>> Looking forward to your feedback ! >> >>>>> >> >>>>> Thanks, >> >>>>> Anish >> >> --------------------------------------------------------------------- >> To unsubscribe e-mail: dev-unsubscr...@spark.apache.org >> >> > > -- > <http://www.databricks.com> > *Bhuwan Sahni* > Staff Software Engineer > > bhuwan.sa...@databricks.com > 500 108th Ave. NE > Bellevue, WA 98004 > USA >
