Thanks Anish for proposing SPIP and initiating this thread! I believe this SPIP will help a bunch of complex use cases on streaming.
dev@: We are coincidentally initiating this discussion in thanksgiving holidays. We understand people in the US may not have time to review the SPIP, and we plan to bump this thread in early next week. We are open for any feedback from non-US during the holiday. We can either address feedback altogether after the holiday (Anish is in the US) or I can answer if the feedback is more about the question. Thanks! On Thu, Nov 23, 2023 at 5:27 AM Anish Shrigondekar < anish.shrigonde...@databricks.com> wrote: > Hi dev, > > I would like to start a discussion on "Structured Streaming - Arbitrary > State API v2". This proposal aims to address a bunch of limitations we see > today using mapGroupsWithState/flatMapGroupsWithState operator. The > detailed set of limitations is described in the SPIP doc. > > We propose to support various features such as multiple state variables > (flexible data modeling), composite types, enhanced timer functionality, > support for chaining operators after new operator, handling initial state > along with state data source, schema evolution etc This will allow users to > write more powerful streaming state management logic primarily used in > operational use-cases. Other built-in stateful operators could also benefit > from such changes in the future. > > JIRA: https://issues.apache.org/jira/browse/SPARK-45939 > SPIP: > https://docs.google.com/document/d/1QtC5qd4WQEia9kl1Qv74WE0TiXYy3x6zeTykygwPWig/edit?usp=sharing > Design Doc: > https://docs.google.com/document/d/1QjZmNZ-fHBeeCYKninySDIoOEWfX6EmqXs2lK097u9o/edit?usp=sharing > > cc - @Jungtaek Lim <kabhwan.opensou...@gmail.com> who has graciously > agreed to be the shepherd for this project > > Looking forward to your feedback ! > > Thanks, > Anish >
