Sorry, please use this link instead for SPIP doc:
https://docs.google.com/document/d/1_iVf_CIu2RZd3yWWF6KoRNlBiz5NbSIK0yThqG0EvPY/edit?usp=sharing
On Thu, Oct 12, 2023 at 3:58 PM Jungtaek Lim
wrote:
> Hi dev,
>
> I'd like to start a discussion on "State Data Source - Reader".
>
> This proposal aims to introduce a new data source "statestore" which
> enables reading the state rows from existing checkpoint via offline (batch)
> query. This will enable users to 1) create unit tests against stateful
> query verifying the state value (especially flatMapGroupsWithState), 2)
> gather more context on the status when an incident occurs, especially for
> incorrect output.
>
> *SPIP*:
> https://docs.google.com/document/d/1HjEupRv8TRFeULtJuxRq_tEG1Wq-9UNu-ctGgCYRke0/edit?usp=sharing
> *JIRA*: https://issues.apache.org/jira/browse/SPARK-45511
>
> Looking forward to your feedback!
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
> ps. The scope of the project is narrowed to the reader in this SPIP, since
> the writer requires us to consider more cases. We are planning on it.
>
