Sorry, some links are wrong. Here I fixed it to the correct link: -
Address some unexpected query results (see the Appendix <https://docs.google.com/document/d/1uWRMbN927cRXhSm5oeV3pbwb6o73am4r1ckEJDhAHa0/edit#heading=h.ehd1najwmeos> of the design document for examples). - Reduce the likelihood of RocksDB corruptions caused by checkpointing a new version based on a previous version that is not the official version. The first section <https://docs.google.com/document/d/1uWRMbN927cRXhSm5oeV3pbwb6o73am4r1ckEJDhAHa0/edit#heading=h.z2wsnxxpksge> of the documentation explains how this can happen. The other links are correct. Thanks, Siying On Tue, Aug 27, 2024 at 7:21 PM Siying Dong <siying.d...@databricks.com> wrote: > Hi, > > We are planning to work on a feature that improves how the RocksDB State > Store organizes checkpoint versions in cloud storage. This improvement aims > to: > > - > > Address some unexpected query results (see the Appendix > > <https://docs.google.com/document/d/1uWRMbN927cRXhSm5oeV3pbwb6o73am4r1ckEJDhAHa0/edit#heading=h.ehd1najwmeos> > of the design document for examples). > - > > Reduce the likelihood of RocksDB corruptions caused by checkpointing a > new version based on a previous version that is not the official version. > The first section > > <https://docs.google.com/document/d/1pT5bwW325VndVH09aZd7TykKXbBP5uCrNIQOC0uDoLc/edit#heading=h.z2wsnxxpksge> > of the documentation explains how this can happen. > > The core idea is to ensure that the state store checkpoint for a batch > version strictly depends on the previously checkpointed version. We propose > achieving this by introducing a unique ID for each state store checkpoint > and persisting this ID in the commit logs. This feature will be controlled > by a new SQL configuration, allowing users to switch between the old and > new modes when running a supported release. > > You can find the design document here > <https://docs.google.com/document/d/1pT5bwW325VndVH09aZd7TykKXbBP5uCrNIQOC0uDoLc/edit> > and the related JIRA ticket SPARK-49374 > <https://issues.apache.org/jira/browse/SPARK-49374>. > > As we believe this improvement isn't significant enough to require going > through the SPIP process, we don't plan to pursue that route. However, if > you have any concerns, please let us know. We would also greatly appreciate > any comments or feedback on the design. > > Best regards, > > > Siying >
