HeartSaVioR commented on pull request #31989: URL: https://github.com/apache/spark/pull/31989#issuecomment-878945636
Thanks @viirya for reviewing! I reflected review comments. Btw, I also made a change via f12cda8 as I realized we don't need to union windows from both (old/new) sides. (Actually I found this during experiment but I forgot to reflect here. My bad.) Simply thinking, given old windows and new windows, we can simply remove all old windows and add all new windows as new windows are the thing we want to "retain" in state store. (See method contract on `updateSessions`.) Adding some optimization on reducing "delta"/"tombstone", we can remove old windows which state keys are not updated by new windows, and add new windows to do "upsert". -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
