HeartSaVioR edited a comment on pull request #32834: URL: https://github.com/apache/spark/pull/32834#issuecomment-857426847
Please be aware of "existing state" when dealing with stateful operation. Existing states should have been stored as not-reordered one, hence once we change the key to be reordered one, it will end up with different partitioning, and also unable to find from state store. These are correctness issues. Even we put aside the existing state a bit, we never ever be able to change the logic on reordering. The different output of reorder breaks compatibility again. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
