HeartSaVioR edited a comment on issue #23634: [SPARK-26154][SS] Streaming left/right outer join should not return outer nulls for already matched rows URL: https://github.com/apache/spark/pull/23634#issuecomment-464929292 I've addressed changing approach from having 3rd state store to adding `matched` field to 2nd state store. Please take a look at last commit I made to change the approach: https://github.com/apache/spark/pull/23634/commits/912a25023c86183a653c52e0fd9aef5651a6ad3d Though it didn't roll back refactoring I've done for state versioning, it would share a view to determine the difference between 3rd state store vs matched field in 2nd store. Honestly, this commit represents the rationalization why refactoring was necessary. Less than 200 lines needed for applying suggestion on new approach, as well as keeping versioning for state formats. The change is done only in state handler side, and doesn't touch any part of join logic.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
