HeartSaVioR commented on PR #57813: URL: https://github.com/apache/spark/pull/57813#issuecomment-5323410873
Codex found the issue which is actually an existing one but severe with outer-like joins... If we have only right side watermark (as we documented left side watermark is optional), it actually does not build the predicate to evict the state on the left side. For outer-like joins e.g. left outer and left anti, this won't produce unmatched outer results. Can you check with the above as constructing tests and confirming the behavior, and if Codex is right about it, block analyzer to allow only right watermark to be present, and also update the doc? We should make a fix for left semi and left outer as well, but for them we may need a SQL config to fall back to old behavior, and update the migration guide doc. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
