ganeshashree commented on PR #57813:
URL: https://github.com/apache/spark/pull/57813#issuecomment-5338341767
> Sorry to push you on more work than anticipated, but I think the best
direction is to fix the case of left semi & left outer with SQL config as a
kill switch, and have left anti on top of the fix. We have common code which
has the issue and it's weird we workaround to left anti to not use common code
and try to fix it only for left anti. (Actually Codex still found a couple
issues and trying to fix the issue in left anti would be thrown out as we will
fix the shared code anyway.)
>
> Could you please consider expanding your work to fix left semi & left
outer as well? I can volunteer if you would like to keep the scope as left
anti. Thanks!
Thanks, expanded in the follow-up commit, no more left-anti workaround. The
check now lives in shared code (`checkStreamStreamJoinWatermarkPlacement`)
across left semi/outer/anti:
- Left semi/outer are gated by a kill switch,
`spark.sql.streaming.join.stricterWatermarkRequirements.enabled` (default
true); left anti sits on top and always enforces it.
- Equi path now requires both eviction keys watermarked; range path requires
the bound between watermarked attributes on both sides.
PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
