ganeshashree commented on PR #57813: URL: https://github.com/apache/spark/pull/57813#issuecomment-5328164000
> Codex found the issue which is actually an existing one but severe with outer-like joins... If we have only right side watermark (as we documented left side watermark is optional), it actually does not build the predicate to evict the state on the left side. For outer-like joins e.g. left outer and left anti, this won't produce unmatched outer results. Can you check with the above as constructing tests and confirming the behavior, and if Codex is right about it, block analyzer to allow only right watermark to be present, and also update the doc? We should make a fix for left semi and left outer as well, but for them we may need a SQL config to fall back to old behavior, and update the migration guide doc. Fix in the latest FOLLOWUP, scoped to left anti (analyzer + docs): - Range-condition path: now requires a watermark on the left side too (right-only is rejected), which lets the left state be evicted. - Equi-join path: requires the watermark on the right join key at the ordinal used for eviction (isWatermarkOnRightEvictionJoinKey), so the right side is actually late-filtered on the matching dimension. This also rejects a left-only key watermark, a watermark on an unrelated right column, and a mismatched-ordinal composite key, all of which would otherwise let a late right row invalidate an already-emitted anti-row. - Docs/matrix updated, plus the immediate-emission caveat for left rows failing a deterministic left-only pre-join filter. Left outer/semi are untouched here, as you suggested, they need a SQL-config fallback + migration-guide note, so I'd do that as a separate change. Two related gaps are also pre-existing and shared across all stream-stream joins, so I left them for that follow-up: (1) same-batch matches when the eviction watermark runs ahead of the late-event watermark (SPARK-49829-style split), and (2) range-path column-level precision (which attributes the range constraints vs which is watermarked). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
