Ganesha S created SPARK-58904:
---------------------------------
Summary: Enforce correct watermark placement for stream-stream
left semi and left outer joins
Key: SPARK-58904
URL: https://issues.apache.org/jira/browse/SPARK-58904
Project: Spark
Issue Type: Improvement
Components: Structured Streaming
Affects Versions: 4.3.0
Reporter: Ganesha S
Fix For: 4.4.0
Stream-stream left semi and left outer joins rely on the left-side state being
evicted by the watermark, but the analyzer does not require the watermark to be
placed where eviction can happen. `checkForStreamStreamJoinWatermark` accepts a
right-only (or mis-placed) watermark, so at runtime no left-eviction predicate
is built: left outer silently drops its unmatched output, and left semi state
grows without bound. Left outer has a second issue: an already-emitted
unmatched row can be invalidated by a late row unless both sides are
late-filtered on the eviction dimension.
This adds a shared analyzer check with two requirements:
(1) left-state eviction for left semi and left outer (equi path: eviction key
watermarked; range path: range bound between watermarked attributes on both
sides), and
(2) no invalidation by late rows for left outer (both eviction-ordinal keys
watermarked). Offending queries are rejected at analysis time.
Since this rejects queries that previously ran, it is gated by a kill switch,
`spark.sql.streaming.join.stricterWatermarkRequirements.enabled` (default
true); set false to restore the old behavior. A migration-guide entry documents
the change.
Right outer and full outer are out of scope. This is a prerequisite for the
stream-stream left anti join (SPARK-58611), which reuses this check and always
enforces it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
