[
https://issues.apache.org/jira/browse/SPARK-58904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated SPARK-58904:
---------------------------------
Description:
Stream-stream left semi and left outer joins rely on the left-side state being
evicted by the watermark, but the analyzer does not require the watermark to be
placed where eviction can happen. `checkForStreamStreamJoinWatermark` accepts a
right-only (or mis-placed) watermark, so at runtime no left-eviction predicate
is built: left outer silently drops its unmatched output, and left semi state
grows without bound. Left outer has a second issue: an already-emitted
unmatched row can be invalidated by a late row unless both sides are
late-filtered on the eviction dimension.
The analyzer must ensure that accepted watermark placement can drive the state
eviction and late-row filtering required by the join type. Otherwise, left
outer joins can omit unmatched output, left semi state can grow without bound,
and late rows can invalidate previously emitted results. The fix should also
audit the corresponding requirements for right/full outer and future left anti
joins.
was:
Stream-stream left semi and left outer joins rely on the left-side state being
evicted by the watermark, but the analyzer does not require the watermark to be
placed where eviction can happen. `checkForStreamStreamJoinWatermark` accepts a
right-only (or mis-placed) watermark, so at runtime no left-eviction predicate
is built: left outer silently drops its unmatched output, and left semi state
grows without bound. Left outer has a second issue: an already-emitted
unmatched row can be invalidated by a late row unless both sides are
late-filtered on the eviction dimension.
This adds a shared analyzer check with two requirements:
(1) left-state eviction for left semi and left outer (equi path: eviction key
watermarked; range path: range bound between watermarked attributes on both
sides), and
(2) no invalidation by late rows for left outer (both eviction-ordinal keys
watermarked). Offending queries are rejected at analysis time.
Since this rejects queries that previously ran, it is gated by a kill switch,
`spark.sql.streaming.join.stricterWatermarkRequirements.enabled` (default
true); set false to restore the old behavior. A migration-guide entry documents
the change.
Right outer and full outer are out of scope. This is a prerequisite for the
stream-stream left anti join (SPARK-58611), which reuses this check and always
enforces it.
> Enforce correct watermark placement for stream-stream left semi and left
> outer joins
> ------------------------------------------------------------------------------------
>
> Key: SPARK-58904
> URL: https://issues.apache.org/jira/browse/SPARK-58904
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 4.3.0
> Reporter: Ganesha S
> Priority: Critical
> Labels: correctness
>
> Stream-stream left semi and left outer joins rely on the left-side state
> being evicted by the watermark, but the analyzer does not require the
> watermark to be placed where eviction can happen.
> `checkForStreamStreamJoinWatermark` accepts a right-only (or mis-placed)
> watermark, so at runtime no left-eviction predicate is built: left outer
> silently drops its unmatched output, and left semi state grows without bound.
> Left outer has a second issue: an already-emitted unmatched row can be
> invalidated by a late row unless both sides are late-filtered on the eviction
> dimension.
> The analyzer must ensure that accepted watermark placement can drive the
> state eviction and late-row filtering required by the join type. Otherwise,
> left outer joins can omit unmatched output, left semi state can grow without
> bound, and late rows can invalidate previously emitted results. The fix
> should also audit the corresponding requirements for right/full outer and
> future left anti joins.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
