[
https://issues.apache.org/jira/browse/SPARK-58904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-58904:
-----------------------------------
Labels: pull-request-available (was: )
> Enforce correct watermark placement for stream-stream left semi and left
> outer joins
> ------------------------------------------------------------------------------------
>
> Key: SPARK-58904
> URL: https://issues.apache.org/jira/browse/SPARK-58904
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 4.3.0
> Reporter: Ganesha S
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.4.0
>
>
> Stream-stream left semi and left outer joins rely on the left-side state
> being evicted by the watermark, but the analyzer does not require the
> watermark to be placed where eviction can happen.
> `checkForStreamStreamJoinWatermark` accepts a right-only (or mis-placed)
> watermark, so at runtime no left-eviction predicate is built: left outer
> silently drops its unmatched output, and left semi state grows without bound.
> Left outer has a second issue: an already-emitted unmatched row can be
> invalidated by a late row unless both sides are late-filtered on the eviction
> dimension.
> This adds a shared analyzer check with two requirements:
> (1) left-state eviction for left semi and left outer (equi path: eviction key
> watermarked; range path: range bound between watermarked attributes on both
> sides), and
> (2) no invalidation by late rows for left outer (both eviction-ordinal keys
> watermarked). Offending queries are rejected at analysis time.
> Since this rejects queries that previously ran, it is gated by a kill switch,
> `spark.sql.streaming.join.stricterWatermarkRequirements.enabled` (default
> true); set false to restore the old behavior. A migration-guide entry
> documents the change.
> Right outer and full outer are out of scope. This is a prerequisite for the
> stream-stream left anti join (SPARK-58611), which reuses this check and
> always enforces it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
