[ 
https://issues.apache.org/jira/browse/SPARK-61509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jungtaek Lim updated SPARK-61509:
---------------------------------
    Affects Version/s: 4.4.0

> Fail at start when a held-back stream-stream join output watermark reaches a 
> later join, event-time timeout or aggregation
> --------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61509
>                 URL: https://issues.apache.org/jira/browse/SPARK-61509
>             Project: Spark
>          Issue Type: Improvement
>          Components: Structured Streaming
>    Affects Versions: 5.0.0, 4.4.0
>            Reporter: Jungtaek Lim
>            Priority: Major
>
> h3. Problem
> SPARK-61345 fails a query at start when an append-mode aggregation uses the 
> event time of a
> stream-stream join input whose state the join cannot bound. The join holds 
> back its output
> watermark for that input (the input's contribution is 0), so the aggregation 
> never emits. The
> check judges only the nearest join (the tracer stops there) and only 
> append-mode aggregations. Three
> shapes with the same cause still start without an error or a warning:
> Let J1 = A inner join B on {{ak = bk AND at = bt}} with a watermark only on 
> {{at}}. J1 holds the
> watermark of {{at}} at 0.
> # J2 = J1 left outer join C on {{ak = ck AND at = ct}}, with C watermarked: 
> J2 never evicts the
>   J1 rows and never emits them as unmatched.
> # An append-mode window aggregation over {{ct}} after J2 never emits, but the 
> check passes because
>   it judges only J2.
> # A {{flatMapGroupsWithState}} or {{transformWithState}} operator with 
> event-time timeouts after
>   J1 never fires its timeouts.
> The analyzer already rejects append-mode aggregations and event-time timeouts 
> without a watermark,
> and outer joins that can never finalize. A watermark held at 0 has the same 
> effect.
> h3. Proposal
> Make {{checkJoinOutputWatermarkConsumers}} follow the tracer through later 
> joins to the join that
> holds the watermark back, and also reject stream-stream outer joins and 
> operators with event-time
> timeouts, naming the operator. The same conf, 
> {{spark.sql.streaming.join.checkFinalization.enabled}},
> turns the error into a warning, as in SPARK-61345.
> h3. Why a separate change
> Covering later joins needs the propagated watermark through each join, not 
> one join's state bound,
> and it touches every stateful operator kind. It deserves its own review. The 
> migration guide
> documents the current scope: such operators keep running and do not emit 
> results that wait for the
> watermark.
> Related: SPARK-61345.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to