[
https://issues.apache.org/jira/browse/SPARK-61509?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated SPARK-61509:
---------------------------------
Affects Version/s: 4.4.0
> Fail at start when a held-back stream-stream join output watermark reaches a
> later join, event-time timeout or aggregation
> --------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61509
> URL: https://issues.apache.org/jira/browse/SPARK-61509
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 5.0.0, 4.4.0
> Reporter: Jungtaek Lim
> Priority: Major
>
> h3. Problem
> SPARK-61345 fails a query at start when an append-mode aggregation uses the
> event time of a
> stream-stream join input whose state the join cannot bound. The join holds
> back its output
> watermark for that input (the input's contribution is 0), so the aggregation
> never emits. The
> check judges only the nearest join (the tracer stops there) and only
> append-mode aggregations. Three
> shapes with the same cause still start without an error or a warning:
> Let J1 = A inner join B on {{ak = bk AND at = bt}} with a watermark only on
> {{at}}. J1 holds the
> watermark of {{at}} at 0.
> # J2 = J1 left outer join C on {{ak = ck AND at = ct}}, with C watermarked:
> J2 never evicts the
> J1 rows and never emits them as unmatched.
> # An append-mode window aggregation over {{ct}} after J2 never emits, but the
> check passes because
> it judges only J2.
> # A {{flatMapGroupsWithState}} or {{transformWithState}} operator with
> event-time timeouts after
> J1 never fires its timeouts.
> The analyzer already rejects append-mode aggregations and event-time timeouts
> without a watermark,
> and outer joins that can never finalize. A watermark held at 0 has the same
> effect.
> h3. Proposal
> Make {{checkJoinOutputWatermarkConsumers}} follow the tracer through later
> joins to the join that
> holds the watermark back, and also reject stream-stream outer joins and
> operators with event-time
> timeouts, naming the operator. The same conf,
> {{spark.sql.streaming.join.checkFinalization.enabled}},
> turns the error into a warning, as in SPARK-61345.
> h3. Why a separate change
> Covering later joins needs the propagated watermark through each join, not
> one join's state bound,
> and it touches every stateful operator kind. It deserves its own review. The
> migration guide
> documents the current scope: such operators keep running and do not emit
> results that wait for the
> watermark.
> Related: SPARK-61345.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]