[
https://issues.apache.org/jira/browse/SPARK-62057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated SPARK-62057:
---------------------------------
Affects Version/s: 4.4.0
> Stream-stream left semi join over a streaming union fails with a missing
> state file when a union input gets its first data after the first micro-batch
> ------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-62057
> URL: https://issues.apache.org/jira/browse/SPARK-62057
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 4.3.0, 5.0.0, 4.4.0
> Reporter: Jungtaek Lim
> Priority: Major
>
> h3. Problem
> A stream-stream left semi join whose left input is a streaming union can fail
> in the second
> micro-batch with {{CANNOT_LOAD_STATE_STORE}} ("... state/1/2/1.changelog does
> not exist"). It
> happens when one union input has no data in the first micro-batch and gets
> data later.
> The physical plan changes between the batches. In batch 0 the plan has one
> {{StreamingSymmetricHashJoinExec}} (operator id 0). In batch 1 a stateful
> operator with id 1
> tries to load state version 1, which batch 0 never wrote. This fits the
> optimizer pushing the
> left semi join below the Union ({{PushDownLeftSemiAntiJoin}}) once both union
> inputs have data,
> which gives one join per union input; the batch 1 plan itself was not
> captured, because the
> batch fails.
> h3. Reproducer
> Scala, {{MemoryStream}},
> {{spark.sql.streaming.noDataMicroBatches.enabled=false}}, RocksDB.
> Fails on state formats 2, 3 and 4.
> {code:scala}
> val first = input1.toDF().toDF("lk", "t")
> .select($"lk", timestamp_seconds($"t").as("lt"))
> .withWatermark("lt", "10 seconds")
> val second = input2.toDF().toDF("lk", "t")
> .select($"lk", timestamp_seconds($"t").as("lt")) // no watermark on
> this input
> val right = rightInput.toDF().toDF("rk", "s")
> .select($"rk", timestamp_seconds($"s").as("rt"))
> val joined = first.union(second)
> .join(right, expr("lk = rk AND lt = rt"), "left_semi")
> .select($"lk", $"lt".cast("long"))
> // batch 0: input1 (1, 5) and rightInput (1, 5) -> (1, 5); one stream-stream
> join, id 0
> // batch 1: input2 (2, 6) and rightInput (2, 6) -> expected (2, 6)
> {code}
> - *Expected:* {{(1, 5)}}, then {{(2, 6)}}.
> - *Actual:* batch 1 fails with
> {{[CANNOT_LOAD_STATE_STORE.CANNOT_READ_STREAMING_STATE_FILE]}},
> for {{state/1/<partition>/1.changelog}}.
> ||State format||Result on unmodified master {{4ccce2781cb}}||
> |2|fails in batch 1|
> |3|fails in batch 1|
> |4|fails in batch 1|
> Note: a variant where both union inputs and the right input have watermarks,
> with
> {{noDataMicroBatches}} on, did not fail; there the final plan had two joins.
> Which shapes avoid
> the plan change was not narrowed down.
> h3. Notes
> - Reproduced on unmodified master {{4ccce2781cb}}, with the query above:
> batch 0 plans one
> stream-stream join (operator id 0), and batch 1 fails loading
> {{state/1/2/1.changelog}}.
> - Fix idea: the number of stateful operators must not depend on whether a
> source has data in a
> batch. Either do not push a streaming left semi or anti join below a Union,
> or plan an empty
> source so that the plan shape stays the same across batches.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]