Jungtaek Lim created SPARK-62075:
------------------------------------
Summary: Stream-stream outer join with state format 4 never emits
rows of an input without its own watermark that a legacy-mode run stored
Key: SPARK-62075
URL: https://issues.apache.org/jira/browse/SPARK-62075
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 5.0.0
Reporter: Jungtaek Lim
h3. Problem
With SPARK-61990 (#60669), a stream-stream join evicts the state of an input
without its own
watermark when the other input's watermark and a range condition bound one of
its TIMESTAMP
columns. With state format version 4, such an input's rows are stored with an
index entry on
that column, and eviction finds them through that index.
In the legacy mode
({{spark.sql.streaming.statefulOperator.allowMultiple=false}}), that input has
no eviction predicate, so rows stored during a legacy-mode run get no index
entry. If a checkpoint
started in the default mode (V4 index version 1) is restarted in the legacy
mode and then in the
default mode again, the restart check passes both times, and the rows stored
during the legacy
run are never evicted afterwards: an outer join never emits them, and they stay
in state for
good. State formats 1 to 3 evict row by row and are not affected.
h3. Reproducer
Scala, {{MemoryStream}}, {{spark.sql.streaming.join.stateFormatVersion=4}},
RocksDB,
{{spark.sql.streaming.noDataMicroBatches.enabled=false}}. Built on #60669 (head
{{5fed6d21d02}}).
{code:scala}
val left = leftInput.toDF().toDF("leftKey", "time")
.select($"leftKey", timestamp_seconds($"time").as("leftTime")) // no
watermark
val right = rightInput.toDF().toDF("rightKey", "time")
.select($"rightKey", timestamp_seconds($"time").as("rightTime"))
.withWatermark("rightTime", "10 seconds")
val joined = left.join(right, expr("leftKey = rightKey AND " +
"leftTime BETWEEN rightTime - INTERVAL 5 SECONDS AND rightTime + INTERVAL 5
SECONDS"),
"left_outer")
.select($"leftKey", $"rightKey", $"leftTime".cast("int"),
$"rightTime".cast("int"))
// Run 1, default mode: leftInput (1, 5)
// Run 2, allowMultiple=false: leftInput (3, 50)
// Run 3, default mode: rightInput (9, 100), then rightInput (9, 100)
// -> the watermark passes 90 s, so neither left
row can match
{code}
- *Expected:* {{[1,null,5,null]}} and {{[3,null,50,null]}}, each once, in run 3.
- *Actual (state format 4):* {{[1,null,5,null]}} only. The row {{(3, 50)}}
stored in run 2 is
never emitted.
- State format 2 returns the expected rows.
h3. Notes
- Before SPARK-61990, every such row stayed in state and an outer join never
emitted it, so
this is not worse than master; it is a gap in the new eviction when the mode
changes.
- The restart check of the stream-stream join operator (state format 4) fails a
legacy-mode
restart of an index-version-1 checkpoint only for an input with an event time
evicted by
another column. For an input without an event time, the legacy mode computes
no column, so
the check passes. The code marks this with a TODO.
- Fix idea: make the restart check also fail a legacy-mode run of an
index-version-1 checkpoint
when an outer join emits the unmatched rows of an input without its own
watermark that the
default mode would evict, as it already does for a checkpoint with index
version 0. Or give
such rows an index entry in the legacy mode too.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]