Jungtaek Lim created SPARK-62075:
------------------------------------

             Summary: Stream-stream outer join with state format 4 never emits 
rows of an input without its own watermark that a legacy-mode run stored
                 Key: SPARK-62075
                 URL: https://issues.apache.org/jira/browse/SPARK-62075
             Project: Spark
          Issue Type: Bug
          Components: Structured Streaming
    Affects Versions: 5.0.0
            Reporter: Jungtaek Lim


h3. Problem

With SPARK-61990 (#60669), a stream-stream join evicts the state of an input 
without its own
watermark when the other input's watermark and a range condition bound one of 
its TIMESTAMP
columns. With state format version 4, such an input's rows are stored with an 
index entry on
that column, and eviction finds them through that index.

In the legacy mode 
({{spark.sql.streaming.statefulOperator.allowMultiple=false}}), that input has
no eviction predicate, so rows stored during a legacy-mode run get no index 
entry. If a checkpoint
started in the default mode (V4 index version 1) is restarted in the legacy 
mode and then in the
default mode again, the restart check passes both times, and the rows stored 
during the legacy
run are never evicted afterwards: an outer join never emits them, and they stay 
in state for
good. State formats 1 to 3 evict row by row and are not affected.

h3. Reproducer

Scala, {{MemoryStream}}, {{spark.sql.streaming.join.stateFormatVersion=4}}, 
RocksDB,
{{spark.sql.streaming.noDataMicroBatches.enabled=false}}. Built on #60669 (head 
{{5fed6d21d02}}).

{code:scala}
val left = leftInput.toDF().toDF("leftKey", "time")
  .select($"leftKey", timestamp_seconds($"time").as("leftTime"))          // no 
watermark
val right = rightInput.toDF().toDF("rightKey", "time")
  .select($"rightKey", timestamp_seconds($"time").as("rightTime"))
  .withWatermark("rightTime", "10 seconds")
val joined = left.join(right, expr("leftKey = rightKey AND " +
    "leftTime BETWEEN rightTime - INTERVAL 5 SECONDS AND rightTime + INTERVAL 5 
SECONDS"),
    "left_outer")
  .select($"leftKey", $"rightKey", $"leftTime".cast("int"), 
$"rightTime".cast("int"))

// Run 1, default mode:        leftInput (1, 5)
// Run 2, allowMultiple=false: leftInput (3, 50)
// Run 3, default mode:        rightInput (9, 100), then rightInput (9, 100)
//                             -> the watermark passes 90 s, so neither left 
row can match
{code}

- *Expected:* {{[1,null,5,null]}} and {{[3,null,50,null]}}, each once, in run 3.
- *Actual (state format 4):* {{[1,null,5,null]}} only. The row {{(3, 50)}} 
stored in run 2 is
  never emitted.
- State format 2 returns the expected rows.


h3. Notes

- Before SPARK-61990, every such row stayed in state and an outer join never 
emitted it, so
  this is not worse than master; it is a gap in the new eviction when the mode 
changes.
- The restart check of the stream-stream join operator (state format 4) fails a 
legacy-mode
  restart of an index-version-1 checkpoint only for an input with an event time 
evicted by
  another column. For an input without an event time, the legacy mode computes 
no column, so
  the check passes. The code marks this with a TODO.
- Fix idea: make the restart check also fail a legacy-mode run of an 
index-version-1 checkpoint
  when an outer join emits the unmatched rows of an input without its own 
watermark that the
  default mode would evict, as it already does for a checkpoint with index 
version 0. Or give
  such rows an index entry in the legacy mode too.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to