[ 
https://issues.apache.org/jira/browse/SPARK-62046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jungtaek Lim updated SPARK-62046:
---------------------------------
    Affects Version/s: 4.4.0

> Stream-stream outer join never outputs a row whose watermarked equality key 
> is NULL, on the input that has that watermark
> -------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62046
>                 URL: https://issues.apache.org/jira/browse/SPARK-62046
>             Project: Spark
>          Issue Type: Bug
>          Components: Structured Streaming
>    Affects Versions: 4.3.0, 5.0.0, 4.4.0
>            Reporter: Jungtaek Lim
>            Priority: Major
>
> h3. Problem
> A stream-stream outer join can evict state by an equality key that carries a 
> watermark, for
> example {{lk = rk AND lw = rw}} with the watermark on the left input's 
> {{lw}}. If a left row has
> a NULL {{lw}}, the row can never match, so a left outer or full outer join 
> should output it
> padded with NULLs once the watermark moves, and then drop it from state. 
> Instead the row stays
> in state for the life of the query and is never output.
> The late-row filter keeps such a row, because {{lw <= watermark}} is NULL for 
> it. Then:
> - State formats 2 and 3: the key eviction predicate is NULL for a NULL key, 
> so the row is never
>   evicted.
> - State format 4: a NULL key is read as event time 0. On the input whose own 
> watermark is the
>   key, eviction after the first micro-batch scans only above the previous 
> watermark, so a row
>   at time 0 that arrives after the first micro-batch is never evicted. In the 
> first micro-batch
>   the scan starts at 0, so the row is output there.
> A NULL watermarked key can reach that input through a streaming union, or 
> from the NULL-padded
> side of an upstream outer join; both keep the watermark metadata.
> h3. Reproducer
> Scala, {{MemoryStream}}, 
> {{spark.sql.streaming.noDataMicroBatches.enabled=false}}, RocksDB state
> store.
> {code:scala}
> val withTime = leftInput.toDF().toDF("lk", "t")
>   .select($"lk", timestamp_seconds($"t").as("lt"))
>   .withWatermark("lt", "10 seconds")
> val windowed = withTime.select($"lk", window($"lt", "10 seconds").as("lw"))
> val nulls = nullInput.toDF().toDF("lk")
>   .select($"lk", lit(null).cast(windowed.schema("lw").dataType).as("lw"))
> val rightTime = when($"s" >= 0, timestamp_seconds($"s"))
> val right = rightInput.toDF().toDF("rk", "s")
>   .select($"rk", when($"s" >= 0, struct(rightTime.as("start"),
>     (rightTime + expr("INTERVAL 10 SECONDS")).as("end"))).as("rw"))
> val joined = windowed.union(nulls)
>   .join(right, expr("lk = rk AND lw = rw"), "left_outer")   // or "full_outer"
>   .select($"lk", $"lw.end".cast("long"), $"rk", $"rw.end".cast("long"))
> // batch 0: leftInput (1, 5)
> // batch 1: nullInput 7
> // batch 2: leftInput (2, 100)    -> the watermark moves to 90 seconds
> // batch 3: leftInput (no rows)
> // batch 4: leftInput (3, 300)
> // batch 5: leftInput (no rows)
> {code}
> The same happens with a timestamp key: use {{$"lt".as("lw")}} instead of the 
> window, and
> {{rightTime.as("rw")}} on the right.
> - *Expected:* {{[1,10,null,null]}}, {{[7,null,null,null]}}, 
> {{[2,110,null,null]}}; after batch 5,
>   state holds only the row of {{(3, 300)}}.
> - *Actual:* {{[1,10,null,null]}}, {{[2,110,null,null]}}. The row {{(7, 
> null)}} is never output
>   and stays in state (2 state rows after batch 5).
> Results (left outer and full outer give the same rows here; full outer has no 
> right rows):
> ||Build||State format 2||State format 3||State format 4||
> |master {{4ccce2781cb}}, window key|actual (row never output)|actual|query 
> fails: NullPointerException (SPARK-61986)|
> |master {{4ccce2781cb}}, timestamp key|actual|actual|actual|
> |with SPARK-61986 (#60665), window or timestamp key|actual|actual|actual|
> If the NULL-key row arrives in the first micro-batch, state format 4 outputs 
> it in that batch,
> while state formats 2 and 3 still never output it.
> h3. Notes
> - Related: SPARK-61986 (#60665) removes the NullPointerException for a NULL 
> window key in state
>   format 4; with it, the window case behaves like the timestamp case above. 
> SPARK-61382 covers a
>   different NULL-key problem: NULL keys of both inputs matching each other in 
> a full outer join.
> - Fix idea: for state format 4, also evict the time-0 slot on each eviction 
> of the input whose
>   own watermark is the key. For state formats 2 and 3, treat a NULL 
> watermarked key as evictable
>   at the first eviction. Either way, an outer join then outputs the row once.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to