Jungtaek Lim created SPARK-62002:
------------------------------------

             Summary: Stream-stream join loses matches when an equality key on 
event time is shifted backward or truncated on one side
                 Key: SPARK-62002
                 URL: https://issues.apache.org/jira/browse/SPARK-62002
             Project: Spark
          Issue Type: Bug
          Components: Structured Streaming
    Affects Versions: 5.0.0
            Reporter: Jungtaek Lim


When a join key on one side carries a watermark, Spark evicts both inputs' 
state when that key position falls to the watermark or below. If the other 
input's key is its event time shifted backward ({{lt = rt - INTERVAL 10 
SECONDS}}) or truncated ({{lt = CAST(CAST(rt AS DATE) AS TIMESTAMP)}}, or 
{{date_trunc}}), a row of that input can be on time but still have a key below 
the watermark. Its match on the first input has already been evicted. Inner 
joins lose the match. Outer joins emit an early null row and then lose the 
match.

Reproducer (10-second watermarks on {{lt}} and {{rt}}):
{code:scala}
left.join(right, expr("lk = rk AND lt = rt - INTERVAL 10 SECONDS"))
{code}
Left (1, 95 s). Rows at 110 s move the watermark to 100 s, and Spark evicts the 
left row. Right (1, 105 s) is on time and should match it (95 = 105 - 10), but 
Spark emits nothing. A left outer join emits (1, 95, null) instead.

With {{lt = CAST(CAST(rt AS DATE) AS TIMESTAMP)}}, left (1, 1970-01-02 00:00) 
is evicted at watermark 00:00:05, and right (1, 02:00 the same day) finds no 
match.

Affects state format versions 2 to 4. Related to SPARK-61381 (inexact bounds) 
and SPARK-61989 (equality-key eviction).




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to