Jungtaek Lim created SPARK-62002:
------------------------------------
Summary: Stream-stream join loses matches when an equality key on
event time is shifted backward or truncated on one side
Key: SPARK-62002
URL: https://issues.apache.org/jira/browse/SPARK-62002
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 5.0.0
Reporter: Jungtaek Lim
When a join key on one side carries a watermark, Spark evicts both inputs'
state when that key position falls to the watermark or below. If the other
input's key is its event time shifted backward ({{lt = rt - INTERVAL 10
SECONDS}}) or truncated ({{lt = CAST(CAST(rt AS DATE) AS TIMESTAMP)}}, or
{{date_trunc}}), a row of that input can be on time but still have a key below
the watermark. Its match on the first input has already been evicted. Inner
joins lose the match. Outer joins emit an early null row and then lose the
match.
Reproducer (10-second watermarks on {{lt}} and {{rt}}):
{code:scala}
left.join(right, expr("lk = rk AND lt = rt - INTERVAL 10 SECONDS"))
{code}
Left (1, 95 s). Rows at 110 s move the watermark to 100 s, and Spark evicts the
left row. Right (1, 105 s) is on time and should match it (95 = 105 - 10), but
Spark emits nothing. A left outer join emits (1, 95, null) instead.
With {{lt = CAST(CAST(rt AS DATE) AS TIMESTAMP)}}, left (1, 1970-01-02 00:00)
is evicted at watermark 00:00:05, and right (1, 02:00 the same day) finds no
match.
Affects state format versions 2 to 4. Related to SPARK-61381 (inexact bounds)
and SPARK-61989 (equality-key eviction).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]