Jungtaek Lim created SPARK-61345:
------------------------------------

             Summary: Fix stream-stream join watermark direction and 
equality-key eviction
                 Key: SPARK-61345
                 URL: https://issues.apache.org/jira/browse/SPARK-61345
             Project: Spark
          Issue Type: Bug
          Components: Structured Streaming
    Affects Versions: 5.0.0
            Reporter: Jungtaek Lim


Stream-stream joins check the wrong watermark direction and evict equality-key 
state with the wrong side's watermark.

1. Admission. A left outer or left semi join finalizes left rows, so it needs a 
watermark on the right input and a condition that limits how late a matching 
right row can arrive. The analyzer accepts such joins with a watermark on the 
left input only. They keep unmatched state indefinitely and never emit outer 
rows. Full outer joins need a bound in both directions.

2. Equality-key eviction. The key-eviction predicate and the V4 timestamp index 
use one equality-key ordinal for both sides. When the event-time keys sit at 
different ordinals, Spark can remove a row whose match is still to come and 
emit a premature outer null row. A key that is the opposite event time shifted 
by a constant interval, such as {{lt = rt + INTERVAL 10 SECONDS}}, is also not 
handled. A truncating key such as {{CAST(CAST(rt AS BIGINT) AS TIMESTAMP)}} 
must not get a bound, because a later row can truncate below the watermark.

3. Null keys in V4. Reading a null selected window as a struct throws a 
NullPointerException. Indexing a null key under another timestamp can remove 
the row or emit an outer row too early.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to