[
https://issues.apache.org/jira/browse/SPARK-61345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61345:
-----------------------------------
Labels: pull-request-available (was: )
> Fix stream-stream join watermark direction and equality-key eviction
> --------------------------------------------------------------------
>
> Key: SPARK-61345
> URL: https://issues.apache.org/jira/browse/SPARK-61345
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 5.0.0
> Reporter: Jungtaek Lim
> Priority: Major
> Labels: pull-request-available
>
> Stream-stream joins check the wrong watermark direction and evict
> equality-key state with the wrong side's watermark.
> 1. Admission. A left outer or left semi join finalizes left rows, so it needs
> a watermark on the right input and a condition that limits how late a
> matching right row can arrive. The analyzer accepts such joins with a
> watermark on the left input only. They keep unmatched state indefinitely and
> never emit outer rows. Full outer joins need a bound in both directions.
> 2. Equality-key eviction. The key-eviction predicate and the V4 timestamp
> index use one equality-key ordinal for both sides. When the event-time keys
> sit at different ordinals, Spark can remove a row whose match is still to
> come and emit a premature outer null row. A key that is the opposite event
> time shifted by a constant interval, such as {{lt = rt + INTERVAL 10
> SECONDS}}, is also not handled. A truncating key such as {{CAST(CAST(rt AS
> BIGINT) AS TIMESTAMP)}} must not get a bound, because a later row can
> truncate below the watermark.
> 3. Null keys in V4. Reading a null selected window as a struct throws a
> NullPointerException. Indexing a null key under another timestamp can remove
> the row or emit an outer row too early.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]