[ 
https://issues.apache.org/jira/browse/SPARK-61508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jungtaek Lim updated SPARK-61508:
---------------------------------
    Affects Version/s: 4.4.0

> Evict stream-stream join state bounded only through a column cast to TIMESTAMP
> ------------------------------------------------------------------------------
>
>                 Key: SPARK-61508
>                 URL: https://issues.apache.org/jira/browse/SPARK-61508
>             Project: Spark
>          Issue Type: Improvement
>          Components: Structured Streaming
>    Affects Versions: 5.0.0, 4.4.0
>            Reporter: Jungtaek Lim
>            Priority: Major
>
> h3. Problem
> A stream-stream join compares a TIMESTAMP column with the other input's event 
> time to decide which
> rows it can evict. When the join condition bounds a side only through a 
> column of another type
> that Spark casts to TIMESTAMP, for example {{CAST(dateCol AS TIMESTAMP)}} for 
> a DATE column, a
> TIMESTAMP_NTZ column, or a numeric column, no eviction cutoff applies to that 
> side.
> * For left, right and full outer joins, the bound is rejected at analysis by 
> SPARK-61348, which
>   asks the user to project the column to TIMESTAMP before the join.
> * For inner and left semi joins, the join runs, but it keeps the side bounded 
> only through the
>   cast for the life of the query. The state grows without bound, although the 
> results are correct.
> h3. Proposal
> Evict by {{CAST(column AS TIMESTAMP) <= cutoff}}:
> * In state formats V1 to V3, the eviction predicate becomes an expression 
> over the stored column
>   instead of a comparison with the column itself.
> * In V4, the timestamp index must hold the value of the cast, not the column. 
> That changes the V4
>   state layout, so the checkpoint compatibility rules need new cases, and the 
> checkpoint
>   watermark-semantics version has to change.
> h3. Why a separate change
> The correctness stack (SPARK-61344 to SPARK-61350) keeps its scope to wrong 
> results. Evicting
> through a cast changes the V4 layout and the restart rules, and it needs its 
> own review.
> h3. Workaround
> Project the column to TIMESTAMP before the join (for example 
> {{withColumn("t", col("d").cast("timestamp"))}})
> and use {{t}} in the join condition. The column then bounds the side's state, 
> and SPARK-61348
> gives this hint in its error for outer joins.
> Related: SPARK-61344, SPARK-61348.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to