[
https://issues.apache.org/jira/browse/SPARK-61508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated SPARK-61508:
---------------------------------
Affects Version/s: 4.4.0
> Evict stream-stream join state bounded only through a column cast to TIMESTAMP
> ------------------------------------------------------------------------------
>
> Key: SPARK-61508
> URL: https://issues.apache.org/jira/browse/SPARK-61508
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 5.0.0, 4.4.0
> Reporter: Jungtaek Lim
> Priority: Major
>
> h3. Problem
> A stream-stream join compares a TIMESTAMP column with the other input's event
> time to decide which
> rows it can evict. When the join condition bounds a side only through a
> column of another type
> that Spark casts to TIMESTAMP, for example {{CAST(dateCol AS TIMESTAMP)}} for
> a DATE column, a
> TIMESTAMP_NTZ column, or a numeric column, no eviction cutoff applies to that
> side.
> * For left, right and full outer joins, the bound is rejected at analysis by
> SPARK-61348, which
> asks the user to project the column to TIMESTAMP before the join.
> * For inner and left semi joins, the join runs, but it keeps the side bounded
> only through the
> cast for the life of the query. The state grows without bound, although the
> results are correct.
> h3. Proposal
> Evict by {{CAST(column AS TIMESTAMP) <= cutoff}}:
> * In state formats V1 to V3, the eviction predicate becomes an expression
> over the stored column
> instead of a comparison with the column itself.
> * In V4, the timestamp index must hold the value of the cast, not the column.
> That changes the V4
> state layout, so the checkpoint compatibility rules need new cases, and the
> checkpoint
> watermark-semantics version has to change.
> h3. Why a separate change
> The correctness stack (SPARK-61344 to SPARK-61350) keeps its scope to wrong
> results. Evicting
> through a cast changes the V4 layout and the restart rules, and it needs its
> own review.
> h3. Workaround
> Project the column to TIMESTAMP before the join (for example
> {{withColumn("t", col("d").cast("timestamp"))}})
> and use {{t}} in the join condition. The column then bounds the side's state,
> and SPARK-61348
> gives this hint in its error for outer joins.
> Related: SPARK-61344, SPARK-61348.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]