Jungtaek Lim created SPARK-61508:
------------------------------------
Summary: Evict stream-stream join state bounded only through a
column cast to TIMESTAMP
Key: SPARK-61508
URL: https://issues.apache.org/jira/browse/SPARK-61508
Project: Spark
Issue Type: Improvement
Components: Structured Streaming
Affects Versions: 5.0.0
Reporter: Jungtaek Lim
h3. Problem
A stream-stream join compares a TIMESTAMP column with the other input's event
time to decide which
rows it can evict. When the join condition bounds a side only through a column
of another type
that Spark casts to TIMESTAMP, for example {{CAST(dateCol AS TIMESTAMP)}} for a
DATE column, a
TIMESTAMP_NTZ column, or a numeric column, no eviction cutoff applies to that
side.
* For left, right and full outer joins, the bound is rejected at analysis by
SPARK-61348, which
asks the user to project the column to TIMESTAMP before the join.
* For inner and left semi joins, the join runs, but it keeps the side bounded
only through the
cast for the life of the query. The state grows without bound, although the
results are correct.
h3. Proposal
Evict by {{CAST(column AS TIMESTAMP) <= cutoff}}:
* In state formats V1 to V3, the eviction predicate becomes an expression over
the stored column
instead of a comparison with the column itself.
* In V4, the timestamp index must hold the value of the cast, not the column.
That changes the V4
state layout, so the checkpoint compatibility rules need new cases, and the
checkpoint
watermark-semantics version has to change.
h3. Why a separate change
The correctness stack (SPARK-61344 to SPARK-61350) keeps its scope to wrong
results. Evicting
through a cast changes the V4 layout and the restart rules, and it needs its
own review.
h3. Workaround
Project the column to TIMESTAMP before the join (for example {{withColumn("t",
col("d").cast("timestamp"))}})
and use {{t}} in the join condition. The column then bounds the side's state,
and SPARK-61348
gives this hint in its error for outer joins.
Related: SPARK-61344, SPARK-61348.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]