Jungtaek Lim created SPARK-61508:
------------------------------------

             Summary: Evict stream-stream join state bounded only through a 
column cast to TIMESTAMP
                 Key: SPARK-61508
                 URL: https://issues.apache.org/jira/browse/SPARK-61508
             Project: Spark
          Issue Type: Improvement
          Components: Structured Streaming
    Affects Versions: 5.0.0
            Reporter: Jungtaek Lim


h3. Problem

A stream-stream join compares a TIMESTAMP column with the other input's event 
time to decide which
rows it can evict. When the join condition bounds a side only through a column 
of another type
that Spark casts to TIMESTAMP, for example {{CAST(dateCol AS TIMESTAMP)}} for a 
DATE column, a
TIMESTAMP_NTZ column, or a numeric column, no eviction cutoff applies to that 
side.

* For left, right and full outer joins, the bound is rejected at analysis by 
SPARK-61348, which
  asks the user to project the column to TIMESTAMP before the join.
* For inner and left semi joins, the join runs, but it keeps the side bounded 
only through the
  cast for the life of the query. The state grows without bound, although the 
results are correct.

h3. Proposal

Evict by {{CAST(column AS TIMESTAMP) <= cutoff}}:

* In state formats V1 to V3, the eviction predicate becomes an expression over 
the stored column
  instead of a comparison with the column itself.
* In V4, the timestamp index must hold the value of the cast, not the column. 
That changes the V4
  state layout, so the checkpoint compatibility rules need new cases, and the 
checkpoint
  watermark-semantics version has to change.

h3. Why a separate change

The correctness stack (SPARK-61344 to SPARK-61350) keeps its scope to wrong 
results. Evicting
through a cast changes the V4 layout and the restart rules, and it needs its 
own review.

h3. Workaround

Project the column to TIMESTAMP before the join (for example {{withColumn("t", 
col("d").cast("timestamp"))}})
and use {{t}} in the join condition. The column then bounds the side's state, 
and SPARK-61348
gives this hint in its error for outer joins.

Related: SPARK-61344, SPARK-61348.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to