[ 
https://issues.apache.org/jira/browse/SPARK-62022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jungtaek Lim updated SPARK-62022:
---------------------------------
    Affects Version/s: 4.3.0
                       4.4.0

> NullPointerException for a NULL window watermark column in stream-stream join 
> state format V4
> ---------------------------------------------------------------------------------------------
>
>                 Key: SPARK-62022
>                 URL: https://issues.apache.org/jira/browse/SPARK-62022
>             Project: Spark
>          Issue Type: Bug
>          Components: Structured Streaming
>    Affects Versions: 4.3.0, 5.0.0, 4.4.0
>            Reporter: Jungtaek Lim
>            Priority: Major
>
> With {{spark.sql.streaming.join.stateFormatVersion=4}}, a stream-stream join 
> whose input has its watermark on a window struct column, with no watermark in 
> the join keys, fails with a {{NullPointerException}} in 
> {{SymmetricHashJoinStateManagerV4.extractEventTimeFn}} when that column is 
> NULL ({{row.getStruct(idx, 2).getLong(1)}}). State formats 2 and 3 return the 
> joined rows.
> Reproducer (inner join on {{lk = rk}}; the left watermark is on a window 
> struct {{lw}} that is NULL for {{t < 0}}):
> {code:scala}
> val lt = timestamp_seconds($"t")
> val left = leftInput.toDF().toDF("lk", "t")
>   .select($"lk", $"t", when($"t" >= 0,
>     struct(lt.as("start"), (lt + expr("INTERVAL 10 
> SECONDS")).as("end"))).as("lw"))
>   .withWatermark("lw", "10 seconds")
> val right = rightInput.toDF().toDF("rk", "s")
> left.join(right, expr("lk = rk")).select($"lk", $"t", $"lw".isNull, $"s")
> // left rows (1, -1), (1, 5); right row (1, 7)
> {code}
> Expected (and what V2 and V3 return): {{(1, -1, true, 7)}} and {{(1, 5, 
> false, 7)}}. V4: the query fails with the NPE. The output must use the whole 
> {{lw}}; if only {{lw.end}} is selected, nested column pruning keeps just that 
> timestamp and the NPE does not happen.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to