[
https://issues.apache.org/jira/browse/SPARK-62022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-62022:
-----------------------------------
Labels: pull-request-available (was: )
> NullPointerException for a NULL window watermark column in stream-stream join
> state format V4
> ---------------------------------------------------------------------------------------------
>
> Key: SPARK-62022
> URL: https://issues.apache.org/jira/browse/SPARK-62022
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 4.3.0, 5.0.0, 4.4.0
> Reporter: Jungtaek Lim
> Priority: Major
> Labels: pull-request-available
>
> With {{spark.sql.streaming.join.stateFormatVersion=4}}, a stream-stream join
> whose input has its watermark on a window struct column, with no watermark in
> the join keys, fails with a {{NullPointerException}} in
> {{SymmetricHashJoinStateManagerV4.extractEventTimeFn}} when that column is
> NULL ({{row.getStruct(idx, 2).getLong(1)}}). State formats 2 and 3 return the
> joined rows.
> Reproducer (inner join on {{lk = rk}}; the left watermark is on a window
> struct {{lw}} that is NULL for {{t < 0}}):
> {code:scala}
> val lt = timestamp_seconds($"t")
> val left = leftInput.toDF().toDF("lk", "t")
> .select($"lk", $"t", when($"t" >= 0,
> struct(lt.as("start"), (lt + expr("INTERVAL 10
> SECONDS")).as("end"))).as("lw"))
> .withWatermark("lw", "10 seconds")
> val right = rightInput.toDF().toDF("rk", "s")
> left.join(right, expr("lk = rk")).select($"lk", $"t", $"lw".isNull, $"s")
> // left rows (1, -1), (1, 5); right row (1, 7)
> {code}
> Expected (and what V2 and V3 return): {{(1, -1, true, 7)}} and {{(1, 5,
> false, 7)}}. V4: the query fails with the NPE. The output must use the whole
> {{lw}}; if only {{lw.end}} is selected, nested column pruning keeps just that
> timestamp and the NPE does not happen.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]