Jungtaek Lim created SPARK-62022:
------------------------------------
Summary: NullPointerException for a NULL window watermark column
in stream-stream join state format V4
Key: SPARK-62022
URL: https://issues.apache.org/jira/browse/SPARK-62022
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 5.0.0
Reporter: Jungtaek Lim
With {{spark.sql.streaming.join.stateFormatVersion=4}}, a stream-stream join
whose input has its watermark on a window struct column, with no watermark in
the join keys, fails with a {{NullPointerException}} in
{{SymmetricHashJoinStateManagerV4.extractEventTimeFn}} when that column is NULL
({{row.getStruct(idx, 2).getLong(1)}}). State formats 2 and 3 return the joined
rows.
Reproducer (inner join on {{lk = rk}}; the left watermark is on a window struct
{{lw}} that is NULL for {{t < 0}}):
{code:scala}
val lt = timestamp_seconds($"t")
val left = leftInput.toDF().toDF("lk", "t")
.select($"lk", $"t", when($"t" >= 0,
struct(lt.as("start"), (lt + expr("INTERVAL 10
SECONDS")).as("end"))).as("lw"))
.withWatermark("lw", "10 seconds")
val right = rightInput.toDF().toDF("rk", "s")
left.join(right, expr("lk = rk")).select($"lk", $"t", $"lw".isNull, $"s")
// left rows (1, -1), (1, 5); right row (1, 7)
{code}
Expected (and what V2 and V3 return): {{(1, -1, true, 7)}} and {{(1, 5, false,
7)}}. V4: the query fails with the NPE. The output must use the whole {{lw}};
if only {{lw.end}} is selected, nested column pruning keeps just that timestamp
and the NPE does not happen.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]