Jungtaek Lim created SPARK-62022:
------------------------------------

             Summary: NullPointerException for a NULL window watermark column 
in stream-stream join state format V4
                 Key: SPARK-62022
                 URL: https://issues.apache.org/jira/browse/SPARK-62022
             Project: Spark
          Issue Type: Bug
          Components: Structured Streaming
    Affects Versions: 5.0.0
            Reporter: Jungtaek Lim


With {{spark.sql.streaming.join.stateFormatVersion=4}}, a stream-stream join 
whose input has its watermark on a window struct column, with no watermark in 
the join keys, fails with a {{NullPointerException}} in 
{{SymmetricHashJoinStateManagerV4.extractEventTimeFn}} when that column is NULL 
({{row.getStruct(idx, 2).getLong(1)}}). State formats 2 and 3 return the joined 
rows.

Reproducer (inner join on {{lk = rk}}; the left watermark is on a window struct 
{{lw}} that is NULL for {{t < 0}}):

{code:scala}
val lt = timestamp_seconds($"t")
val left = leftInput.toDF().toDF("lk", "t")
  .select($"lk", $"t", when($"t" >= 0,
    struct(lt.as("start"), (lt + expr("INTERVAL 10 
SECONDS")).as("end"))).as("lw"))
  .withWatermark("lw", "10 seconds")
val right = rightInput.toDF().toDF("rk", "s")
left.join(right, expr("lk = rk")).select($"lk", $"t", $"lw".isNull, $"s")
// left rows (1, -1), (1, 5); right row (1, 7)
{code}

Expected (and what V2 and V3 return): {{(1, -1, true, 7)}} and {{(1, 5, false, 
7)}}. V4: the query fails with the NPE. The output must use the whole {{lw}}; 
if only {{lw.end}} is selected, nested column pruning keeps just that timestamp 
and the NPE does not happen.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to