[
https://issues.apache.org/jira/browse/SPARK-61382?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18146977#comment-18146977
]
Jungtaek Lim commented on SPARK-61382:
--------------------------------------
The same NULL = NULL match also happens when the NULL is in the watermarked
equality key
itself, and in every state format. Note that SPARK-61344 and SPARK-61345 were
closed as Won't
Do, so the "What the open fixes already change" part of the description does
not apply.
Reproducer: a full outer join on {{lk = rk AND lw = rw}}, the left watermark on
{{lt}} with
{{lw = window(lt, "10 seconds")}} (or {{lw = lt}}), the right input without a
watermark and
{{rw}} NULL for a negative input. A NULL {{lw}} reaches the left input through
a streaming
union:
{code:scala}
val withTime = leftInput.toDF().toDF("lk", "t")
.select($"lk", timestamp_seconds($"t").as("lt"))
.withWatermark("lt", "10 seconds")
val windowed = withTime.select($"lk", window($"lt", "10 seconds").as("lw"))
val nulls = nullInput.toDF().toDF("lk")
.select($"lk", lit(null).cast(windowed.schema("lw").dataType).as("lw"))
val rightTime = when($"s" >= 0, timestamp_seconds($"s"))
val right = rightInput.toDF().toDF("rk", "s")
.select($"rk", when($"s" >= 0, struct(rightTime.as("start"),
(rightTime + expr("INTERVAL 10 SECONDS")).as("end"))).as("rw"))
val joined = windowed.union(nulls)
.join(right, expr("lk = rk AND lw = rw"), "full_outer")
.select($"lk", $"lw.end".cast("long"), $"rk", $"rw.end".cast("long"))
// batch 0: leftInput (1, 5)
// batch 1: nullInput 7 and rightInput (7, -1), in one micro-batch
// batch 2: leftInput (2, 100) -> the watermark moves to 90 seconds
// batch 3: leftInput (no rows)
{code}
- *Expected:* {{[1,10,null,null]}}, {{[7,null,null,null]}},
{{[null,null,7,null]}}.
- *Actual:* {{[1,10,null,null]}}, {{[7,null,7,null]}}.
||Build||State format 2||State format 3||State format 4||
|master {{4ccce2781cb}}, window key|actual|actual|query fails:
NullPointerException (SPARK-61986)|
|master {{4ccce2781cb}}, timestamp key|actual|actual|actual|
|with SPARK-61986 (#60665), window or timestamp key|actual|actual|actual|
The same rows come out when both NULL-key rows arrive in the first micro-batch.
A left outer join
is not affected: the inferred {{IS NOT NULL}} filter removes the right row.
> Stream-stream full outer join joins rows whose equality keys are both NULL
> --------------------------------------------------------------------------
>
> Key: SPARK-61382
> URL: https://issues.apache.org/jira/browse/SPARK-61382
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 3.5.9, 4.3.0, 5.0.0, 4.2.1, 4.0.5, 4.1.4, 4.4.0
> Reporter: Jungtaek Lim
> Priority: Major
>
> A stream-stream full outer join can join a left row and a right row whose
> equality key is NULL on both sides. SQL {{=}} never matches NULL, so the
> correct output is two rows, each padded with nulls. A batch full outer join
> on the same data returns those two rows. The streaming join returns one
> joined row instead.
> The streaming join finds matches by looking up the other input's state with
> the generated key row, and that lookup compares the key bytes. Two keys that
> are NULL in the same field have the same bytes, so they match. The batch
> joins skip a key with a NULL field, but the stream-stream join has no such
> check.
> h3. Which join types are affected
> For most join types, the optimizer keeps NULL keys from meeting. When
> {{spark.sql.constraintPropagation.enabled}} is true (the default),
> {{InferFiltersFromConstraints}} adds {{IS NOT NULL}} filters on the equality
> keys:
> - on both inputs of an inner or left semi join;
> - on the right input of a left outer join;
> - on the left input of a right outer join.
> A full outer join gets no such filter, so:
> - *With the default setting, only full outer joins are affected.* In a left
> outer join, a left row with a NULL key cannot meet a right row with a NULL
> key, because the filter removes that right row. The left row is later emitted
> padded with nulls, which is correct. Right outer joins behave the same way,
> mirrored.
> - *With {{spark.sql.constraintPropagation.enabled=false}}, inner, left outer
> and right outer joins are affected too.* No filter is added, so NULL keys
> meet in all of them. The reproducer below shows this for inner, left outer
> and right outer joins; left semi joins were not run.
> Null-safe equality ({{<=>}}) is not affected. The planner rewrites it into
> keys that are never NULL.
> h3. Reproducer
> - Setup: Scala with {{MemoryStream}}, a 0-second watermark on both inputs,
> and the default settings.
> - A negative key stands for a NULL key.
> {code:scala}
> val leftInput = MemoryStream[(Int, Int)]
> val rightInput = MemoryStream[(Int, Int)]
> val left = leftInput.toDF().toDF("k", "t")
> .select(when($"k" >= 0, $"k").as("lk"), timestamp_seconds($"t").as("lt"))
> .withWatermark("lt", "0 seconds")
> val right = rightInput.toDF().toDF("k", "t")
> .select(when($"k" >= 0, $"k").as("rk"), timestamp_seconds($"t").as("rt"))
> .withWatermark("rt", "0 seconds")
> val joined = left.join(right, expr("lk = rk AND lt = rt"), "full_outer")
> .select($"lk", $"lt".cast("long"), $"rk", $"rt".cast("long"))
> // batch 1: left (NULL, 10); right (NULL, 10)
> // batch 2: left (1, 100); right (1, 100) -> the watermark passes 10
> // batch 3: left (2, 200); right (2, 200)
> {code}
> - *Expected:* {{[1,100,1,100]}}, {{[2,200,2,200]}}, {{[null,10,null,null]}},
> {{[null,null,null,10]}}.
> - *Actual:* {{[1,100,1,100]}}, {{[2,200,2,200]}}, {{[null,10,null,10]}}.
> The same reproducer was run for each join type, with constraint propagation
> on and off, in state formats 2 and 4. It was run on master at f90fdcb2802,
> and on the stream-stream join correctness fixes listed below. Both gave the
> same results:
> ||Join type||Constraint propagation on||Constraint propagation off||
> |inner|correct: no NULL-key row|wrong: {{[null,10,null,10]}}|
> |left outer|correct: {{[null,10,null,null]}}|wrong: {{[null,10,null,10]}}|
> |right outer|correct: {{[null,null,null,10]}}|wrong: {{[null,10,null,10]}}|
> |full outer|*wrong:* {{[null,10,null,10]}}|wrong: {{[null,10,null,10]}}|
> h3. What the open fixes already change
> The open stream-stream join correctness changes SPARK-61344 and SPARK-61345
> (#60279, #60280) handle one case. If the equality key that state is evicted
> on is NULL, the row can never match, so it is handled like a row that fails
> the pre-join filter:
> - a preserved side emits it padded with nulls in the batch it arrives;
> - no side stores it.
> So two rows whose NULL is in that eviction key no longer pair. This ticket
> covers what remains: a NULL in any other equality key. In the reproducer,
> {{lk}} is NULL but the state is evicted on {{lt}} and {{rt}}, so the rows
> still pair with those changes applied.
> h3. Possible direction
> Handle a row whose generated equality key has a NULL field
> ({{UnsafeRow.anyNull}}) like a row that fails the pre-join filter, in every
> join type:
> - a preserved side emits it padded with nulls at once;
> - no side probes the other input's state with it or stores it.
> This makes the streaming join match batch semantics. It also removes NULL-key
> rows from state that would otherwise stay there until eviction.
> Output changes only for rows that are now joined although {{=}} never matches
> them.
> Rows with a NULL key that are already in a checkpoint stay in state after the
> upgrade and are evicted as today. They no longer pair with new rows, because
> a new NULL-key row does not probe state.
> h3. How the affected versions were checked
> I read the join code with {{git show <ref>:<path>}}, read-only, at these refs:
> ||Ref||Version||Commit||
> |v3.5.9|3.5.9|7c14a3c28b1|
> |v4.0.5|4.0.5|b472a3aa3c4|
> |v4.1.4|4.1.4|45b9b176e6e|
> |v4.2.1|4.2.1|61761839608|
> |v4.3.0|4.3.0|98bc0212c94|
> |branch-4.x|4.4.0-SNAPSHOT|106f81f8891|
> |master|5.0.0-SNAPSHOT|f35395086c9|
> At every ref:
> - *No NULL check in the join.*
> {{StreamingSymmetricHashJoinExec.OneSideHashJoiner.storeAndJoinWithOtherSide}}
> builds the key with {{keyGenerator(thisRow)}} and uses it to probe the other
> input's state and to store the row. Neither the exec nor
> {{SymmetricHashJoinStateManager}} checks the key for NULL; neither has
> {{anyNull}} or a key {{isNullAt}} check. The files are under
> {{sql/execution/streaming/}} in 3.5.9 and 4.0.5, and under
> {{sql/execution/streaming/operators/stateful/join/}} from 4.1.4 on.
> - *Inferred filters.* In
> {{sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/Optimizer.scala}},
> {{InferFiltersFromConstraints}} runs only when
> {{conf.constraintPropagationEnabled}} is true. It has cases for {{InnerLike |
> LeftSemi}} (both sides), {{RightOuter}} (left side) and {{LeftOuter |
> LeftAnti}} (right side), and none for {{FullOuter}}.
> So all seven refs behave the same way. The reproducer ran only on master and
> on the open fixes, not on the release tags.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]