Jungtaek Lim created SPARK-61514:
------------------------------------
Summary: Derive V4 stream-stream join scan windows from the
incoming row
Key: SPARK-61514
URL: https://issues.apache.org/jira/browse/SPARK-61514
Project: Spark
Issue Type: Improvement
Components: Structured Streaming
Affects Versions: 5.0.0
Reporter: Jungtaek Lim
In state format version 4, a stream-stream join stores each side's rows ordered
by a timestamp. A new row reads the other side's rows for its key. When the
join condition bounds the two event times by constant offsets, it reads only
the rows in a time window. In every other case it reads all rows for the key.
A common interval join bounds the time by another column, as in Nexmark q9:
{{bid.dateTime BETWEEN auction.dateTime AND auction.expires}}. The auction side
then evicts on {{expires}}, so its state is ordered by {{expires}}, not by its
event time. Spark turns off the window for both sides, and every row reads all
rows for its key. Such queries can be slower in V4 than in V2.
Proposal: each side bounds the column that the other side's state is ordered
by. It uses the comparisons of the join condition on that column, evaluated on
the new row.
* An auction row reads bids with {{dateTime}} between its {{dateTime}} and its
{{expires}}.
* A bid row reads auctions with {{expires}} at or after its {{dateTime}}. This
needs a range scan with no upper end.
A bound must follow from the condition, so it never skips a row that the
condition matches. Shifts by whole days or months, casts other than to BIGINT,
and other functions of the stored column give no bound. The constant-offset
window also missed matches for whole-day intervals across a daylight saving
time change, because it counts a day as 24 hours; the new windows fix that too.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]