Jungtaek Lim created SPARK-61347:
------------------------------------
Summary: Evict stream-stream join range state without a watermark
on the stored side
Key: SPARK-61347
URL: https://issues.apache.org/jira/browse/SPARK-61347
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 5.0.0
Reporter: Jungtaek Lim
A stream-stream join only infers a range-eviction bound for one side when that
side has its own watermark.
Example: a left outer join of impressions and clicks with a watermark on
{{clickTime}} only and the condition {{clickTime BETWEEN impressionTime AND
impressionTime + INTERVAL 1 HOUR}}. The right watermark and the time range are
enough to know when a left row can no longer match. Because the left side has
no watermark, Spark never removes left rows and never emits their unmatched
outer rows. The right outer case is symmetric.
The same happens when the stored side's event-time column has been pruned from
the physical join because no downstream operator uses it.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]