[
https://issues.apache.org/jira/browse/SPARK-61347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61347:
-----------------------------------
Labels: pull-request-available (was: )
> Evict stream-stream join range state without a watermark on the stored side
> ---------------------------------------------------------------------------
>
> Key: SPARK-61347
> URL: https://issues.apache.org/jira/browse/SPARK-61347
> Project: Spark
> Issue Type: Bug
> Components: Structured Streaming
> Affects Versions: 5.0.0
> Reporter: Jungtaek Lim
> Priority: Major
> Labels: pull-request-available
>
> A stream-stream join only infers a range-eviction bound for one side when
> that side has its own watermark.
> Example: a left outer join of impressions and clicks with a watermark on
> {{clickTime}} only and the condition {{clickTime BETWEEN impressionTime AND
> impressionTime + INTERVAL 1 HOUR}}. The right watermark and the time range
> are enough to know when a left row can no longer match. Because the left side
> has no watermark, Spark never removes left rows and never emits their
> unmatched outer rows. The right outer case is symmetric.
> The same happens when the stored side's event-time column has been pruned
> from the physical join because no downstream operator uses it.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]