[
https://issues.apache.org/jira/browse/SPARK-41146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jungtaek Lim updated SPARK-41146:
---------------------------------
Description:
We encountered the issue where the data source did not properly implement the
offset with Trigger.AvailableNow, and the query ran with processing same data
continuously without stopping.
We would like to proactively avoid such case for most used data sources.
was:
We encountered the issue where the data source did not properly implement the
offset with Trigger.AvailableNow, and the query ran with processing same data
continuously without stopping.
We would like to proactively avoid such case for most used data sources. I'll
create a new JIRA ticket for Kafka data source as well.
> Assert the offset range for Kafka source in Trigger.AvailableNow
> ----------------------------------------------------------------
>
> Key: SPARK-41146
> URL: https://issues.apache.org/jira/browse/SPARK-41146
> Project: Spark
> Issue Type: Improvement
> Components: Structured Streaming
> Affects Versions: 3.4.0
> Reporter: Jungtaek Lim
> Priority: Major
>
> We encountered the issue where the data source did not properly implement the
> offset with Trigger.AvailableNow, and the query ran with processing same data
> continuously without stopping.
> We would like to proactively avoid such case for most used data sources.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
