Jungtaek Lim created SPARK-37970:
------------------------------------
Summary: Introduce a new interface on streaming data source to
notify the latest seen offset
Key: SPARK-37970
URL: https://issues.apache.org/jira/browse/SPARK-37970
Project: Spark
Issue Type: New Feature
Components: Structured Streaming
Affects Versions: 3.3.0
Reporter: Jungtaek Lim
We figure out the case of streaming data source that knowing the latest seen
offset when restarting query would be handy and useful to implement some
feature. One useful case is enabling the data source to track the offset by
itself, for the case where the external storage of data source is not exposing
any API to provide the latest available offset.
We will propose a new interface on streaming data source, which indicates Spark
to give the latest seen offset whenever the query is being restarted. For the
first start of the query, the initial offset of the data source should be
retrieved from calling initialOffset.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
