[
https://issues.apache.org/jira/browse/SPARK-61820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gengliang Wang reassigned SPARK-61820:
--------------------------------------
Assignee: Gengliang Wang
> Clarify that native data source offsets are progress boundaries, and that
> adjacent ranges must not repeat or skip records
> -------------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-61820
> URL: https://issues.apache.org/jira/browse/SPARK-61820
> Project: Spark
> Issue Type: Documentation
> Components: SQL
> Affects Versions: 5.0.0
> Reporter: Gengliang Wang
> Assignee: Gengliang Wang
> Priority: Major
>
> Version 1 of the native data source ABI ({{spark_datasource.h}}) describes
> the offsets of a streaming read in terms of the offset that a micro-batch
> starts after, and a micro-batch as the data after its start up to and
> including its end. This describes them as progress boundaries instead: the
> library defines how offsets map to records, equal boundaries describe no
> data, and adjacent ranges must neither repeat nor skip records. Committing an
> offset acknowledges all the data before its progress boundary.
> The rule on adjacent ranges is a requirement on libraries that exactly-once
> reads imply but that the header did not state. It applies to every
> implementation of the ABI, such as the Rust SDK (SPARK-61282) and the C++ SDK
> (https://github.com/apache/spark/pull/59726).
> The change updates the comments of {{spark_datasource.h}}, the Native Data
> Sources guide ({{docs/sql-native-data-sources.md}}), and the doc comments of
> {{NativeLibrary.latestOffset}}, {{ColumnarStreamReader}} and
> {{ColumnarMicroBatchStream}}. It does not change the behavior of Spark.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]