[ 
https://issues.apache.org/jira/browse/SPARK-61820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gengliang Wang resolved SPARK-61820.
------------------------------------
    Resolution: Fixed

Resolved in https://github.com/apache/spark/pull/59520

> Clarify that native data source offsets are progress boundaries, and that 
> adjacent ranges must not repeat or skip records
> -------------------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-61820
>                 URL: https://issues.apache.org/jira/browse/SPARK-61820
>             Project: Spark
>          Issue Type: Documentation
>          Components: SQL
>    Affects Versions: 5.0.0
>            Reporter: Gengliang Wang
>            Assignee: Gengliang Wang
>            Priority: Major
>
> Version 1 of the native data source ABI ({{spark_datasource.h}}) describes 
> the offsets of a streaming read in terms of the offset that a micro-batch 
> starts after, and a micro-batch as the data after its start up to and 
> including its end. This describes them as progress boundaries instead: the 
> library defines how offsets map to records, equal boundaries describe no 
> data, and adjacent ranges must neither repeat nor skip records. Committing an 
> offset acknowledges all the data before its progress boundary.
> The rule on adjacent ranges is a requirement on libraries that exactly-once 
> reads imply but that the header did not state. It applies to every 
> implementation of the ABI, such as the Rust SDK (SPARK-61282) and the C++ SDK 
> (https://github.com/apache/spark/pull/59726).
> The change updates the comments of {{spark_datasource.h}}, the Native Data 
> Sources guide ({{docs/sql-native-data-sources.md}}), and the doc comments of 
> {{NativeLibrary.latestOffset}}, {{ColumnarStreamReader}} and 
> {{ColumnarMicroBatchStream}}. It does not change the behavior of Spark.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to