Gengliang Wang created SPARK-61820:
--------------------------------------

             Summary: Clarify that native data source offsets are progress 
boundaries, and that adjacent ranges must not repeat or skip records
                 Key: SPARK-61820
                 URL: https://issues.apache.org/jira/browse/SPARK-61820
             Project: Spark
          Issue Type: Documentation
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Gengliang Wang


Version 1 of the native data source ABI ({{spark_datasource.h}}) describes the 
offsets of a streaming read in terms of the offset that a micro-batch starts 
after, and a micro-batch as the data after its start up to and including its 
end. This describes them as progress boundaries instead: the library defines 
how offsets map to records, equal boundaries describe no data, and adjacent 
ranges must neither repeat nor skip records. Committing an offset acknowledges 
all the data before its progress boundary.

The rule on adjacent ranges is a requirement on libraries that exactly-once 
reads imply but that the header did not state. It applies to every 
implementation of the ABI, such as the Rust SDK (SPARK-61282) and the C++ SDK 
(https://github.com/apache/spark/pull/59726).

The change updates the comments of {{spark_datasource.h}}, the Native Data 
Sources guide ({{docs/sql-native-data-sources.md}}), and the doc comments of 
{{NativeLibrary.latestOffset}}, {{ColumnarStreamReader}} and 
{{ColumnarMicroBatchStream}}. It does not change the behavior of Spark.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to