XiaoHongbo-Hope opened a new pull request, #11021:
URL: https://github.com/apache/paimon/pull/11021

   Spark `foreachBatch` consumers cannot obtain the Paimon source offset range 
for a batch through a supported connector API. Applications that need a stable 
source-batch identity currently inspect Spark RDD internals, and an empty full 
snapshot has no input partitions to inspect at all.
   
   This adds `PaimonSparkMicroBatchMetadata.offsetRange(batch, batchId)`. For 
batches with input partitions, it returns the exact planned start and end 
offsets already carried by Paimon. For an empty full snapshot with a consumer 
completion marker, it reads the active query's end offset and its preceding 
start offset, including the first-batch case. It returns an empty `Optional` 
when the source, batch ID, lineage, or range cannot be established safely. The 
API only exposes metadata; it does not change source progress or commit 
behavior.
   
   Tests cover split full snapshots, an empty full snapshot followed by a later 
batch, an incorrect batch ID, and a query with multiple distinct sources. 
Documentation describes the API and the empty-snapshot limitation.
   
   Validation: Spark 3 `PaimonSourceTest` 28/28 passed; Spotless check passed; 
`git diff --check` passed. A standalone offline Checkstyle invocation could not 
resolve locally uncached snapshot dependencies, so CI should run the normal 
project checks.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to