XiaoHongbo-Hope opened a new pull request, #11021: URL: https://github.com/apache/paimon/pull/11021
Spark `foreachBatch` consumers cannot obtain the Paimon source offset range for a batch through a supported connector API. Applications that need a stable source-batch identity currently inspect Spark RDD internals, and an empty full snapshot has no input partitions to inspect at all. This adds `PaimonSparkMicroBatchMetadata.offsetRange(batch, batchId)`. For batches with input partitions, it returns the exact planned start and end offsets already carried by Paimon. For an empty full snapshot with a consumer completion marker, it reads the active query's end offset and its preceding start offset, including the first-batch case. It returns an empty `Optional` when the source, batch ID, lineage, or range cannot be established safely. The API only exposes metadata; it does not change source progress or commit behavior. Tests cover split full snapshots, an empty full snapshot followed by a later batch, an incorrect batch ID, and a query with multiple distinct sources. Documentation describes the API and the empty-snapshot limitation. Validation: Spark 3 `PaimonSourceTest` 28/28 passed; Spotless check passed; `git diff --check` passed. A standalone offline Checkstyle invocation could not resolve locally uncached snapshot dependencies, so CI should run the normal project checks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
