LsomeYeah opened a new pull request, #9273: URL: https://github.com/apache/paimon/pull/9273
## Summary Add Paimon Consumer progress support to the Spark Structured Streaming source. This gives a long-running Spark query a table-side, snapshot-level recovery position and retention fence in addition to its Spark checkpoint. ## Changes - Advance Consumer progress from `MicroBatchStream.commit` only after Spark commits the micro-batch containing the final split of a snapshot. - Persist optional split-count metadata in the source offset while preserving the existing three-field offset API and legacy checkpoint decoding. - Keep partial and legacy offsets conservative, propagate Consumer update failures, and resume a completed snapshot from the following incremental snapshot. - Support branch-scoped Consumers and long-lived changelog recovery, and reject expired checkpoint ranges that would mix full and incremental scan semantics. - Document startup precedence, checkpoint and Consumer recovery behavior, expiration, and concurrent Consumer ID limitations. ## Semantics - Spark checkpoint recovery remains precise when the checkpoint is available. - Consumer-only recovery is snapshot-granular and may replay data, but it does not advance past a partially consumed snapshot. - Existing checkpoints without split-count metadata remain readable; Consumer progress stays unchanged until snapshot completion can be proven. - No change is made to the core Consumer format or Flink Consumer behavior. ## Testing - [x] Spark 3 source offset, unit, and integration tests - [x] Spark 4 source offset, unit, and integration tests - [x] Full snapshot split across multiple micro-batches - [x] Consumer update failure and retry - [x] Consumer-only restart and legacy checkpoint compatibility - [x] Branch scope, snapshot expiration, and long-lived changelog recovery - [x] Real Spark source commit callback integration - [x] Spotless, Checkstyle, Enforcer, and diff checks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
