lilei1128 opened a new pull request, #9401:
URL: https://github.com/apache/paimon/pull/9401
### Purpose
When a Spark streaming source started in latest-full or
from-snapshot-full mode from an empty snapshot, no end offset was
generated and the source could not continue with subsequent delta
snapshots. With consumer-id configured, the consumer checkpoint also
remained unchanged.
Generate a completion offset for empty full snapshots, advance the
source and consumer to the next snapshot, preserve correct
ReadMinRows behavior, and support checkpoint recovery while keeping
legacy offset compatibility.
- Mark an existing zero-split full snapshot as completed and generate an
end offset.
- Store the next snapshot position in that offset and resume subsequent
reads in delta mode.
- Advance the Paimon consumer when the empty full snapshot is committed.
- Keep ReadMinRows semantics for batches containing actual delta data.
- Preserve the empty-snapshot state across Spark checkpoint serialization
and restart.
- Keep legacy offsets without totalSplits readable.
- Prevent inconsistent totalSplits metadata when offsets are copied.
### Tests
Tests cover：
- Empty snapshot offset serialization and copy behavior;
- Source advancement with and without consumer-id;
- Consumer checkpoint updates and retry failures;
- ReadMinRows delay and recovery;
- Spark query restart from checkpoint;
- Continued consumption of data after the empty snapshot;
- Existing full-snapshot, delta, trigger, and read-limit behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
