lxy-9602 commented on code in PR #269:
URL: https://github.com/apache/paimon-cpp/pull/269#discussion_r3915043100
##########
src/paimon/core/operation/commit/realtime_commit_properties.cpp:
##########
@@ -285,9 +280,9 @@ Result<std::map<std::string, std::string>>
RealtimeCommitProperties::Build(
}
auto offset_iter = merged_offsets.find(partition_bucket);
int64_t previous_end_offset = offset_iter == merged_offsets.end() ? 0
: offset_iter->second;
- if (offset_range.begin != previous_end_offset) {
+ if (offset_range.begin < previous_end_offset) {
Review Comment:
Good point. This can happen if the caller passes only P1, but we consider
that invalid input to `CommitWithProgress`.
The range here describes the envelope of actual row offsets, which may be
sparse. Therefore, the committer cannot distinguish a legitimate offset gap
from an omitted prepared segment, and requiring numeric continuity would reject
valid sparse offsets.
Streaming engines generally enforce this at the checkpoint coordinator
layer. For example, Flink’s checkpoint-subsuming contract requires a later
completed checkpoint to include all artifacts from earlier uncommitted
checkpoints. The sink coordinator retains prepared-but-uncommitted committables
and submits the complete pending prefix when committing a later checkpoint.
Low-level offset APIs such as Kafka also trust the caller to commit the next
valid consumption position rather than validating that every preceding record
was processed.
We will keep sparse ranges supported and document that the upstream
coordinator must provide all earlier prepared-but-uncommitted progress for each
partition-bucket. Omitting an earlier prepared segment violates the API
contract and may cause its unpublished data to be treated as committed and
reclaimed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
