shangxinli commented on code in PR #18889:
URL: https://github.com/apache/hudi/pull/18889#discussion_r4433996441
##########
hudi-utilities/src/main/java/org/apache/hudi/utilities/streamer/StreamSync.java:
##########
@@ -947,10 +950,21 @@ Map<String, String> extractCheckpointMetadata(InputBatch
inputBatch, TypedProper
return Collections.emptyMap();
}
- // If we have a next checkpoint batch, use its metadata
+ // If we have a next checkpoint batch, use its metadata. Some Source
implementations (e.g.
+ // DFSPathSelector-backed *DFSSource) unconditionally return a
StreamerCheckpointV2 regardless
+ // of the target table version. The persisted checkpoint key must match
the table version
+ // contract enforced by CheckpointUtils.shouldTargetCheckpointV2 (V2 keys
are only valid for
+ // writeTableVersion >= 8 and outside the not-supported set). Normalize
here so that a v6
+ // write does not get a V2 key stamped into its commit metadata.
if (inputBatch.getCheckpointForNextBatch() != null) {
- return inputBatch.getCheckpointForNextBatch()
- .getCheckpointCommitMetadata(cfg.checkpoint, cfg.ignoreCheckpoint);
+ Checkpoint sourceCheckpoint = inputBatch.getCheckpointForNextBatch();
+ boolean targetV2 = CheckpointUtils.shouldTargetCheckpointV2(versionCode,
cfg.sourceClassName);
+ if (targetV2 && sourceCheckpoint instanceof StreamerCheckpointV1) {
+ sourceCheckpoint = new StreamerCheckpointV2(sourceCheckpoint);
Review Comment:
this does a raw field copy, not a semantic translation. V1 checkpoint keys
are request-time-based and V2 keys are completion-time-based for timeline
sources, and CheckpointUtils already has
convertToCheckpointV1ForCommitTime/convertToCheckpointV2ForCommitTime that do
the real timeline lookup. HoodieIncrSource isn't in
DATASOURCES_NOT_SUPPORTED_WITH_CKPT_V2, so it can hit this naive copy at a
version boundary and get its checkpoint silently reinterpreted under the wrong
time semantics.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
