fhan688 commented on code in PR #20486:
URL: https://github.com/apache/hudi/pull/20486#discussion_r4438547285
##########
hudi-common/src/main/java/org/apache/hudi/common/util/KafkaCheckpointUtils.java:
##########
@@ -109,28 +160,61 @@ public static Map<Integer, Long>
parseCheckpoint(CheckpointFormat format, String
}
/**
- * Calculate offset difference between two checkpoints.
- * Handles partition additions, removals, and resets.
+ * Calculate the known offset difference between two checkpoints.
*
* Algorithm:
* 1. For each partition in current checkpoint:
* - If partition exists in previous: diff = current - previous
* - If partition is new: skip (start offset unknown, would overcount)
* - If diff is negative (reset): skip (start offset unknown, would
overcount)
- * 2. Sum all partition diffs
+ * 2. Sum all comparable partition diffs
+ *
+ * <p>If partitions are added or removed, or an offset reset is detected,
the returned value
+ * is only a partial difference. Callers that require an exact difference
should use
+ * {@link #compareOffsets(CheckpointFormat, String, String)} and check
+ * {@link OffsetComparisonResult#isExact()}.</p>
*
* @param format Checkpoint format
* @param previousCheckpoint Previous checkpoint string
* @param currentCheckpoint Current checkpoint string
- * @return Total offset difference across all partitions
+ * @return Known offset difference across comparable partitions
*/
public static long calculateOffsetDifference(CheckpointFormat format,
Review Comment:
Confirmed that `calculateOffsetDifference()` has no production callers in
the current master. I removed the method and migrated the common and Flink
tests to `compareOffsets()`, asserting both the known offset difference and
whether the comparison is exact.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
