deepthi912 commented on code in PR #19832:
URL: https://github.com/apache/pinot/pull/19832#discussion_r4252190068
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/BasePartitionUpsertMetadataManager.java:
##########
@@ -1177,6 +1191,34 @@ protected void doTakeSnapshot(@Nullable
DocIdsSnapshot.Trigger trigger) {
numConsumingSegments, System.currentTimeMillis() - startTimeMs);
}
+ /// Returns the TTL watermark to persist after taking snapshots. A segment
still tracked as updated after the snapshot
+ /// pass, e.g. skipped on segmentLock contention or added while the pass
ran, has a stale snapshot or no snapshot on
+ /// disk. If such segment were out of TTL after a restart, it would be added
with the stale snapshot, or with all
+ /// docs treated as valid when there is no snapshot, which shows up as
duplicate rows on this replica only. So cap
+ /// the persisted watermark to keep such segments within TTL, and get them
added with the regular flow after restart.
+ /// The watermark in memory is not changed.
+ private double getWatermarkToPersist() {
Review Comment:
It will be consistent if we use the same watermark when removing the
metadata in:
`doRemoveExpiredPrimaryKeys`. In place of `largestSeenComparisonValue`, we
might have to use `getWatermarkToPersist` there
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
