AkshayUpadhyay2411 commented on code in PR #19832:
URL: https://github.com/apache/pinot/pull/19832#discussion_r4254937715
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/BasePartitionUpsertMetadataManager.java:
##########
@@ -1177,6 +1191,34 @@ protected void doTakeSnapshot(@Nullable
DocIdsSnapshot.Trigger trigger) {
numConsumingSegments, System.currentTimeMillis() - startTimeMs);
}
+ /// Returns the TTL watermark to persist after taking snapshots. A segment
still tracked as updated after the snapshot
+ /// pass, e.g. skipped on segmentLock contention or added while the pass
ran, has a stale snapshot or no snapshot on
+ /// disk. If such segment were out of TTL after a restart, it would be added
with the stale snapshot, or with all
+ /// docs treated as valid when there is no snapshot, which shows up as
duplicate rows on this replica only. So cap
+ /// the persisted watermark to keep such segments within TTL, and get them
added with the regular flow after restart.
+ /// The watermark in memory is not changed.
+ private double getWatermarkToPersist() {
Review Comment:
Kept both intentionally separate: if getWatermarkToPersist() lags behind
largestSeenComparisonValue, and we have a high rate of snapshot misses, PKs
might not expire.
So for in-memory expiry, kept largestSeenComparisonValue, and only for
restarts/reloads etc, using getWatermarkToPersist(). I think the right balance
depends mostly on how often snapshot misses happen — what should be the ideal
way?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
