tarun11Mavani opened a new pull request, #16006:
URL: https://github.com/apache/pinot/pull/16006
This PR addresses an issue discovered in #15846 where inconsistent
segment.creation.time across replicas and due to reliance on server-local
creation.meta files, can lead to divergent states during upsert tie-breaking.
To ensure consistency, this change modifies the
`shouldReplaceOnComparisonTie` logic to use `segment.creation.time` from
controller ZK metadata as the single source of truth.
Key changes:
- Introduces a _segmentCreationTimeCache (Map of segmentName to
creationTime) within BasePartitionUpsertMetadataManager.
- Adds a segmentZKCreationTimeProvider that fetches ZK metadata using
TableDataManager.fetchZKMetadata().
- The cache is populated and maintained during segment lifecycle operations:
- doAddSegment(): Caches creation time for new segments.
- doPreloadSegment(): Caches creation time during preloading.
- doReplaceSegment(): Updates cache for the new segment and removes the
old.
- doRemoveSegment(): Removes the entry for deleted segments.
- The lookup strategy prioritizes the cached ZK creation time, with lazy
loading from PinotHelixResourceManager (fetching from ZK) on a cache miss. If
the creation time is not found from controller, we rely on creation.meta stored
on server.
This approach ensures that all replicas use the same creation time for
tie-breaking, leading to a consistent state across the cluster.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
