goutamadwant opened a new pull request, #19735:
URL: https://github.com/apache/pinot/pull/19735
## Summary
- honor `lineageEntryCleanupRetentionPeriod` for both `IN_PROGRESS` and
`REVERTED` lineage entries
- protect source segments and destinations referenced by surviving lineage
entries before selecting cleanup candidates
- derive cleanup candidates independently for each optimistic-write attempt
so failed attempts cannot leak stale candidates
- prevent force cleanup and revert cleanup from deleting segments reused by
another lineage entry
- atomically fence lineage cleanup against concurrent lineage updates by
version-checking the lineage record in the same ZooKeeper transaction that
removes segments from IdealState
- add regression coverage for reused destinations, retention windows, stale
lineage versions, force cleanup, and failed optimistic writes
Fixes #19707.
## Problem
A failed segment replacement followed by a successful retry can leave
`REVERTED` and `COMPLETED` lineage entries containing the same destination
segment names.
The previous cleanup logic handled `REVERTED` entries immediately, without
applying `lineageEntryCleanupRetentionPeriod` or checking whether another
lineage entry still referenced those destinations. As a result,
RetentionManager could remove a segment that belonged to the successful
replacement.
Cleanup candidates could also survive a failed optimistic-write attempt, and
the previous privileged cleanup path was not fenced against a lineage mutation
committed by another controller between metadata cleanup and IdealState removal.
## Implementation
The retention pass now computes the cleanup threshold once and applies it
consistently to both `IN_PROGRESS` and `REVERTED` entries. It builds the
protected segment set before mutating lineage metadata, making the result
independent of lineage-entry iteration order.
Cleanup candidates in retention and force-cleanup paths are scoped to the
optimistic-write attempt that produced them. A candidate is removed when the
resulting lineage contains more references than the cleanup operation owns.
Privileged cleanup now requires the exact lineage version used to authorize
the candidate set. The lineage version check and IdealState removal are
committed in one ZooKeeper transaction. If another controller changes lineage
first, deletion fails closed without modifying IdealState.
No configuration, serialized lineage format, or external dependency is added.
## Testing
Executed with JDK 25:
- `./mvnw -pl pinot-controller -Ddevelocity.enabled=false -Dskip.npm=true
-Dtest=DefaultLineageManagerTest,RetentionManagerTest#testLineageCleanupDiscardsDeleteCandidatesFromFailedWriteAttempt,LineageDeleteExclusionTest,LineageDeleteInterleavingIntegrationTest,SegmentLineageCleanupTest,PinotHelixResourceManagerStatelessTest#testSegmentReplacementForRefresh
test`
- 42 tests passed
- 0 failures
- 0 errors
- 0 skipped
- `./mvnw -pl pinot-controller -DskipTests spotless:apply license:format`
- Checkstyle, dependency convergence, and Java-version enforcement passed as
part of the Maven test run.
- `git diff --check` passed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
