Neta-Rose opened a new pull request, #5359: URL: https://github.com/apache/solr/pull/5359
https://issues.apache.org/jira/browse/SOLR-18663 # Description `RecoveryStrategy.doSyncOrReplicateRecovery()` puts the update log into BUFFERING on every attempt, but only the success path replays the buffer. When an attempt ends any other way, the update log stays in BUFFERING: cancelled by leader election, max retries, or the "we are now the leader" return. From then on `UpdateLog.preCommit()` skips every hard commit, so the tlog never rolls and grows at the full indexing rate while the core looks healthy. The common trigger is a replica promoted to leader mid-recovery: election calls `cancelRecovery()`, the loop exits via `isClosed()`, and the new leader is left BUFFERING. I hit this on 9.10.1: TLOG.state=1 on an indexing leader, tlog generation frozen, +33 GB in 12 days. This is the NRT side of SOLR-14679; that fix only covered TLOG replicas. # Solution The retry loop moves into a private method that returns whether recovery succeeded, and `doSyncOrReplicateRecovery()` calls `UpdateLog.dropBufferedUpdates()` in a `finally` block when it didn't. Two guards: - only if this recovery started buffering, so a buffer owned by MIGRATE or a split is left alone (see SOLR-18601) - not during shutdown, where `UpdateLog.close()` keeps the buffer tlog as the "previous recovery did not complete" marker for the next startup This doesn't change which updates get applied: nothing in the buffer was ever applied, and the next attempt's `bufferUpdates()` would discard it anyway. I investigated this bug myself, and used an AI coding assistant during the fix implementation and for parts of the tests; I reviewed and verified every change myself. # Tests New `RecoveryUlogStateTest`, one test per exit path plus the two guards: - cancelled recovery → ACTIVE - replica promoted to leader mid-recovery → ACTIVE, and hard commits roll the tlog again - max retries exceeded → ACTIVE - recovery on a leader buffering for someone else → buffer untouched - shutdown during buffering → buffer tlog kept on disk On unpatched main the first three fail every time (the promoted leader's tlog id stays at 1 across three hard commits), and each guard test fails if its guard is removed. I also ran the recovery, leader election, tlog, PeerSync, split and MIGRATE suites (the core ones with `-Ptests.nightly=true`) and the same tests on branch_9x. # Checklist Please review the following and check all that apply: - [x] I have reviewed the guidelines for [How to Contribute](https://github.com/apache/solr/blob/main/CONTRIBUTING.md) and my code conforms to the standards described there to the best of my ability. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
