Neta-Rose opened a new pull request, #5359:
URL: https://github.com/apache/solr/pull/5359

   https://issues.apache.org/jira/browse/SOLR-18663
   
   # Description
   
   `RecoveryStrategy.doSyncOrReplicateRecovery()` puts the update log into 
BUFFERING on every attempt, but only the success path replays the buffer. When 
an attempt ends any other way, the update log stays in BUFFERING: cancelled by 
leader election, max retries, or the "we are now the leader" return.
   
   From then on `UpdateLog.preCommit()` skips every hard commit, so the tlog 
never rolls and grows at the full indexing rate while the core looks healthy. 
The common trigger is a replica promoted to leader mid-recovery: election calls 
`cancelRecovery()`, the loop exits via `isClosed()`, and the new leader is left 
BUFFERING. I hit this on 9.10.1: TLOG.state=1 on an indexing leader, tlog 
generation frozen, +33 GB in 12 days. This is the NRT side of SOLR-14679; that 
fix only covered TLOG replicas.
   
   # Solution
   
   The retry loop moves into a private method that returns whether recovery 
succeeded, and `doSyncOrReplicateRecovery()` calls 
`UpdateLog.dropBufferedUpdates()` in a `finally` block when it didn't. Two 
guards:
   
   - only if this recovery started buffering, so a buffer owned by MIGRATE or a 
split is left alone (see SOLR-18601)
   - not during shutdown, where `UpdateLog.close()` keeps the buffer tlog as 
the "previous recovery did not complete" marker for the next startup
   
   This doesn't change which updates get applied: nothing in the buffer was 
ever applied, and the next attempt's `bufferUpdates()` would discard it anyway.
   
   I investigated this bug myself, and used an AI coding assistant during the 
fix implementation and for parts of the tests; I reviewed and verified every 
change myself.
   
   # Tests
   
   New `RecoveryUlogStateTest`, one test per exit path plus the two guards:
   
   - cancelled recovery → ACTIVE
   - replica promoted to leader mid-recovery → ACTIVE, and hard commits roll 
the tlog again
   - max retries exceeded → ACTIVE
   - recovery on a leader buffering for someone else → buffer untouched
   - shutdown during buffering → buffer tlog kept on disk
   
   On unpatched main the first three fail every time (the promoted leader's 
tlog id stays at 1 across three hard commits), and each guard test fails if its 
guard is removed. I also ran the recovery, leader election, tlog, PeerSync, 
split and MIGRATE suites (the core ones with `-Ptests.nightly=true`) and the 
same tests on branch_9x.
   
   # Checklist
   
   Please review the following and check all that apply:
   
   - [x] I have reviewed the guidelines for [How to 
Contribute](https://github.com/apache/solr/blob/main/CONTRIBUTING.md) and my 
code conforms to the standards described there to the best of my ability.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to