Neta Rose created SOLR-18663:
--------------------------------

             Summary: Incomplete recovery leaves the update log in BUFFERING 
forever; tlog never rolls (NRT replicas)
                 Key: SOLR-18663
                 URL: https://issues.apache.org/jira/browse/SOLR-18663
             Project: Solr
          Issue Type: Bug
          Components: SolrCloud
    Affects Versions: 9.10.1, 10.1, 10.0, 9.9.0, 10.2
         Environment: SolrCloud 9.10.1 on Kubernetes, NRT replicas, continuous 
indexing
            Reporter: Neta Rose



{\{RecoveryStrategy.doSyncOrReplicateRecovery()}} calls 
\{{ulog.bufferUpdates()}} on each attempt, but only the success path replays 
the buffer. Every other exit (recovery closed, interrupted, max retries, "we 
are now the leader") leaves the update log in BUFFERING. Nothing calls 
\{{UpdateLog.dropBufferedUpdates()}} in production code.

 

Once stuck, \{{UpdateLog.preCommit()}} returns early on every hard commit 
(state != ACTIVE), so the tlog never rolls and grows at the full indexing rate, 
while the cluster sees a healthy replica or leader. It only clears when the 
core goes back into recovery or restarts.

 

*Most common trigger:* the leader goes away while a replica is recovering. 
Leader election calls \{{cancelRecovery()}} (~2.5 s before 
\{{setLeader(true)}}), the recovery loop exits via \{{isClosed()}}, and the new 
leader is left BUFFERING. This is the sequence in the original SOLR-14679 
report. That issue's fix (9.1) only changed TLOG-replica replication; the 2025 
recurrence there (9.9.0, \{{replica_n4}}, an NRT replica) fits this path.

 

*Production (9.10.1):* TLOG.state=1 on an indexing leader, tlog generation 
frozen, +33 GB in 12 days.

 

*Workaround:* restart the core, or core admin \{{REQUESTAPPLYUPDATES}} on it.

 

*Proposed fix (PR to follow):* when a recovery that started buffering ends 
without replaying it, call \{{dropBufferedUpdates()}}. It doesn't depend on 
leadership, which avoids the cancel/setLeader race. It only drops a buffer this 
recovery started (MIGRATE / SPLITSHARD buffers are left alone, see SOLR-18601) 
and skips interrupted threads (shutdown). No change to which updates get 
applied: buffered updates from a failed attempt are never applied today either, 
and the next attempt's \{{bufferUpdates()}} drops them.

 

A new test reproduces it on main: the promoted leader stays BUFFERING and its 
tlog id stays at 1 across 3 hard commits; it fails every time unpatched.

 

The code path is unchanged since at least the 2020 SOLR-14679 report, so older 
versions are likely affected too.

The code path is unchanged since at least the 2020 SOLR-14679 report, so older 
versions are likely affected too.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to