[ 
https://issues.apache.org/jira/browse/SOLR-18663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18145860#comment-18145860
 ] 

Neta Rose commented on SOLR-18663:
----------------------------------

Regarding 14679:

[~houston] [~gjourdan] I believe the 9.9.0 recurrence is a different path from 
the TLOG-replica fix: {{replica_n4}} is an NRT replica. In 
{{RecoveryStrategy}}, any recovery exit that doesn't replay the buffer (most 
often leader election cancelling recovery) leaves the ulog BUFFERING, exactly 
as in the original report. I hit it on 9.10.1 (+33 GB tlog on a leader) and 
reproduced it on main with a test. Details and a fix in SOLR-18663, PR coming 
shortly.


> Incomplete recovery leaves the update log in BUFFERING forever; tlog never 
> rolls (NRT replicas)
> -----------------------------------------------------------------------------------------------
>
>                 Key: SOLR-18663
>                 URL: https://issues.apache.org/jira/browse/SOLR-18663
>             Project: Solr
>          Issue Type: Bug
>          Components: SolrCloud
>    Affects Versions: 9.9.0, 10.0, 10.1, 9.10.1, 10.2
>         Environment: SolrCloud 9.10.1 on Kubernetes, NRT replicas, continuous 
> indexing
>            Reporter: Neta Rose
>            Priority: Major
>
> {\{RecoveryStrategy.doSyncOrReplicateRecovery()}} calls 
> \{{ulog.bufferUpdates()}} on each attempt, but only the success path replays 
> the buffer. Every other exit (recovery closed, interrupted, max retries, "we 
> are now the leader") leaves the update log in BUFFERING. Nothing calls 
> \{{UpdateLog.dropBufferedUpdates()}} in production code.
>  
> Once stuck, \{{UpdateLog.preCommit()}} returns early on every hard commit 
> (state != ACTIVE), so the tlog never rolls and grows at the full indexing 
> rate, while the cluster sees a healthy replica or leader. It only clears when 
> the core goes back into recovery or restarts.
>  
> *Most common trigger:* the leader goes away while a replica is recovering. 
> Leader election calls \{{cancelRecovery()}} (~2.5 s before 
> \{{setLeader(true)}}), the recovery loop exits via \{{isClosed()}}, and the 
> new leader is left BUFFERING. This is the sequence in the original SOLR-14679 
> report. That issue's fix (9.1) only changed TLOG-replica replication; the 
> 2025 recurrence there (9.9.0, \{{replica_n4}}, an NRT replica) fits this path.
>  
> *Production (9.10.1):* TLOG.state=1 on an indexing leader, tlog generation 
> frozen, +33 GB in 12 days.
>  
> *Workaround:* restart the core, or core admin \{{REQUESTAPPLYUPDATES}} on it.
>  
> *Proposed fix (PR to follow):* when a recovery that started buffering ends 
> without replaying it, call \{{dropBufferedUpdates()}}. It doesn't depend on 
> leadership, which avoids the cancel/setLeader race. It only drops a buffer 
> this recovery started (MIGRATE / SPLITSHARD buffers are left alone, see 
> SOLR-18601) and skips interrupted threads (shutdown). No change to which 
> updates get applied: buffered updates from a failed attempt are never applied 
> today either, and the next attempt's \{{bufferUpdates()}} drops them.
>  
> A new test reproduces it on main: the promoted leader stays BUFFERING and its 
> tlog id stays at 1 across 3 hard commits; it fails every time unpatched.
>  
> The code path is unchanged since at least the 2020 SOLR-14679 report, so 
> older versions are likely affected too.
> The code path is unchanged since at least the 2020 SOLR-14679 report, so 
> older versions are likely affected too.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to