[
https://issues.apache.org/jira/browse/SOLR-18663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18145860#comment-18145860
]
Neta Rose commented on SOLR-18663:
----------------------------------
Regarding 14679:
[~houston] [~gjourdan] I believe the 9.9.0 recurrence is a different path from
the TLOG-replica fix: {{replica_n4}} is an NRT replica. In
{{RecoveryStrategy}}, any recovery exit that doesn't replay the buffer (most
often leader election cancelling recovery) leaves the ulog BUFFERING, exactly
as in the original report. I hit it on 9.10.1 (+33 GB tlog on a leader) and
reproduced it on main with a test. Details and a fix in SOLR-18663, PR coming
shortly.
> Incomplete recovery leaves the update log in BUFFERING forever; tlog never
> rolls (NRT replicas)
> -----------------------------------------------------------------------------------------------
>
> Key: SOLR-18663
> URL: https://issues.apache.org/jira/browse/SOLR-18663
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
> Affects Versions: 9.9.0, 10.0, 10.1, 9.10.1, 10.2
> Environment: SolrCloud 9.10.1 on Kubernetes, NRT replicas, continuous
> indexing
> Reporter: Neta Rose
> Priority: Major
>
> {\{RecoveryStrategy.doSyncOrReplicateRecovery()}} calls
> \{{ulog.bufferUpdates()}} on each attempt, but only the success path replays
> the buffer. Every other exit (recovery closed, interrupted, max retries, "we
> are now the leader") leaves the update log in BUFFERING. Nothing calls
> \{{UpdateLog.dropBufferedUpdates()}} in production code.
>
> Once stuck, \{{UpdateLog.preCommit()}} returns early on every hard commit
> (state != ACTIVE), so the tlog never rolls and grows at the full indexing
> rate, while the cluster sees a healthy replica or leader. It only clears when
> the core goes back into recovery or restarts.
>
> *Most common trigger:* the leader goes away while a replica is recovering.
> Leader election calls \{{cancelRecovery()}} (~2.5 s before
> \{{setLeader(true)}}), the recovery loop exits via \{{isClosed()}}, and the
> new leader is left BUFFERING. This is the sequence in the original SOLR-14679
> report. That issue's fix (9.1) only changed TLOG-replica replication; the
> 2025 recurrence there (9.9.0, \{{replica_n4}}, an NRT replica) fits this path.
>
> *Production (9.10.1):* TLOG.state=1 on an indexing leader, tlog generation
> frozen, +33 GB in 12 days.
>
> *Workaround:* restart the core, or core admin \{{REQUESTAPPLYUPDATES}} on it.
>
> *Proposed fix (PR to follow):* when a recovery that started buffering ends
> without replaying it, call \{{dropBufferedUpdates()}}. It doesn't depend on
> leadership, which avoids the cancel/setLeader race. It only drops a buffer
> this recovery started (MIGRATE / SPLITSHARD buffers are left alone, see
> SOLR-18601) and skips interrupted threads (shutdown). No change to which
> updates get applied: buffered updates from a failed attempt are never applied
> today either, and the next attempt's \{{bufferUpdates()}} drops them.
>
> A new test reproduces it on main: the promoted leader stays BUFFERING and its
> tlog id stays at 1 across 3 hard commits; it fails every time unpatched.
>
> The code path is unchanged since at least the 2020 SOLR-14679 report, so
> older versions are likely affected too.
> The code path is unchanged since at least the 2020 SOLR-14679 report, so
> older versions are likely affected too.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]