Neta Rose created SOLR-18663:
--------------------------------
Summary: Incomplete recovery leaves the update log in BUFFERING
forever; tlog never rolls (NRT replicas)
Key: SOLR-18663
URL: https://issues.apache.org/jira/browse/SOLR-18663
Project: Solr
Issue Type: Bug
Components: SolrCloud
Affects Versions: 9.10.1, 10.1, 10.0, 9.9.0, 10.2
Environment: SolrCloud 9.10.1 on Kubernetes, NRT replicas, continuous
indexing
Reporter: Neta Rose
{\{RecoveryStrategy.doSyncOrReplicateRecovery()}} calls
\{{ulog.bufferUpdates()}} on each attempt, but only the success path replays
the buffer. Every other exit (recovery closed, interrupted, max retries, "we
are now the leader") leaves the update log in BUFFERING. Nothing calls
\{{UpdateLog.dropBufferedUpdates()}} in production code.
Once stuck, \{{UpdateLog.preCommit()}} returns early on every hard commit
(state != ACTIVE), so the tlog never rolls and grows at the full indexing rate,
while the cluster sees a healthy replica or leader. It only clears when the
core goes back into recovery or restarts.
*Most common trigger:* the leader goes away while a replica is recovering.
Leader election calls \{{cancelRecovery()}} (~2.5 s before
\{{setLeader(true)}}), the recovery loop exits via \{{isClosed()}}, and the new
leader is left BUFFERING. This is the sequence in the original SOLR-14679
report. That issue's fix (9.1) only changed TLOG-replica replication; the 2025
recurrence there (9.9.0, \{{replica_n4}}, an NRT replica) fits this path.
*Production (9.10.1):* TLOG.state=1 on an indexing leader, tlog generation
frozen, +33 GB in 12 days.
*Workaround:* restart the core, or core admin \{{REQUESTAPPLYUPDATES}} on it.
*Proposed fix (PR to follow):* when a recovery that started buffering ends
without replaying it, call \{{dropBufferedUpdates()}}. It doesn't depend on
leadership, which avoids the cancel/setLeader race. It only drops a buffer this
recovery started (MIGRATE / SPLITSHARD buffers are left alone, see SOLR-18601)
and skips interrupted threads (shutdown). No change to which updates get
applied: buffered updates from a failed attempt are never applied today either,
and the next attempt's \{{bufferUpdates()}} drops them.
A new test reproduces it on main: the promoted leader stays BUFFERING and its
tlog id stays at 1 across 3 hard commits; it fails every time unpatched.
The code path is unchanged since at least the 2020 SOLR-14679 report, so older
versions are likely affected too.
The code path is unchanged since at least the 2020 SOLR-14679 report, so older
versions are likely affected too.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]