Hi all,

I'd like to report a bug I've hit in production and have a tested fix ready
for. I've requested a JIRA account (pending approval), so I'm raising it
here in the meantime.

Problem: RecoveryStrategy.doSyncOrReplicateRecovery() calls
ulog.bufferUpdates(), but only the success path replays the buffer. If
recovery ends any other way (cancelled by leader election, max retries, "we
are now the leader"), the update log stays in BUFFERING.
UpdateLog.preCommit() then skips every hard commit, so the tlog never rolls
and grows at full indexing rate while the core looks healthy.

The most common trigger is the leader going away while a replica is
recovering. Election calls cancelRecovery() before setLeader(true), and the
new leader is left BUFFERING. This matches the original SOLR-14679 report.
That issue's fix only covered TLOG replicas, and the 2025 recurrence there
was on an NRT replica (9.9.0).

In production (9.10.1) I see TLOG.state=1 on an indexing leader, the tlog
generation frozen, and +33 GB in 12 days.

Fix: when a recovery that started buffering ends without replaying, call
the existing (currently unused) UpdateLog.dropBufferedUpdates(). It only
drops a buffer this recovery started, so MIGRATE/SPLITSHARD buffers are
untouched (see SOLR-18601), and it skips interrupted threads during
shutdown. A new test reproduces both paths on main (fails every time
unpatched), and the recovery, election, tlog and MIGRATE suites pass in
nightly mode.

I'll open a JIRA linked to SOLR-14679 and SOLR-18601, plus a PR, as soon as
my account is approved. Feedback on the approach is welcome before then.

Thanks,
Neta

Reply via email to