Hi all, I'd like to report a bug I've hit in production and have a tested fix ready for. I've requested a JIRA account (pending approval), so I'm raising it here in the meantime.
Problem: RecoveryStrategy.doSyncOrReplicateRecovery() calls ulog.bufferUpdates(), but only the success path replays the buffer. If recovery ends any other way (cancelled by leader election, max retries, "we are now the leader"), the update log stays in BUFFERING. UpdateLog.preCommit() then skips every hard commit, so the tlog never rolls and grows at full indexing rate while the core looks healthy. The most common trigger is the leader going away while a replica is recovering. Election calls cancelRecovery() before setLeader(true), and the new leader is left BUFFERING. This matches the original SOLR-14679 report. That issue's fix only covered TLOG replicas, and the 2025 recurrence there was on an NRT replica (9.9.0). In production (9.10.1) I see TLOG.state=1 on an indexing leader, the tlog generation frozen, and +33 GB in 12 days. Fix: when a recovery that started buffering ends without replaying, call the existing (currently unused) UpdateLog.dropBufferedUpdates(). It only drops a buffer this recovery started, so MIGRATE/SPLITSHARD buffers are untouched (see SOLR-18601), and it skips interrupted threads during shutdown. A new test reproduces both paths on main (fails every time unpatched), and the recovery, election, tlog and MIGRATE suites pass in nightly mode. I'll open a JIRA linked to SOLR-14679 and SOLR-18601, plus a PR, as soon as my account is approved. Feedback on the approach is welcome before then. Thanks, Neta
