[
https://issues.apache.org/jira/browse/SOLR-18548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18128924#comment-18128924
]
Boyi Dai edited comment on SOLR-18548 at 10/19/26 11:01 PM:
------------------------------------------------------------
[~psalagnac]
Thanks for taking a look! I found it by analyzing the code rather than in a
real cluster. I haven't seen it in production.
The failure needs runningMap.put to fail with something other than
NodeExistsException, e.g. a ConnectionLossException that outlasts the ZK
client's retries, or an OperationTimeoutException, during a ZK hiccup. That's
rare, but when it happens the lock is never released, so it doesn't clear up on
its own: every later task for that collection stays blocked until the Overseer
restarts. That's why I thought it was worth fixing even without a field report.
was (Author: JIRAUSER314998):
[~psalagnac]
Thanks for taking a look! I found it by analyzing the code rather than in a
real cluster. I haven't seen it in production.
The failure needs runningMap.put to fail with something other than
NodeExistsException, e.g. a ConnectionLossException that outlasts the ZK
client's retries, or an OperationTimeoutException, during a ZK hiccup. That's
rare, but when it happens the lock is never released, so it doesn't clear up on
its own: every later task for that collection stays blocked until the Overseer
restarts. That's why I thought it was worth fixing even without a field report.
> Overseer collection lock leaks when marking an async task as running fails
> --------------------------------------------------------------------------
>
> Key: SOLR-18548
> URL: https://issues.apache.org/jira/browse/SOLR-18548
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
> Affects Versions: 10.0
> Reporter: Boyi Dai
> Priority: Minor
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> OverseerTaskProcessor takes the lock for a task (collection, shard or
> replica) before storing the task’s async ID in the running map in ZooKeeper.
> If that {{runningMap.put}} fails, only {{NodeExistsException}} and
> {{InterruptedException}} release the lock. Any other {{KeeperException}} (for
> example {{{}OperationTimeoutException{}}}, or a {{ConnectionLossException}}
> after retries) or a {{RuntimeException}} goes to the outer loop’s handler
> with the lock still held.
> No Runner has been created at that point, so nothing ever releases the lock.
> Every later task that needs the same lock is blocked until the Overseer
> restarts, so a single transient ZooKeeper error during that write can stall
> all collection API operations on that collection.
> This has the same symptom as SOLR-17754 (a lock left held so tasks never
> run), but a different cause: SOLR-17754 fixed a race with the runner thread
> pool, while this happens when the ZooKeeper write fails before a Runner
> exists.
> Expected: the lock is released whenever the task is not handed to a Runner,
> so the task can be retried and other tasks for the same collection can
> proceed. A fix with a test is coming in a PR.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
