[
https://issues.apache.org/jira/browse/SOLR-18316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Justin Pealing updated SOLR-18316:
----------------------------------
Description:
Since updating to Solrcloud 10 we have been experiencing periods of downtime
when the Zookeeper leader restarts. I have a docker-compose file which
reproduces this issue, and I believe this is a regression from 9.10.1.
h2. Steps to reproduce
The attached docker-compose.yml creates a standard 2 node Solrcloud cluster
with a 3 node Zookeeper, as well as a short script to create some empty
collections. (This doesn't exactly match our production environment but I
wanted to make sure none of our configuration was causing this issue).
{{docker compose up -d}}
{{Wait for Solr to come up}}
{{docker compose run --rm collections 10}}
Once the collections are up, identify and restart the Zookeeper leader (e.g.
docker compose restart zoo1). This results in 1 or more of the Solr nodes
showing as "(DEAD)" in the dashboard, and a number of replicas showing as
"Down" for a period of time (which may or may not result in collections being
unavailable, depending on which nodes are affected), after which the Solrcloud
cluster self-recovers.
The amount of downtime appears to be proportional to the number of collections.
With 1 collection recovery takes a few seconds, with 10 recovery takes a few
minutes. In live we have hundreds of collections and we have seen Solr remain
down for hours in this scenario (presumably it would eventually recover).
Restarting the down Solr nodes also works for recovery.
This is reproducible with 100% reliability when restarting the Zookeeper leader
with the attached docker-compose.yml, but when testing with `solr:9.10.1`
(replace the image tags in the docker-compose.yml) there is no perceptible
downtime.
h2. Logs
I've not included too many logs here as the issue is reproducible in
docker-compose, but I see a large volume of these logs on the affected node:
{{{\{2026-07-23 10:49:44.838 | 2026-07-23 09:49:44.838 INFO
(zkCallback-14-thread-5) [] o.a.s.c.Overseer Overseer
(id=216172940400852994-172.21.0.6:8983_solr-n_0000000000) closing }}}}
{{{\{2026-07-23 10:49:44.838 | 2026-07-23 09:49:44.838 INFO
(zkCallback-14-thread-5) [] o.a.s.c.Overseer Overseer
(id=216172940400852994-172.21.0.6:8983_solr-n_0000000000) closing }}}}
{{{\{2026-07-23 10:49:44.838 | 2026-07-23 09:49:44.838 INFO
(zkCallback-14-thread-5) [] o.a.s.c.Overseer Overseer
(id=216172940400852994-172.21.0.6:8983_solr-n_0000000000) closing }}}}
{{{\{2026-07-23 10:49:44.840 | 2026-07-23 09:49:44.839 INFO
(zkCallback-14-thread-5) [] o.a.s.c.Overseer Overseer
(id=216172940400852994-172.21.0.6:8983_solr-n_0000000000) closing }}}}
{{{\{2026-07-23 10:49:44.840 | 2026-07-23 09:49:44.840 INFO
(zkCallback-14-thread-5) [] o.a.s.c.Overseer Overseer
(id=216172940400852994-172.21.0.6:8983_solr-n_0000000000) closing }}}}
{{{\{2026-07-23 10:49:44.841 | 2026-07-23 09:49:44.841 INFO
(zkCallback-14-thread-5) [] o.a.s.c.OverseerElectionContext I am going to be
the leader 172.21.0.6:8983_solr }}}}
{{{\{2026-07-23 10:49:44.843 | 2026-07-23 09:49:44.843 ERROR
(zkCallback-14-thread-5) [] o.a.s.c.LeaderElector node exists
=>org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
NodeExists for /overseer_elect/leader }}}}
{{{{2026-07-23 10:49:44.843 | at
org.apache.zookeeper.KeeperException.create(KeeperException.java:125)}}}}
was:
Since updating to Solrcloud 10 we have been experiencing periods of downtime
when the Zookeeper leader restarts. I have a docker-compose file which
reproduces this issue, and I believe this is a regression from 9.10.1.
h2. Steps to reproduce
The attached docker-compose.yml creates a standard 2 node Solrcloud cluster
with a 3 node Zookeeper, as well as a short script to create some empty
collections. (This doesn't exactly match our production environment but I
wanted to make sure none of our configuration was causing this issue).
{{docker compose up -d}}
{{Wait for Solr to come up}}
{{docker compose run --rm collections 10}}
Once the collections are up, identify and restart the Zookeeper leader (e.g.
`docker compose restart zoo1`). This results in 1 or more of the Solr nodes
showing as "(DEAD)" in the dashboard, and a number of replicas showing as
"Down" for a period of time (which may or may not result in collections being
unavailable, depending on which nodes are affected), after which the Solrcloud
cluster self-recovers.
The amount of downtime appears to be proportional to the number of collections.
With 1 collection recovery takes a few seconds, with 10 recovery takes a few
minutes. In live we have hundreds of collections and we have seen Solr remain
down for hours in this scenario (presumably it would eventually recover).
Restarting the down Solr nodes also works for recovery.
This is reproducible with 100% reliability when restarting the Zookeeper leader
with the attached docker-compose.yml, but when testing with `solr:9.10.1`
(replace the image tags in the docker-compose.yml) there is no perceptible
downtime.
h2. Logs
I've not included too many logs here as the issue is reproducible in
docker-compose, but I see a large volume of these logs on the affected node:
{{2026-07-23 10:49:44.838 | 2026-07-23 09:49:44.838 INFO
(zkCallback-14-thread-5) [] o.a.s.c.Overseer Overseer
(id=216172940400852994-172.21.0.6:8983_solr-n_0000000000) closing }}
{{2026-07-23 10:49:44.838 | 2026-07-23 09:49:44.838 INFO
(zkCallback-14-thread-5) [] o.a.s.c.Overseer Overseer
(id=216172940400852994-172.21.0.6:8983_solr-n_0000000000) closing }}
{{2026-07-23 10:49:44.838 | 2026-07-23 09:49:44.838 INFO
(zkCallback-14-thread-5) [] o.a.s.c.Overseer Overseer
(id=216172940400852994-172.21.0.6:8983_solr-n_0000000000) closing }}
{{2026-07-23 10:49:44.840 | 2026-07-23 09:49:44.839 INFO
(zkCallback-14-thread-5) [] o.a.s.c.Overseer Overseer
(id=216172940400852994-172.21.0.6:8983_solr-n_0000000000) closing }}
{{2026-07-23 10:49:44.840 | 2026-07-23 09:49:44.840 INFO
(zkCallback-14-thread-5) [] o.a.s.c.Overseer Overseer
(id=216172940400852994-172.21.0.6:8983_solr-n_0000000000) closing }}
{{2026-07-23 10:49:44.841 | 2026-07-23 09:49:44.841 INFO
(zkCallback-14-thread-5) [] o.a.s.c.OverseerElectionContext I am going to be
the leader 172.21.0.6:8983_solr }}
{{2026-07-23 10:49:44.843 | 2026-07-23 09:49:44.843 ERROR
(zkCallback-14-thread-5) [] o.a.s.c.LeaderElector node exists
=>org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
NodeExists for /overseer_elect/leader }}
{{2026-07-23 10:49:44.843 | at
org.apache.zookeeper.KeeperException.create(KeeperException.java:125)}}
> Slow Solrcloud recovery following restart of Zookeeper leader
> -------------------------------------------------------------
>
> Key: SOLR-18316
> URL: https://issues.apache.org/jira/browse/SOLR-18316
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
> Affects Versions: 10.0
> Environment: Docker Desktop (Windows)
> Reporter: Justin Pealing
> Priority: Major
> Attachments: docker-compose.yml
>
>
> Since updating to Solrcloud 10 we have been experiencing periods of downtime
> when the Zookeeper leader restarts. I have a docker-compose file which
> reproduces this issue, and I believe this is a regression from 9.10.1.
> h2. Steps to reproduce
> The attached docker-compose.yml creates a standard 2 node Solrcloud cluster
> with a 3 node Zookeeper, as well as a short script to create some empty
> collections. (This doesn't exactly match our production environment but I
> wanted to make sure none of our configuration was causing this issue).
> {{docker compose up -d}}
> {{Wait for Solr to come up}}
> {{docker compose run --rm collections 10}}
> Once the collections are up, identify and restart the Zookeeper leader (e.g.
> docker compose restart zoo1). This results in 1 or more of the Solr nodes
> showing as "(DEAD)" in the dashboard, and a number of replicas showing as
> "Down" for a period of time (which may or may not result in collections being
> unavailable, depending on which nodes are affected), after which the
> Solrcloud cluster self-recovers.
> The amount of downtime appears to be proportional to the number of
> collections. With 1 collection recovery takes a few seconds, with 10 recovery
> takes a few minutes. In live we have hundreds of collections and we have seen
> Solr remain down for hours in this scenario (presumably it would eventually
> recover). Restarting the down Solr nodes also works for recovery.
> This is reproducible with 100% reliability when restarting the Zookeeper
> leader with the attached docker-compose.yml, but when testing with
> `solr:9.10.1` (replace the image tags in the docker-compose.yml) there is no
> perceptible downtime.
> h2. Logs
> I've not included too many logs here as the issue is reproducible in
> docker-compose, but I see a large volume of these logs on the affected node:
> {{{\{2026-07-23 10:49:44.838 | 2026-07-23 09:49:44.838 INFO
> (zkCallback-14-thread-5) [] o.a.s.c.Overseer Overseer
> (id=216172940400852994-172.21.0.6:8983_solr-n_0000000000) closing }}}}
> {{{\{2026-07-23 10:49:44.838 | 2026-07-23 09:49:44.838 INFO
> (zkCallback-14-thread-5) [] o.a.s.c.Overseer Overseer
> (id=216172940400852994-172.21.0.6:8983_solr-n_0000000000) closing }}}}
> {{{\{2026-07-23 10:49:44.838 | 2026-07-23 09:49:44.838 INFO
> (zkCallback-14-thread-5) [] o.a.s.c.Overseer Overseer
> (id=216172940400852994-172.21.0.6:8983_solr-n_0000000000) closing }}}}
> {{{\{2026-07-23 10:49:44.840 | 2026-07-23 09:49:44.839 INFO
> (zkCallback-14-thread-5) [] o.a.s.c.Overseer Overseer
> (id=216172940400852994-172.21.0.6:8983_solr-n_0000000000) closing }}}}
> {{{\{2026-07-23 10:49:44.840 | 2026-07-23 09:49:44.840 INFO
> (zkCallback-14-thread-5) [] o.a.s.c.Overseer Overseer
> (id=216172940400852994-172.21.0.6:8983_solr-n_0000000000) closing }}}}
> {{{\{2026-07-23 10:49:44.841 | 2026-07-23 09:49:44.841 INFO
> (zkCallback-14-thread-5) [] o.a.s.c.OverseerElectionContext I am going to be
> the leader 172.21.0.6:8983_solr }}}}
> {{{\{2026-07-23 10:49:44.843 | 2026-07-23 09:49:44.843 ERROR
> (zkCallback-14-thread-5) [] o.a.s.c.LeaderElector node exists
> =>org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
> NodeExists for /overseer_elect/leader }}}}
> {{{{2026-07-23 10:49:44.843 | at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:125)}}}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
