Yes, Vincenzo it is deployed in kubernetes...any suggestions? On Mon, Jan 23, 2023 at 12:22 PM Rohit Walecha <rohi...@fnp.com> wrote:
> Hi Shawn, > > Will try applying the changes..you have suggested..and get back on this. > > On Thu, Jan 19, 2023 at 4:08 PM Rohit Walecha <rohi...@fnp.com> wrote: > >> We have multiple collections inside our cluster(3 node), but we have some >> collections having replication factor 1 and some collections having >> replication factor 2..should this be impacting our nodes..and sending them >> in recovery state..and restart !! >> >> On Wed, Jan 18, 2023 at 7:07 PM Rohit Walecha <rohi...@fnp.com> wrote: >> >>> [image: Screenshot from 2023-01-18 19-06-33.png] >>> >>> Restart pattern is above. >>> >>> On Wed, Jan 18, 2023 at 2:43 PM Rohit Walecha <rohi...@fnp.com> wrote: >>> >>>> Hi, >>>> >>>> We have a 3 node *solr(8.8.0)* cluster deployed on multiple >>>> environments which is connected to a 3 node *zookeeper(3.6.2)* cluster >>>> And, we have been facing frequent restarts of solr cloud nodes since the >>>> last few months..tried to debug this and while looking into the logs and >>>> other stats we have been seeing that the node which has restarted says : >>>> >>>> *1. * >>>> 2023-01-04 21:50:09.186 WARN (zkConnectionManagerCallback-15-thread-1) >>>> [ ] o.a.s.c.c.ConnectionManager Watcher >>>> org.apache.solr.common.cloud.ConnectionManager@731cf36d name: >>>> ZooKeeperConnection >>>> Watcher:apache-solrcloud-zookeeper-0.apache-solrcloud-zookeeper-headless.production.svc.cluster.local:2181,apache-solrcloud-zookeeper-1.apache-solrcloud-zookeeper-headless.production.svc.cluster.local:2181,apache-solrcloud-zookeeper-2.apache-solrcloud-zookeeper-headless.production.svc.cluster.local:2181/ >>>> got event WatchedEvent state:Disconnected type:None path:null path: null >>>> type: None >>>> which probably says *event state is either disconnected or expired*, >>>> and says following as a warning : >>>> WARN (zkConnectionManagerCallback-13-thread-1) [ ] >>>> o.a.s.c.c.ConnectionManager zkClient has disconnected >>>> >>>> >>>> >>>> *2*. >>>> Client session timed out, have not heard from server in 30018ms for >>>> sessionid 0x1000091fcbe0001 A session timeout from ZkClient inside Solr. >>>> *And 3.* 2023-01-04 21:50:10.685 INFO (ShutdownMonitor) [ ] >>>> o.a.s.c.ZkController Publish this node as DOWN... 2023-01-04 >>>> 21:50:10.685 INFO (ShutdownMonitor) [ ] o.a.s.c.ZkController Publish >>>> node=apache-solrcloud-0.apache-solrcloud-headless.production:8983_solr as >>>> DOWN >>>> Attached *050120223-solr-cloud-0.log* >>>> >>>> >>>> >>>> *Meanwhile zookeeper node says following the time at which solr node >>>> gets restarted : * >>>> >>>> 2023-01-15 07:11:44,349 [myid:2] - WARN >>>> [NIOWorkerThread-2:ZooKeeperServer@1384] - Connection request from old >>>> client /10.70.26.0:54584; will be dropped if server is in r-o mode >>>> 2023-01-15 07:11:44,350 [myid:2] - INFO >>>> [CommitProcessor:2:LearnerSessionTracker@116] - Committing global session >>>> 0x200042f19cf130f >>>> 2023-01-15 07:11:44,352 [myid:2] - INFO >>>> [RequestThrottler:QuorumZooKeeperServer@159] - Submitting global >>>> closeSession request for session 0x200042f19cf130f >>>> >>>> >>>> Now we are at a point where *we know that when the solr node is getting >>>> restarted, who is is pushed down the node and as we can see in the logs at >>>> [#2]* which says something like Client session timed out and it is a >>>> session which is getting established between solr node and zookeeper also >>>> while debugging this issue we have went through a series of issues >>>> reported in the current version of *zookeeper *we are using which in gist >>>> says about slower leader election and zookeeper nodes getting restarted >>>> and the whole zookeeper cluster going down while a leader is getting >>>> unhealthy/stopped/restarted and leader election happening again which is >>>> taking a long time which leads to client sessions are getting timed out >>>> during that period of time. >>>> >>>> We have tried to replicate the same on the local env by setting up a solr >>>> and zookeeper cluster by forcefully restarting/stopping leader zookeeper >>>> nodes and we have got something like : >>>> *have-not-heard-back-local-cluster.log *and We could replicate [#2]. >>>> >>>> Seeking help here..to find out what could be the possible reason for these >>>> frequent restarts of solr cloud nodes. >>>> *Regards. >>>> * >>>> >>>>
