Got no response on the solr-user mailing list and so trying the dev-mailing list.
Please guide me if this should not be done. But I thought that the issue looks strange enough to post it here. Thanks SG On Mon, Jun 19, 2017 at 8:13 PM, S G <sg.online.em...@gmail.com> wrote: > Hi, > > We are stuck in a strange problem. > Whole cluster is red. All nodes are being shown as down. > Restart of the nodes is not helping either. > > All our nodes seem to have gone into a distributed lock. > Here is the grep command I ran on all the solr.log files: > > grep "Error trying to proxy request" $f | cut -d" " -f14 | sort | uniq > -c > And the output from 10 different solr-nodes' solr.log file is shown below: > (Basically each node is calling admin/system on other nodes and throwing > exceptions. You can see number of exceptions thrown by each server for > every other server). > > > > SVR_1.log > 13 http://SVR_2:8983/solr/my-collection/admin/system > 18 http://SVR_3:8983/solr/my-collection/admin/system > 19 http://SVR_4:8983/solr/my-collection/admin/system > 15 http://SVR_6:8983/solr/my-collection/admin/system > 13 http://SVR_7:8983/solr/my-collection/admin/system > 13 http://SVR_9:8983/solr/my-collection/admin/system > > SVR_2.log > 335 http://SVR_3:8983/solr/my-collection/admin/system > 23 http://SVR_4:8983/solr/my-collection/admin/system > 21 http://SVR_6:8983/solr/my-collection/admin/system > 23 http://SVR_7:8983/solr/my-collection/admin/system > 23 http://SVR_9:8983/solr/my-collection/admin/system > > SVR_3.log > 24 http://SVR_2:8983/solr/my-collection/admin/system > 14 http://SVR_4:8983/solr/my-collection/admin/system > 13 http://SVR_6:8983/solr/my-collection/admin/system > 14 http://SVR_7:8983/solr/my-collection/admin/system > 16 http://SVR_9:8983/solr/my-collection/admin/system > > SVR_4.log > 11 http://SVR_2:8983/solr/my-collection/admin/system > 29 http://SVR_3:8983/solr/my-collection/admin/system > 7 http://SVR_6:8983/solr/my-collection/admin/system > 16 http://SVR_7:8983/solr/my-collection/admin/system > 11 http://SVR_9:8983/solr/my-collection/admin/system > > SVR_5.log > 18 http://SVR_2:8983/solr/my-collection/admin/system > 16 http://SVR_3:8983/solr/my-collection/admin/system > 13 http://SVR_4:8983/solr/my-collection/admin/system > 12 http://SVR_6:8983/solr/my-collection/admin/system > 16 http://SVR_7:8983/solr/my-collection/admin/system > 11 http://SVR_9:8983/solr/my-collection/admin/system > > SVR_6.log > 44 http://SVR_2:8983/solr/my-collection/admin/system > 296 http://SVR_3:8983/solr/my-collection/admin/system > 40 http://SVR_4:8983/solr/my-collection/admin/system > 15 http://SVR_7:8983/solr/my-collection/admin/system > 15 http://SVR_9:8983/solr/my-collection/admin/system > > SVR_7.log > 59 http://SVR_2:8983/solr/my-collection/admin/system > 215 http://SVR_3:8983/solr/my-collection/admin/system > 62 http://SVR_4:8983/solr/my-collection/admin/system > 47 http://SVR_6:8983/solr/my-collection/admin/system > 61 http://SVR_9:8983/solr/my-collection/admin/system > > SVR_8.log > 13 http://SVR_2:8983/solr/my-collection/admin/system > 18 http://SVR_3:8983/solr/my-collection/admin/system > 10 http://SVR_4:8983/solr/my-collection/admin/system > 7 http://SVR_6:8983/solr/my-collection/admin/system > 12 http://SVR_7:8983/solr/my-collection/admin/system > 13 http://SVR_9:8983/solr/my-collection/admin/system > > SVR_9.log > 38 http://SVR_2:8983/solr/my-collection/admin/system > 229 http://SVR_3:8983/solr/my-collection/admin/system > 15 http://SVR_4:8983/solr/my-collection/admin/system > 22 http://SVR_6:8983/solr/my-collection/admin/system > 26 http://SVR_7:8983/solr/my-collection/admin/system > > SVR_10.log > 9 http://SVR_2:8983/solr/my-collection/admin/system > 22 http://SVR_3:8983/solr/my-collection/admin/system > 18 http://SVR_4:8983/solr/my-collection/admin/system > 14 http://SVR_6:8983/solr/my-collection/admin/system > 18 http://SVR_7:8983/solr/my-collection/admin/system > 10 http://SVR_9:8983/solr/my-collection/admin/system > > > Thanks > SG >
