Hi, We are stuck in a strange problem. Whole cluster is red. All nodes are being shown as down. Restart of the nodes is not helping either.
All our nodes seem to have gone into a distributed lock.
Here is the grep command I ran on all the solr.log files:
grep "Error trying to proxy request" $f | cut -d" " -f14 | sort | uniq
-c
And the output from 10 different solr-nodes' solr.log file is shown below:
(Basically each node is calling admin/system on other nodes and throwing
exceptions. You can see number of exceptions thrown by each server for
every other server).
SVR_1.log
13 http://SVR_2:8983/solr/my-collection/admin/system
18 http://SVR_3:8983/solr/my-collection/admin/system
19 http://SVR_4:8983/solr/my-collection/admin/system
15 http://SVR_6:8983/solr/my-collection/admin/system
13 http://SVR_7:8983/solr/my-collection/admin/system
13 http://SVR_9:8983/solr/my-collection/admin/system
SVR_2.log
335 http://SVR_3:8983/solr/my-collection/admin/system
23 http://SVR_4:8983/solr/my-collection/admin/system
21 http://SVR_6:8983/solr/my-collection/admin/system
23 http://SVR_7:8983/solr/my-collection/admin/system
23 http://SVR_9:8983/solr/my-collection/admin/system
SVR_3.log
24 http://SVR_2:8983/solr/my-collection/admin/system
14 http://SVR_4:8983/solr/my-collection/admin/system
13 http://SVR_6:8983/solr/my-collection/admin/system
14 http://SVR_7:8983/solr/my-collection/admin/system
16 http://SVR_9:8983/solr/my-collection/admin/system
SVR_4.log
11 http://SVR_2:8983/solr/my-collection/admin/system
29 http://SVR_3:8983/solr/my-collection/admin/system
7 http://SVR_6:8983/solr/my-collection/admin/system
16 http://SVR_7:8983/solr/my-collection/admin/system
11 http://SVR_9:8983/solr/my-collection/admin/system
SVR_5.log
18 http://SVR_2:8983/solr/my-collection/admin/system
16 http://SVR_3:8983/solr/my-collection/admin/system
13 http://SVR_4:8983/solr/my-collection/admin/system
12 http://SVR_6:8983/solr/my-collection/admin/system
16 http://SVR_7:8983/solr/my-collection/admin/system
11 http://SVR_9:8983/solr/my-collection/admin/system
SVR_6.log
44 http://SVR_2:8983/solr/my-collection/admin/system
296 http://SVR_3:8983/solr/my-collection/admin/system
40 http://SVR_4:8983/solr/my-collection/admin/system
15 http://SVR_7:8983/solr/my-collection/admin/system
15 http://SVR_9:8983/solr/my-collection/admin/system
SVR_7.log
59 http://SVR_2:8983/solr/my-collection/admin/system
215 http://SVR_3:8983/solr/my-collection/admin/system
62 http://SVR_4:8983/solr/my-collection/admin/system
47 http://SVR_6:8983/solr/my-collection/admin/system
61 http://SVR_9:8983/solr/my-collection/admin/system
SVR_8.log
13 http://SVR_2:8983/solr/my-collection/admin/system
18 http://SVR_3:8983/solr/my-collection/admin/system
10 http://SVR_4:8983/solr/my-collection/admin/system
7 http://SVR_6:8983/solr/my-collection/admin/system
12 http://SVR_7:8983/solr/my-collection/admin/system
13 http://SVR_9:8983/solr/my-collection/admin/system
SVR_9.log
38 http://SVR_2:8983/solr/my-collection/admin/system
229 http://SVR_3:8983/solr/my-collection/admin/system
15 http://SVR_4:8983/solr/my-collection/admin/system
22 http://SVR_6:8983/solr/my-collection/admin/system
26 http://SVR_7:8983/solr/my-collection/admin/system
SVR_10.log
9 http://SVR_2:8983/solr/my-collection/admin/system
22 http://SVR_3:8983/solr/my-collection/admin/system
18 http://SVR_4:8983/solr/my-collection/admin/system
14 http://SVR_6:8983/solr/my-collection/admin/system
18 http://SVR_7:8983/solr/my-collection/admin/system
10 http://SVR_9:8983/solr/my-collection/admin/system
Thanks
SG
