Hi,
I have a 5 nodes cluster (solr-7.4.0 + zookeeper-3.4.12) with relatively
small collection count. Each core has 2 shards, 1 replica (totalling 4
items).
>From time to time, without notice, I see that shards get relocated but
sometimes this happens [1]: shard (shard2 in this example) seems to go
under leader election but it never terminates. In this case, it seems that
shard2 needs to migrate leadership from solr01 to solr03. Logs get filled
with those FROMLEADER TOLEADER indefinitely and sometimes it terminates on
Out of Memory errors which leads to said shard without a leader nor
recovery and i need to force a leader manually based on my perception of
the best leader.
I'm pretty sure I'm missing some important information here. Could someone
help?
Thanks
Luca
[1]
2024-04-12 09:39:38.071 INFO (qtp551479935-12193) [c:porcelletto s:shard2
r:core_node132 x:porcelletto_shard2_replica_n131]
o.a.s.u.p.LogUpdateProcessorFactory [porcelletto_shard2_replica_n131]
webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=
http://solr01:8983/solr/porcelletto_shard2_replica_n129/&wt=javabin&version=2}{add=[20694b35fdf43a7964726754e25aafc9
(1796121326327431168)]}
0 1
2024-04-12 09:39:38.072 INFO (qtp551479935-18092) [c:porcelletto s:shard2
r:core_node130 x:porcelletto_shard2_replica_n129]
o.a.s.u.p.LogUpdateProcessorFactory [porcelletto_shard2_replica_n129]
webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=
http://solr03:8983/solr/porcelletto_shard2_replica_n139/&wt=javabin&version=2}{add=[20694b35fdf43a7964726754e25aafc9
(1796121326327431168)]}
0 3
2024-04-12 09:39:38.642 INFO (qtp551479935-16945) [c:porcelletto s:shard2
r:core_node132 x:porcelletto_shard2_replica_n131]
o.a.s.u.p.LogUpdateProcessorFactory [porcelletto_shard2_replica_n131]
webapp=/solr path=/update params={update.distrib=FROMLEADER&distrib.from=
http://solr01:8983/solr/porcelletto_shard2_replica_n129/&wt=javabin&version=2}{add=[b56fc1ed62e88c560cb9b7491fe3dc3b
(1796121326916730880)]}
0 10
2024-04-12 09:39:38.642 INFO (qtp551479935-18538) [c:porcelletto s:shard2
r:core_node130 x:porcelletto_shard2_replica_n129]
o.a.s.u.p.LogUpdateProcessorFactory [porcelletto_shard2_replica_n129]
webapp=/solr path=/update params={update.distrib=TOLEADER&distrib.from=
http://solr03:8983/solr/porcelletto_shard1_replica_n137/&wt=javabin&version=2}{add=[b56fc1ed62e88c560cb9b7491fe3dc3b
(1796121326916730880)]}
0 1
