[
https://issues.apache.org/jira/browse/SOLR-5724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13901543#comment-13901543
]
Joseph Duchesne edited comment on SOLR-5724 at 2/14/14 3:27 PM:
----------------------------------------------------------------
Here are the logs from replica 1:
{quote}
ERROR - 2014-02-13 13:39:01.159; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ...
retries: 12
ERROR - 2014-02-13 13:39:01.175; org.apache.solr.common.SolrException; Error
while trying to recover.
core=listsC:org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
We are not the leader
at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:495)
at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)
at
org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:224)
at
org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:371)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:247)
ERROR - 2014-02-13 13:39:01.175; org.apache.solr.cloud.RecoveryStrategy;
Recovery failed - trying again... (0) core=listsC
WARN - 2014-02-13 13:39:01.180; org.apache.solr.cloud.RecoveryStrategy;
Stopping recovery for zkNodeName=core_node2core=listsC
ERROR - 2014-02-13 13:39:01.183; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/ABC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FABC%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.183; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/ABC/ failed - retrying ...
retries: 11
ERROR - 2014-02-13 13:39:01.271; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/ABC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FABC%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.271; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/ABC/ failed - retrying ...
retries: 4
ERROR - 2014-02-13 13:39:01.340; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.340; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ...
retries: 1
ERROR - 2014-02-13 13:39:01.409; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.409; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ...
retries: 6
ERROR - 2014-02-13 13:39:01.648; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.648; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.648; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ...
retries: 2
ERROR - 2014-02-13 13:39:01.648; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ...
retries: 2
ERROR - 2014-02-13 13:39:01.663; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.663; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ...
retries: 13
ERROR - 2014-02-13 13:39:01.687; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/ABC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FABC%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.688; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/ABC/ failed - retrying ...
retries: 12
ERROR - 2014-02-13 13:39:01.776; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/ABC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FABC%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.776; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/ABC/ failed - retrying ...
retries: 5
ERROR - 2014-02-13 13:39:01.845; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.845; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ...
retries: 2
ERROR - 2014-02-13 13:39:01.913; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.914; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ...
retries: 7
ERROR - 2014-02-13 13:39:02.153; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:02.153; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
{quote}
And replica 2:
{quote}
ERROR - 2014-02-13 13:39:01.143; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ...
retries: 1
ERROR - 2014-02-13 13:39:01.158; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.159; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ...
retries: 12
ERROR - 2014-02-13 13:39:01.175; org.apache.solr.common.SolrException; Error
while trying to recover.
core=listsC:org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
We are not the leader
at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:495)
at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)
at
org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:224)
at
org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:371)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:247)
ERROR - 2014-02-13 13:39:01.175; org.apache.solr.cloud.RecoveryStrategy;
Recovery failed - trying again... (0) core=listsC
WARN - 2014-02-13 13:39:01.180; org.apache.solr.cloud.RecoveryStrategy;
Stopping recovery for zkNodeName=core_node2core=listsC
ERROR - 2014-02-13 13:39:01.183; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/ABC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FABC%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.183; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/ABC/ failed - retrying ...
retries: 11
ERROR - 2014-02-13 13:39:01.271; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/ABC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FABC%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.271; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/ABC/ failed - retrying ...
retries: 4
ERROR - 2014-02-13 13:39:01.340; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.340; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ...
retries: 1
ERROR - 2014-02-13 13:39:01.409; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.409; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ...
retries: 6
ERROR - 2014-02-13 13:39:01.648; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.648; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.648; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ...
retries: 2
ERROR - 2014-02-13 13:39:01.648; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ...
retries: 2
ERROR - 2014-02-13 13:39:01.663; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.663; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ...
retries: 13
ERROR - 2014-02-13 13:39:01.687; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/ABC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FABC%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.688; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/ABC/ failed - retrying ...
retries: 12
ERROR - 2014-02-13 13:39:01.776; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/ABC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FABC%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.776; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/ABC/ failed - retrying ...
retries: 5
ERROR - 2014-02-13 13:39:01.845; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.845; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ...
retries: 2
ERROR - 2014-02-13 13:39:01.913; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.914; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ...
retries: 7
ERROR - 2014-02-13 13:39:02.153; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:02.153; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:02.153; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ...
retries: 3
ERROR - 2014-02-13 13:39:02.153; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ...
retries: 3
ERROR - 2014-02-13 13:39:02.167; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
{quote}
I have saved the full log files, but they're over 4MB each.
was (Author: josephduchesne):
Here are the logs from replica 1:
{quote}
ERROR - 2014-02-13 13:39:01.159; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ...
retries: 12
ERROR - 2014-02-13 13:39:01.175; org.apache.solr.common.SolrException; Error
while trying to recover.
core=listsC:org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
We are not the leader
at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:495)
at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)
at
org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:224)
at
org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:371)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:247)
ERROR - 2014-02-13 13:39:01.175; org.apache.solr.cloud.RecoveryStrategy;
Recovery failed - trying again... (0) core=listsC
WARN - 2014-02-13 13:39:01.180; org.apache.solr.cloud.RecoveryStrategy;
Stopping recovery for zkNodeName=core_node2core=listsC
ERROR - 2014-02-13 13:39:01.183; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/PMG/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FPMG%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.183; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/PMG/ failed - retrying ...
retries: 11
ERROR - 2014-02-13 13:39:01.271; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/PMG/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FPMG%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.271; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/PMG/ failed - retrying ...
retries: 4
ERROR - 2014-02-13 13:39:01.340; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.340; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ...
retries: 1
ERROR - 2014-02-13 13:39:01.409; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.409; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ...
retries: 6
ERROR - 2014-02-13 13:39:01.648; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.648; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.648; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ...
retries: 2
ERROR - 2014-02-13 13:39:01.648; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ...
retries: 2
ERROR - 2014-02-13 13:39:01.663; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.663; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ...
retries: 13
ERROR - 2014-02-13 13:39:01.687; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/PMG/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FPMG%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.688; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/PMG/ failed - retrying ...
retries: 12
ERROR - 2014-02-13 13:39:01.776; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/PMG/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FPMG%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.776; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/PMG/ failed - retrying ...
retries: 5
ERROR - 2014-02-13 13:39:01.845; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.845; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ...
retries: 2
ERROR - 2014-02-13 13:39:01.913; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.914; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ...
retries: 7
ERROR - 2014-02-13 13:39:02.153; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:02.153; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
{quote}
And replica 2:
{quote}
ERROR - 2014-02-13 13:39:01.143; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ...
retries: 1
ERROR - 2014-02-13 13:39:01.158; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.159; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ...
retries: 12
ERROR - 2014-02-13 13:39:01.175; org.apache.solr.common.SolrException; Error
while trying to recover.
core=listsC:org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
We are not the leader
at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:495)
at
org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:199)
at
org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:224)
at
org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:371)
at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:247)
ERROR - 2014-02-13 13:39:01.175; org.apache.solr.cloud.RecoveryStrategy;
Recovery failed - trying again... (0) core=listsC
WARN - 2014-02-13 13:39:01.180; org.apache.solr.cloud.RecoveryStrategy;
Stopping recovery for zkNodeName=core_node2core=listsC
ERROR - 2014-02-13 13:39:01.183; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/PMG/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FPMG%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.183; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/PMG/ failed - retrying ...
retries: 11
ERROR - 2014-02-13 13:39:01.271; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/PMG/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FPMG%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.271; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/PMG/ failed - retrying ...
retries: 4
ERROR - 2014-02-13 13:39:01.340; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.340; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ...
retries: 1
ERROR - 2014-02-13 13:39:01.409; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.409; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ...
retries: 6
ERROR - 2014-02-13 13:39:01.648; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.648; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.648; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ...
retries: 2
ERROR - 2014-02-13 13:39:01.648; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ...
retries: 2
ERROR - 2014-02-13 13:39:01.663; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.663; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ...
retries: 13
ERROR - 2014-02-13 13:39:01.687; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/PMG/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FPMG%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.688; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/PMG/ failed - retrying ...
retries: 12
ERROR - 2014-02-13 13:39:01.776; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/PMG/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FPMG%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.776; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/PMG/ failed - retrying ...
retries: 5
ERROR - 2014-02-13 13:39:01.845; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.845; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ...
retries: 2
ERROR - 2014-02-13 13:39:01.913; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:01.914; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ...
retries: 7
ERROR - 2014-02-13 13:39:02.153; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:02.153; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
request:
http://10.0.2.48:8983/solr/listsC/update?update.distrib=TOLEADER&distrib.from=http%3A%2F%2F10.0.0.156%3A8983%2Fsolr%2FlistsC%2F&wt=javabin&version=2
at
org.apache.solr.client.solrj.impl.ConcurrentUpdateSolrServer$Runner.run(ConcurrentUpdateSolrServer.java:240)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
ERROR - 2014-02-13 13:39:02.153; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ...
retries: 3
ERROR - 2014-02-13 13:39:02.153; org.apache.solr.common.SolrException;
forwarding update to http://10.0.2.48:8983/solr/listsC/ failed - retrying ...
retries: 3
ERROR - 2014-02-13 13:39:02.167; org.apache.solr.update.StreamingSolrServers$1;
error
org.apache.solr.common.SolrException: Service Unavailable
{quote}
I have saved the full log files, but they're over 4MB each.
> Two node, one shard solr instance intermittently going offline
> ---------------------------------------------------------------
>
> Key: SOLR-5724
> URL: https://issues.apache.org/jira/browse/SOLR-5724
> Project: Solr
> Issue Type: Bug
> Affects Versions: 4.6.1
> Environment: Ubuntu 12.04.3 LTS, 64 bit, java version "1.6.0_45"
> Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
> Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)
> Reporter: Joseph Duchesne
>
> One server is stuck in state "recovering" while the other is stuck in state
> "down". After waiting 45 minutes or so for the cluster to recover, the
> statuses were the same.
> Log messages on the "recovering" server: (Just the individual errors for
> brevity, I can provide full stack traces if that is helpful)
> {quote}
> We are not the leader
> ClusterState says we are the leader, but locally we don't think so
> cancelElection did not find election node to remove
> We are not the leader
> No registered leader was found, collection:listsC slice:shard1
> No registered leader was found, collection:listsC slice:shard1
> {quote}
> On the "down" server at the same timeframe:
> {quote}
> org.apache.solr.common.SolrException; forwarding update to
> http://10.0.2.48:8983/solr/listsC/ failed - retrying ... retries: 3
> org.apache.solr.update.StreamingSolrServers$1; error
> Error while trying to recover.
> core=listsC:org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException:
> We are not the leader
> Recovery failed - trying again... (0) core=listsC
> Stopping recovery for zkNodeName=core_node2core=listsC
> org.apache.solr.update.StreamingSolrServers$1; error
> org.apache.solr.common.SolrException: Service Unavailable
> {quote}
> I am not sure what is causing this, however it has happened a 3 times in the
> past week. If there are any additional logs I can provide, or if there is
> anything I can do to try to figure this out myself I will gladly try to help.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
