Hello, Running solr cloud with 2 collections 5 shards and 3 replicas for each collection, 5 zookeeper instance. solr-4.6.0 apache-tomcat-7.0.39 zookeeper-3.4.5 jre1.7.0_21
When I try to restart a solr servers in my solr cloud I am receiving this errors : 1861449 [localhost-startStop-1-EventThread] INFO org.apache.solr.cloud.ShardLeaderElectionContext â Running the leader process for shard shard1 1861451 [localhost-startStop-1-EventThread] INFO org.apache.solr.cloud.ShardLeaderElectionContext â Checking if I should try and be the leader. 1861451 [localhost-startStop-1-EventThread] INFO org.apache.solr.cloud.ShardLeaderElectionContext â My last published State was down, I won't be the leader. 1861451 [localhost-startStop-1-EventThread] INFO org.apache.solr.cloud.ShardLeaderElectionContext â There may be a better leader candidate than us - going back into recovery 1861452 [localhost-startStop-1-EventThread] INFO org.apache.solr.update.DefaultSolrCoreState â Running recovery - first canceling any ongoing recovery 1861452 [localhost-startStop-1-EventThread] WARN org.apache.solr.cloud.RecoveryStrategy â Stopping recovery for zkNodeName=core_node3core=Current1_shard1_replica3 1862223 [RecoveryThread] INFO org.apache.solr.cloud.RecoveryStrategy â Finished recovery process. core=Current1_shard1_replica3 1862223 [RecoveryThread] INFO org.apache.solr.cloud.RecoveryStrategy â Starting recovery process. core=Current1_shard1_replica3 recoveringAfterStartup=false 1862223 [RecoveryThread] ERROR org.apache.solr.update.UpdateLog â Exception reading versions from log java.nio.channels.ClosedChannelException at sun.nio.ch.FileChannelImpl.ensureOpen(Unknown Source) at sun.nio.ch.FileChannelImpl.read(Unknown Source) at org.apache.solr.update.ChannelFastInputStream.readWrappedStream(TransactionLog.java:778) at org.apache.solr.common.util.FastInputStream.refill(FastInputStream.java:89) at org.apache.solr.common.util.FastInputStream.readUnsignedByte(FastInputStream.java:71) at org.apache.solr.common.util.FastInputStream.readInt(FastInputStream.java:216) at org.apache.solr.update.TransactionLog$FSReverseReader.<init>(TransactionLog.java:696) at org.apache.solr.update.TransactionLog.getReverseReader(TransactionLog.java:575) at org.apache.solr.update.UpdateLog$RecentUpdates.update(UpdateLog.java:942) at org.apache.solr.update.UpdateLog$RecentUpdates.access$000(UpdateLog.java:885) at org.apache.solr.update.UpdateLog.getRecentUpdates(UpdateLog.java:1042) at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:280) at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:244) 1862223 [RecoveryThread] ERROR org.apache.solr.cloud.RecoveryStrategy â Error while trying to recover. core=Current1_shard1_replica3:org.apache.solr.common.SolrException: Cloud state sti ll says we are leader. at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:354) at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:244) 1862224 [RecoveryThread] ERRwwwwOR org.apache.solr.cloud.RecoveryStrategy â Recovery failed - trying again... (0) core=Current1_shard1_replica3 1862224 [RecoveryThread] INFO org.apache.solr.cloud.RecoveryStrategy â Wait 2.0 seconds before trying to recover again (1) 1862541 [localhost-startStop-1-SendThread(10.0.5.230:2281)] WARN org.apache.zookeeper.ClientCnxn â Session 0x542fd3f2be100e6 for server 10.0.5.230/10.0.5.230:2281, unexpected error, clo sing socket connection and attempting reconnect java.io.IOException: Packet len11106511 is out of range! at org.apache.zookeeper.ClientCnxnSocket.readLength(ClientCnxnSocket.java:112) at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:79) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:355) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068) 1862641 [localhost-startStop-1-EventThread] INFO org.apache.solr.cloud.DistributedQueue â Watcher fired on path: null state: Disconnected type None 1862641 [localhost-startStop-1-EventThread] INFO org.apache.solr.cloud.DistributedQueue â Watcher fired on path: null state: Disconnected type None 1862641 [localhost-startStop-1-EventThread] INFO org.apache.solr.cloud.DistributedQueue â Watcher fired on path: null state: Disconnected type None 1862641 [localhost-startStop-1-EventThread] INFO org.apache.solr.cloud.DistributedQueue â Watcher fired on path: null state: Disconnected type None .............. 1270268 [http-bio-8201-exec-26] INFO org.apache.solr.handler.admin.CoreAdminHandler â Going to wait for coreNodeName: core_node10, state: recovering, checkLive: true, onlyIfLeader: true 1270268 [http-bio-8201-exec-10] INFO org.apache.solr.handler.admin.CoreAdminHandler â Going to wait for coreNodeName: core_node11, state: recovering, checkLive: true, onlyIfLeader: true 1270268 [http-bio-8201-exec-26] ERROR org.apache.solr.core.SolrCore â org.apache.solr.common.SolrException: We are not the leader at org.apache.solr.handler.admin.CoreAdminHandler.handleWaitForStateAction(CoreAdminHandler.java:873) at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:198) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135) at org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:662) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:248) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:947) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1009) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) .............. 1778261 [http-bio-8201-exec-1] INFO org.apache.solr.handler.admin.CoreAdminHandler â Going to wait for coreNodeName: core_node15, state: recovering, checkLive: true, onlyIfLeader: true 1778261 [http-bio-8201-exec-1] ERROR org.apache.solr.core.SolrCore â org.apache.solr.common.SolrException: We are not the leader at org.apache.solr.handler.admin.CoreAdminHandler.handleWaitForStateAction(CoreAdminHandler.java:873) at org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:198) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135) at org.apache.solr.servlet.SolrDispatchFilter.handleAdminRequest(SolrDispatchFilter.java:662) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:248) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:947) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1009) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) 1778261 [http-bio-8201-exec-1] INFO org.apache.solr.servlet.SolrDispatchFilter â [admin] webapp=null path=/admin/cores params={coreNodeName=core_node15&state=recovering&nodeName=10.0.5. 230:8201_solr&action=PREPRECOVERY&checkLive=true&core=Current1_shard5_replica1&wt=javabin&onlyIfLeader=true&version=2} status=400 QTime=0 1780238 [http-bio-8201-exec-29] ERROR org.apache.solr.core.SolrCore â org.apache.solr.common.SolrException: no servers hosting shard: at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:148) at org.apache.solr.handler.component.HttpShardHandler$1.call(HttpShardHandler.java:118) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) at java.util.concurrent.FutureTask$Sync.innerRun(Unknown Source) at java.util.concurrent.FutureTask.run(Unknown Source) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) ............. 2260241 [http-bio-8201-exec-32] ERROR org.apache.solr.core.SolrCore â org.apache.solr.common.SolrException: Error opening new searcher at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1477) at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1589) at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1346) at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1281) at org.apache.solr.request.SolrQueryRequestBase.getSearcher(SolrQueryRequestBase.java:96) at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:237) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:208) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1859) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:710) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:413) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:197) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:947) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1009) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589) at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Caused by: org.apache.lucene.store.AlreadyClosedException: Already closed at org.apache.solr.core.CachingDirectoryFactory.get(CachingDirectoryFactory.java:340) at org.apache.solr.core.SolrCore.getNewIndexDir(SolrCore.java:251) at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1372) ... 26 more -- View this message in context: http://lucene.472066.n3.nabble.com/Error-when-restarting-solr-servers-tp4114461.html Sent from the Solr - User mailing list archive at Nabble.com.