Again, just finished reindexing, server utilization was about 5-10%, I started index optimization. As result I now lost (again) entire index, got a lot of errors, they are appear so fast and contain 0 useful information.
<http://lucene.472066.n3.nabble.com/file/n4062434/Screenshot_546.png> <http://lucene.472066.n3.nabble.com/file/n4062434/Screenshot_547.png> You can see that server is not loaded at all, and load was the same when I started the optimization process. BTW, it seems like an infinite loop, the picture does not change, replica is down, shard in recovering. In the shard log I see: org.apache.solr.common.SolrException: No registered leader was found, collection:crm-prod slice:shard1 And the same in replica + sometimes: Error getting leader from zk org.apache.solr.common.SolrException: No registered leader was found, collection:crm-test slice:shard1 at org.apache.solr.common.cloud.ZkStateReader.getLeaderRetry(ZkStateReader.java:484) at org.apache.solr.common.cloud.ZkStateReader.getLeaderUrl(ZkStateReader.java:458) at org.apache.solr.cloud.ZkController.getLeader(ZkController.java:843) at org.apache.solr.cloud.ZkController.register(ZkController.java:776) at org.apache.solr.cloud.ZkController$1.command(ZkController.java:216) at org.apache.solr.common.cloud.ConnectionManager$1.update(ConnectionManager.java:117) at org.apache.solr.common.cloud.DefaultConnectionStrategy.reconnect(DefaultConnectionStrategy.java:46) at org.apache.solr.common.cloud.ConnectionManager.process(ConnectionManager.java:91) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495) in zookeeper log: 2013-05-10 06:12:21,788 [myid:3] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2183:NIOServerCnxn@349] - caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x33e89eb26e40006, likely client has closed socket at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) at java.lang.Thread.run(Thread.java:679) 2013-05-10 06:13:16,346 [myid:3] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2183:NIOServerCnxn@349] - caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x23e89eb1ab20004, likely client has closed socket at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) at java.lang.Thread.run(Thread.java:679) 2013-05-10 07:57:50,677 [myid:3] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2183:NIOServerCnxn@349] - caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x33e89eb26e40004, likely client has closed socket at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) at java.lang.Thread.run(Thread.java:679) 2013-05-10 07:58:05,956 [myid:3] - ERROR [CommitProcessor:3:NIOServerCnxn@180] - Unexpected Exception: java.nio.channels.CancelledKeyException at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73) at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77) at org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:153) at org.apache.zookeeper.server.NIOServerCnxn.sendResponse(NIOServerCnxn.java:1076) at org.apache.zookeeper.server.NIOServerCnxn.process(NIOServerCnxn.java:1113) at org.apache.zookeeper.server.DataTree.setWatches(DataTree.java:1293) at org.apache.zookeeper.server.ZKDatabase.setWatches(ZKDatabase.java:384) at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:304) at org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:74) 2013-05-10 07:58:05,957 [myid:3] - ERROR [CommitProcessor:3:NIOServerCnxn@180] - Unexpected Exception: java.nio.channels.CancelledKeyException at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:73) at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:77) at org.apache.zookeeper.server.NIOServerCnxn.s2013-05-10 11:07:55,805 [myid:2] - WARN [SyncThread:2:FileTxnLog@321] - fsync-ing the write ahead log in SyncThread:2 took 2302ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide 2013-05-10 11:10:31,354 [myid:2] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2182:NIOServerCnxn@349] - caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x33e89eb26e40008, likely client has closed socket at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) at java.lang.Thread.run(Thread.java:679) 2013-05-10 11:13:28,610 [myid:2] - ERROR [LearnerHandler-/127.0.0.1:53815:LearnerHandler@562] - Unexpected exception causing shutdown while sock still open java.net.SocketTimeoutException: Read timed out at java.net.SocketInputStream.socketRead0(Native Method) at java.net.SocketInputStream.read(SocketInputStream.java:146) at java.io.BufferedInputStream.fill(BufferedInputStream.java:235) at java.io.BufferedInputStream.read(BufferedInputStream.java:254) at java.io.DataInputStream.readInt(DataInputStream.java:387) at org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63) at org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:83) at org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:108) at org.apache.zookeeper.server.quorum.LearnerHandler.run(LearnerHandler.java:476) 2013-05-10 11:13:28,611 [myid:2] - WARN [LearnerHandler-/127.0.0.1:53815:LearnerHandler@575] - ******* GOODBYE /127.0.0.1:53815 ******** 2013-05-10 11:23:28,186 [myid:2] - WARN [SyncThread:2:FileTxnLog@321] - fsync-ing the write ahead log in SyncThread:2 took 1079ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide ocket at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) at java.lang.Thread.run(Thread.java:679) 2013-05-10 10:59:06,943 [myid:3] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2183:NIOServerCnxn@349] - caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x23e89eb1ab20004, likely client has closed socket at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) at java.lang.Thread.run(Thread.java:679) 2013-05-10 11:07:55,805 [myid:3] - WARN [SyncThread:3:FileTxnLog@321] - fsync-ing the write ahead log in SyncThread:3 took 2625ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide 2013-05-10 11:08:31,101 [myid:3] - WARN [SyncThread:3:FileTxnLog@321] - fsync-ing the write ahead log in SyncThread:3 took 3102ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide 2013-05-10 11:10:45,152 [myid:3] - WARN [SyncThread:3:FileTxnLog@321] - fsync-ing the write ahead log in SyncThread:3 took 1587ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide 2013-05-09 08:32:34,358 [myid:1] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:Follower@118] - Got zxid 0x300000001 expected 0x1 2013-05-10 08:49:35,497 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@349] - caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x33e89eb26e40008, likely client has closed socket at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) at java.lang.Thread.run(Thread.java:679) 2013-05-10 09:58:31,378 [myid:1] - WARN [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181:NIOServerCnxn@349] - caught end of stream exception EndOfStreamException: Unable to read additional data from client sessionid 0x13e89eb1aa90005, likely client has closed socket at org.apache.zookeeper.server.NIOServerCnxn.doIO(NIOServerCnxn.java:220) at org.apache.zookeeper.server.NIOServerCnxnFactory.run(NIOServerCnxnFactory.java:208) at java.lang.Thread.run(Thread.java:679) 2013-05-10 11:08:31,101 [myid:1] - WARN [SyncThread:1:FileTxnLog@321] - fsync-ing the write ahead log in SyncThread:1 took 1548ms which will adversely effect operation latency. See the ZooKeeper troubleshooting guide 2013-05-10 11:13:29,793 [myid:1] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:Follower@89] - Exception when following the leader java.net.SocketException: Broken pipe at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109) at java.net.SocketOutputStream.write(SocketOutputStream.java:153) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) at org.apache.zookeeper.server.quorum.Learner.writePacket(Learner.java:138) at org.apache.zookeeper.server.quorum.Learner.ping(Learner.java:465) at org.apache.zookeeper.server.quorum.Follower.processPacket(Follower.java:112) at org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:86) at org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:740) 2013-05-10 11:13:29,794 [myid:1] - ERROR [FollowerRequestProcessor:1:FollowerRequestProcessor@93] - Unexpected exception causing exit java.net.SocketException: Broken pipe at java.net.SocketOutputStream.socketWrite0(Native Method) at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109) at java.net.SocketOutputStream.write(SocketOutputStream.java:153) at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82) at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140) at org.apache.zookeeper.server.quorum.Learner.writePacket(Learner.java:138) at org.apache.zookeeper.server.quorum.Learner.request(Learner.java:187) at org.apache.zookeeper.server.quorum.FollowerRequestProcessor.run(FollowerRequestProcessor.java:88) 2013-05-10 11:13:30,076 [myid:1] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:Learner@373] - Got zxid 0x300001697 expected 0x1 2013-05-10 11:13:30,648 [myid:1] - WARN [QuorumPeer[myid=1]/0:0:0:0:0:0:0:0:2181:Follower@118] - Got zxid 0x30000169d expected 0x1 Load wasn't high at all, there were maximum 10 requests per minute. Best, Alex -- View this message in context: http://lucene.472066.n3.nabble.com/ColrCloud-IOException-occured-when-talking-to-server-at-tp4061831p4062434.html Sent from the Solr - User mailing list archive at Nabble.com.
