Re: SolrCloud(5x) - Errors while recovering
Mark Miller-3 wrote FYI, I've fixed this 5x issue a few days ago. - Mark Yep, after the patch, it is not occuring anymore, thank you - Zeki ama calismiyor... Calissa yapar... -- View this message in context: http://lucene.472066.n3.nabble.com/SolrCloud-5x-Errors-while-recovering-tp4022542p4023858.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: SolrCloud(5x) - Errors while recovering
FYI, I've fixed this 5x issue a few days ago. - Mark On Nov 27, 2012, at 10:57 AM, Mark Miller markrmil...@gmail.com wrote: Someone else has been seeing this on 5x as well - their must be a bug in the new file handling code (which is why it's still baking in 5x and not on 4x yet). I tried to trigger it in tests a while back, but had no look in the brief time I had. I'll try some manual tests when I get chance, as well as a little code review. Something is off. - Mark On Nov 26, 2012, at 10:58 PM, deniz denizdurmu...@gmail.com wrote: Here is briefly what is happening: I have a simple SolrCloud environment for test purposes, running with a zookeeper ensemble, not the ones embedded in Solr. I have 3 instances in the cloud, all of them are using RAMDirectory (which is enabled by new Solr release to use with cloud) After running zookeepers and connecting my solrs to them, the cloud is up without any errors or problems. Then I have started indexing (which is much slower than a single instance, i will open a topic about it too) and everything is okay once again, all of the nodes get the sync'ed data from the leader node. After that I have killed one Solr instance. then I have restarted it and in the logs it keeps showing me these errors: SEVERE: Error while trying to recover:org.apache.solr.common.SolrException: Server at http://myhost:8995/solr/mycore returned non ok status:500, message:Server Error at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:372) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181) at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117) at org.apache.solr.cloud.RecoveryStrategy.commitOnLeader(RecoveryStrategy.java:182) at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:134) at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:407) at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:222) . . . . . Nov 27, 2012 11:49:04 AM org.apache.solr.handler.SnapPuller$DirectoryFileFetcher fetchPackets WARNING: Error in fetching packets java.io.EOFException at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:151) at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:144) at org.apache.solr.handler.SnapPuller$DirectoryFileFetcher.fetchPackets(SnapPuller.java:1143) at org.apache.solr.handler.SnapPuller$DirectoryFileFetcher.fetchFile(SnapPuller.java:1107) at org.apache.solr.handler.SnapPuller.downloadIndexFiles(SnapPuller.java:716) at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:387) at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:273) at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:152) at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:407) at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:222) . . . . . SEVERE: SnapPull failed :org.apache.solr.common.SolrException: Unable to download _41y.fdt completely. Downloaded 3145728!=3243906 at org.apache.solr.handler.SnapPuller$DirectoryFileFetcher.cleanup(SnapPuller.java:1237) at org.apache.solr.handler.SnapPuller$DirectoryFileFetcher.fetchFile(SnapPuller.java:1118) at org.apache.solr.handler.SnapPuller.downloadIndexFiles(SnapPuller.java:716) at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:387) at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:273) at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:152) at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:407) at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:222) SEVERE: Error while trying to recover:org.apache.solr.common.SolrException: Replication for recovery failed. at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:155) at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:407) at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:222) can anyone explain why i am getting this error? - Zeki ama calismiyor... Calissa yapar... -- View this message in context: http://lucene.472066.n3.nabble.com/SolrCloud-5x-Errors-while-recovering-tp4022542.html Sent from the Solr - User mailing list archive at Nabble.com.
RE: SolrCloud(5x) - Errors while recovering
Seems you got this issue: https://issues.apache.org/jira/browse/SOLR-4032 -Original message- From:deniz denizdurmu...@gmail.com Sent: Tue 27-Nov-2012 05:04 To: solr-user@lucene.apache.org Subject: SolrCloud(5x) - Errors while recovering Here is briefly what is happening: I have a simple SolrCloud environment for test purposes, running with a zookeeper ensemble, not the ones embedded in Solr. I have 3 instances in the cloud, all of them are using RAMDirectory (which is enabled by new Solr release to use with cloud) After running zookeepers and connecting my solrs to them, the cloud is up without any errors or problems. Then I have started indexing (which is much slower than a single instance, i will open a topic about it too) and everything is okay once again, all of the nodes get the sync'ed data from the leader node. After that I have killed one Solr instance. then I have restarted it and in the logs it keeps showing me these errors: SEVERE: Error while trying to recover:org.apache.solr.common.SolrException: Server at http://myhost:8995/solr/mycore returned non ok status:500, message:Server Error at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:372) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181) at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117) at org.apache.solr.cloud.RecoveryStrategy.commitOnLeader(RecoveryStrategy.java:182) at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:134) at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:407) at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:222) . . . . . Nov 27, 2012 11:49:04 AM org.apache.solr.handler.SnapPuller$DirectoryFileFetcher fetchPackets WARNING: Error in fetching packets java.io.EOFException at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:151) at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:144) at org.apache.solr.handler.SnapPuller$DirectoryFileFetcher.fetchPackets(SnapPuller.java:1143) at org.apache.solr.handler.SnapPuller$DirectoryFileFetcher.fetchFile(SnapPuller.java:1107) at org.apache.solr.handler.SnapPuller.downloadIndexFiles(SnapPuller.java:716) at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:387) at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:273) at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:152) at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:407) at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:222) . . . . . SEVERE: SnapPull failed :org.apache.solr.common.SolrException: Unable to download _41y.fdt completely. Downloaded 3145728!=3243906 at org.apache.solr.handler.SnapPuller$DirectoryFileFetcher.cleanup(SnapPuller.java:1237) at org.apache.solr.handler.SnapPuller$DirectoryFileFetcher.fetchFile(SnapPuller.java:1118) at org.apache.solr.handler.SnapPuller.downloadIndexFiles(SnapPuller.java:716) at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:387) at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:273) at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:152) at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:407) at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:222) SEVERE: Error while trying to recover:org.apache.solr.common.SolrException: Replication for recovery failed. at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:155) at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:407) at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:222) can anyone explain why i am getting this error? - Zeki ama calismiyor... Calissa yapar... -- View this message in context: http://lucene.472066.n3.nabble.com/SolrCloud-5x-Errors-while-recovering-tp4022542.html Sent from the Solr - User mailing list archive at Nabble.com.
RE: SolrCloud(5x) - Errors while recovering
Markus Jelsma-2 wrote Seems you got this issue: https://issues.apache.org/jira/browse/SOLR-4032 thank you for the heads up and a surprising thing about my error.. when i use smaller size of documents, i do not get any errors at all... I dont know why but I have just tried to index only 12K docs, with few fields, with the same configuration, and after a solr node is restarted, there is no errors at all and i got the sync'ed index with the cloud for that node... there is nobody using solrcloud on their prod envs or with too large datasets? or they are using one that they have customized for their own needs? - Zeki ama calismiyor... Calissa yapar... -- View this message in context: http://lucene.472066.n3.nabble.com/SolrCloud-5x-Errors-while-recovering-tp4022542p4022564.html Sent from the Solr - User mailing list archive at Nabble.com.
RE: SolrCloud(5x) - Errors while recovering
another update having 300K docs causes the same error... I think there is something going on with the size of the index stored... after some point replication fails... any ideas how to bypass this? - Zeki ama calismiyor... Calissa yapar... -- View this message in context: http://lucene.472066.n3.nabble.com/SolrCloud-5x-Errors-while-recovering-tp4022542p4022570.html Sent from the Solr - User mailing list archive at Nabble.com.
RE: SolrCloud(5x) - Errors while recovering
It only seems to happen if a node dies while indexing. -Original message- From:deniz denizdurmu...@gmail.com Sent: Tue 27-Nov-2012 10:34 To: solr-user@lucene.apache.org Subject: RE: SolrCloud(5x) - Errors while recovering another update having 300K docs causes the same error... I think there is something going on with the size of the index stored... after some point replication fails... any ideas how to bypass this? - Zeki ama calismiyor... Calissa yapar... -- View this message in context: http://lucene.472066.n3.nabble.com/SolrCloud-5x-Errors-while-recovering-tp4022542p4022570.html Sent from the Solr - User mailing list archive at Nabble.com.
RE: SolrCloud(5x) - Errors while recovering
i have that issue with some larger size of indexes only... 12 - 14K docs are working totally okay even after a node dies and then starts again but if index is bigger, somehow i keep getting the lines above - Zeki ama calismiyor... Calissa yapar... -- View this message in context: http://lucene.472066.n3.nabble.com/SolrCloud-5x-Errors-while-recovering-tp4022542p4022610.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: SolrCloud(5x) - Errors while recovering
Someone else has been seeing this on 5x as well - their must be a bug in the new file handling code (which is why it's still baking in 5x and not on 4x yet). I tried to trigger it in tests a while back, but had no look in the brief time I had. I'll try some manual tests when I get chance, as well as a little code review. Something is off. - Mark On Nov 26, 2012, at 10:58 PM, deniz denizdurmu...@gmail.com wrote: Here is briefly what is happening: I have a simple SolrCloud environment for test purposes, running with a zookeeper ensemble, not the ones embedded in Solr. I have 3 instances in the cloud, all of them are using RAMDirectory (which is enabled by new Solr release to use with cloud) After running zookeepers and connecting my solrs to them, the cloud is up without any errors or problems. Then I have started indexing (which is much slower than a single instance, i will open a topic about it too) and everything is okay once again, all of the nodes get the sync'ed data from the leader node. After that I have killed one Solr instance. then I have restarted it and in the logs it keeps showing me these errors: SEVERE: Error while trying to recover:org.apache.solr.common.SolrException: Server at http://myhost:8995/solr/mycore returned non ok status:500, message:Server Error at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:372) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181) at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117) at org.apache.solr.cloud.RecoveryStrategy.commitOnLeader(RecoveryStrategy.java:182) at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:134) at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:407) at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:222) . . . . . Nov 27, 2012 11:49:04 AM org.apache.solr.handler.SnapPuller$DirectoryFileFetcher fetchPackets WARNING: Error in fetching packets java.io.EOFException at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:151) at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:144) at org.apache.solr.handler.SnapPuller$DirectoryFileFetcher.fetchPackets(SnapPuller.java:1143) at org.apache.solr.handler.SnapPuller$DirectoryFileFetcher.fetchFile(SnapPuller.java:1107) at org.apache.solr.handler.SnapPuller.downloadIndexFiles(SnapPuller.java:716) at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:387) at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:273) at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:152) at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:407) at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:222) . . . . . SEVERE: SnapPull failed :org.apache.solr.common.SolrException: Unable to download _41y.fdt completely. Downloaded 3145728!=3243906 at org.apache.solr.handler.SnapPuller$DirectoryFileFetcher.cleanup(SnapPuller.java:1237) at org.apache.solr.handler.SnapPuller$DirectoryFileFetcher.fetchFile(SnapPuller.java:1118) at org.apache.solr.handler.SnapPuller.downloadIndexFiles(SnapPuller.java:716) at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:387) at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:273) at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:152) at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:407) at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:222) SEVERE: Error while trying to recover:org.apache.solr.common.SolrException: Replication for recovery failed. at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:155) at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:407) at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:222) can anyone explain why i am getting this error? - Zeki ama calismiyor... Calissa yapar... -- View this message in context: http://lucene.472066.n3.nabble.com/SolrCloud-5x-Errors-while-recovering-tp4022542.html Sent from the Solr - User mailing list archive at Nabble.com.
SolrCloud(5x) - Errors while recovering
Here is briefly what is happening: I have a simple SolrCloud environment for test purposes, running with a zookeeper ensemble, not the ones embedded in Solr. I have 3 instances in the cloud, all of them are using RAMDirectory (which is enabled by new Solr release to use with cloud) After running zookeepers and connecting my solrs to them, the cloud is up without any errors or problems. Then I have started indexing (which is much slower than a single instance, i will open a topic about it too) and everything is okay once again, all of the nodes get the sync'ed data from the leader node. After that I have killed one Solr instance. then I have restarted it and in the logs it keeps showing me these errors: SEVERE: Error while trying to recover:org.apache.solr.common.SolrException: Server at http://myhost:8995/solr/mycore returned non ok status:500, message:Server Error at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:372) at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:181) at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:117) at org.apache.solr.cloud.RecoveryStrategy.commitOnLeader(RecoveryStrategy.java:182) at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:134) at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:407) at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:222) . . . . . Nov 27, 2012 11:49:04 AM org.apache.solr.handler.SnapPuller$DirectoryFileFetcher fetchPackets WARNING: Error in fetching packets java.io.EOFException at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:151) at org.apache.solr.common.util.FastInputStream.readFully(FastInputStream.java:144) at org.apache.solr.handler.SnapPuller$DirectoryFileFetcher.fetchPackets(SnapPuller.java:1143) at org.apache.solr.handler.SnapPuller$DirectoryFileFetcher.fetchFile(SnapPuller.java:1107) at org.apache.solr.handler.SnapPuller.downloadIndexFiles(SnapPuller.java:716) at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:387) at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:273) at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:152) at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:407) at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:222) . . . . . SEVERE: SnapPull failed :org.apache.solr.common.SolrException: Unable to download _41y.fdt completely. Downloaded 3145728!=3243906 at org.apache.solr.handler.SnapPuller$DirectoryFileFetcher.cleanup(SnapPuller.java:1237) at org.apache.solr.handler.SnapPuller$DirectoryFileFetcher.fetchFile(SnapPuller.java:1118) at org.apache.solr.handler.SnapPuller.downloadIndexFiles(SnapPuller.java:716) at org.apache.solr.handler.SnapPuller.fetchLatestIndex(SnapPuller.java:387) at org.apache.solr.handler.ReplicationHandler.doFetch(ReplicationHandler.java:273) at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:152) at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:407) at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:222) SEVERE: Error while trying to recover:org.apache.solr.common.SolrException: Replication for recovery failed. at org.apache.solr.cloud.RecoveryStrategy.replicate(RecoveryStrategy.java:155) at org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:407) at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:222) can anyone explain why i am getting this error? - Zeki ama calismiyor... Calissa yapar... -- View this message in context: http://lucene.472066.n3.nabble.com/SolrCloud-5x-Errors-while-recovering-tp4022542.html Sent from the Solr - User mailing list archive at Nabble.com.
