fetchindex not working.
I'm trying to get solr to fetch an index from a master server. This does not work with our configuration, and throws an error that pollInterval cannot be null. I thought to get fetchindex working, that pollInterval should not be in the solrconfig.xml file? I have tried adding pollInterval to the configuraiton, but it still will not work. Does anybody have any experience with this issue? # REQUEST # http://solr:8080/solr/core1/replication?command=fetchindexmasterUrl=http://10.10.0.61:8080/solr/core1/replicationcommand=fetchindex # SOLRCONFIG.XML # !-- Slave config -- requestHandler name=/replication class=solr.ReplicationHandler lst name=slave str name=masterUrlhttp://127.0.0.1:8080/solr/${solr.core.name}/replication/str /lst /requestHandler # CATALINA.OUT # INFO: [blogs_2009_53_1] webapp=/solr path=/replication params={masterUrl=http://10.10.0.61:8080/solr/core1/replicationcommand=fetchindex} status=0 QTime=0 Feb 2, 2010 10:47:51 AM org.apache.solr.handler.SnapPuller init INFO: No value set for 'pollInterval'. Timer Task not started. Feb 2, 2010 10:47:51 AM org.apache.solr.core.SolrCore execute Jeremy Carroll Sr. Network Engineer
RE: fetchindex not working.
I do not see a stack trace, but the replication does not happen. I do not see a request in the logs of the master server that it even tried to pull the index. From: Shalin Shekhar Mangar [shalinman...@gmail.com] Sent: Tuesday, February 02, 2010 11:36 AM To: solr-user@lucene.apache.org Subject: Re: fetchindex not working. On Tue, Feb 2, 2010 at 10:26 PM, Jeremy Carroll jeremy.carr...@networkedinsights.com wrote: I'm trying to get solr to fetch an index from a master server. This does not work with our configuration, and throws an error that pollInterval cannot be null. I thought to get fetchindex working, that pollInterval should not be in the solrconfig.xml file? I have tried adding pollInterval to the configuraiton, but it still will not work. Does anybody have any experience with this issue? # REQUEST # http://solr:8080/solr/core1/replication?command=fetchindexmasterUrl=http://10.10.0.61:8080/solr/core1/replicationcommand=fetchindex # SOLRCONFIG.XML # !-- Slave config -- requestHandler name=/replication class=solr.ReplicationHandler lst name=slave str name=masterUrl http://127.0.0.1:8080/solr/${solr.core.name}/replication/str /lst /requestHandler # CATALINA.OUT # INFO: [blogs_2009_53_1] webapp=/solr path=/replication params={masterUrl= http://10.10.0.61:8080/solr/core1/replicationcommand=fetchindex} status=0 QTime=0 Feb 2, 2010 10:47:51 AM org.apache.solr.handler.SnapPuller init INFO: No value set for 'pollInterval'. Timer Task not started. Feb 2, 2010 10:47:51 AM org.apache.solr.core.SolrCore execute That is just an INFO level log message. Have you seen an exception saying that pollInterval cannot be null? If yes, can you please paste the stack trace. -- Regards, Shalin Shekhar Mangar.
RE: fetchindex not working.
I apologize. There is not a fetchindex command on the master server. But I do see this in the logs. Replication does not take place at all. I'm trying to figure out why it's not sending the index from the master to the slave after sending the fetchindex command. ### SLAVE CATALINA.OUT ### INFO: [core1] webapp=/solr path=/replication params={masterUrl=http://10.10.0.61:8080/solr/core1/replicationcommand=fetchindex} status=0 QTime=0 Feb 2, 2010 11:49:43 AM org.apache.solr.handler.SnapPuller init INFO: No value set for 'pollInterval'. Timer Task not started. ### MASTER CATALINA.OUT ### INFO: [core1] webapp=/solr path=/replication params={command=indexversionwt=javabin} status=0 QTime=0 From: Shalin Shekhar Mangar [shalinman...@gmail.com] Sent: Tuesday, February 02, 2010 11:36 AM To: solr-user@lucene.apache.org Subject: Re: fetchindex not working. On Tue, Feb 2, 2010 at 10:26 PM, Jeremy Carroll jeremy.carr...@networkedinsights.com wrote: I'm trying to get solr to fetch an index from a master server. This does not work with our configuration, and throws an error that pollInterval cannot be null. I thought to get fetchindex working, that pollInterval should not be in the solrconfig.xml file? I have tried adding pollInterval to the configuraiton, but it still will not work. Does anybody have any experience with this issue? # REQUEST # http://solr:8080/solr/core1/replication?command=fetchindexmasterUrl=http://10.10.0.61:8080/solr/core1/replicationcommand=fetchindex # SOLRCONFIG.XML # !-- Slave config -- requestHandler name=/replication class=solr.ReplicationHandler lst name=slave str name=masterUrl http://127.0.0.1:8080/solr/${solr.core.name}/replication/str /lst /requestHandler # CATALINA.OUT # INFO: [blogs_2009_53_1] webapp=/solr path=/replication params={masterUrl= http://10.10.0.61:8080/solr/core1/replicationcommand=fetchindex} status=0 QTime=0 Feb 2, 2010 10:47:51 AM org.apache.solr.handler.SnapPuller init INFO: No value set for 'pollInterval'. Timer Task not started. Feb 2, 2010 10:47:51 AM org.apache.solr.core.SolrCore execute That is just an INFO level log message. Have you seen an exception saying that pollInterval cannot be null? If yes, can you please paste the stack trace. -- Regards, Shalin Shekhar Mangar.
RE: fetchindex not working.
Here are the debugging statements. Replication is not happening from the slave to the master. I see that the slave is issuing an indexversion request, but does nothing with it. ### SLAVE CATALINA.OUT ### INFO: [core1] webapp=/solr path=/replication params={command=fetchindexmasterUrl=http://10.10.0.61:8080/solr/core1/replication} status=0 QTime=0 Feb 2, 2010 1:14:04 PM org.apache.solr.handler.SnapPuller init INFO: No value set for 'pollInterval'. Timer Task not started. Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.MultiThreadedHttpConnectionManager getConnectionWithTimeout FINE: HttpConnectionManager.getConnection: config = HostConfiguration[host=http://10.10.0.61:8080], timeout = 0 Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool getFreeConnection FINE: Getting free connection, hostConfig=HostConfiguration[host=http://10.10.0.61:8080] Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.HttpConnection closeIfStale FINE: Connection is stale, closing... Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.HttpConnection open FINE: Open connection to 10.10.0.61:8080 Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.Wire wire FINE: POST /solr/core1/replication HTTP/1.1[\r][\n] Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.HttpMethodBase addHostRequestHeader FINE: Adding Host request header Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.HttpMethodBase getContentCharSet FINE: Default charset used: ISO-8859-1 Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.HttpMethodBase getContentCharSet FINE: Default charset used: ISO-8859-1 Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.Wire wire FINE: User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n] Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.Wire wire FINE: Host: 10.10.0.61:8080[\r][\n] Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.Wire wire FINE: Content-Length: 31[\r][\n] Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.Wire wire FINE: Content-Type: application/x-www-form-urlencoded[\r][\n] Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.Wire wire FINE: [\r][\n] Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.Wire wire FINE: command=indexversionwt=javabin Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.methods.EntityEnclosingMethod writeRequestBody FINE: Request body sent Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.Wire wire FINE: HTTP/1.1 200 OK[\r][\n] Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.Wire wire FINE: HTTP/1.1 200 OK[\r][\n] Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.Wire wire FINE: Content-Type: application/octet-stream[\r][\n] Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.Wire wire FINE: Content-Length: 72[\r][\n] Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.Wire wire FINE: Date: Tue, 02 Feb 2010 19:14:04 GMT[\r][\n] Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.Wire wire FINE: Server: Apache[\r][\n] Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.Wire wire FINE: [\r][\n] Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.Wire wire FINE: [0x1][0xa3][0xe0].responseHeader[0xa2][0xe0]status[0x6][0x0][0x0][0x0][0x0][0xe0]%QTime[0x6][0x0][0x0][0x0][0x0][0xe0],indexversion`[0xe0]*generation` Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.HttpMethodBase shouldCloseConnection FINE: Resorting to protocol version default close connection policy Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.HttpMethodBase shouldCloseConnection FINE: Should NOT close connection, using HTTP/1.1 Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.HttpConnection releaseConnection FINE: Releasing connection back to connection manager. Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool freeConnection FINE: Freeing connection, hostConfig=HostConfiguration[host=http://10.10.0.61:8080] Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.util.IdleConnectionHandler add FINE: Adding connection at: 1265138044622 Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool notifyWaitingThread FINE: Notifying no-one, there are no waiting threads ### MASTER CATALINA.OUT ### Feb 2, 2010 1:16:00 PM org.apache.solr.core.SolrCore execute INFO: [core1] webapp=/solr path=/replication params={wt=javabincommand=indexversion} status=0 QTime=0 Feb 2, 2010 1:16:01 PM org.apache.solr.core.Config getVal FINE: solrconfig.xml admin/healthcheck/text()=/var/lib/solr/server-enabled From: Jeremy Carroll [jeremy.carr...@networkedinsights.com] Sent: Tuesday, February 02, 2010 11:47 AM To: solr-user@lucene.apache.org Subject: RE: fetchindex not working. I do not see a stack trace, but the replication does not happen. I do not see a request in the logs of the master server that it even tried to pull the index
RE: fetchindex not working.
Found out my issue. solrconfig.xml on the MASTER servers did not have this statement. str name=replicateAfterstartup/str My solrconfig.xml should have looked like this. !--Master config-- requestHandler name=/replication class=solr.ReplicationHandler lst name=master str name=replicateAftercommit/str str name=replicateAfteroptimize/str str name=replicateAfterstartup/str str name=confFilessolrconfig_slave.xml:solrconfig.xml,schema.xml,stopwords.txt/str /lst /requestHandler After adding this statement, replication worked. From: Jeremy Carroll [jeremy.carr...@networkedinsights.com] Sent: Tuesday, February 02, 2010 1:15 PM To: solr-user@lucene.apache.org Subject: RE: fetchindex not working. Here are the debugging statements. Replication is not happening from the slave to the master. I see that the slave is issuing an indexversion request, but does nothing with it. ### SLAVE CATALINA.OUT ### INFO: [core1] webapp=/solr path=/replication params={command=fetchindexmasterUrl=http://10.10.0.61:8080/solr/core1/replication} status=0 QTime=0 Feb 2, 2010 1:14:04 PM org.apache.solr.handler.SnapPuller init INFO: No value set for 'pollInterval'. Timer Task not started. Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.MultiThreadedHttpConnectionManager getConnectionWithTimeout FINE: HttpConnectionManager.getConnection: config = HostConfiguration[host=http://10.10.0.61:8080], timeout = 0 Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool getFreeConnection FINE: Getting free connection, hostConfig=HostConfiguration[host=http://10.10.0.61:8080] Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.HttpConnection closeIfStale FINE: Connection is stale, closing... Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.HttpConnection open FINE: Open connection to 10.10.0.61:8080 Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.Wire wire FINE: POST /solr/core1/replication HTTP/1.1[\r][\n] Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.HttpMethodBase addHostRequestHeader FINE: Adding Host request header Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.HttpMethodBase getContentCharSet FINE: Default charset used: ISO-8859-1 Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.HttpMethodBase getContentCharSet FINE: Default charset used: ISO-8859-1 Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.Wire wire FINE: User-Agent: Jakarta Commons-HttpClient/3.1[\r][\n] Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.Wire wire FINE: Host: 10.10.0.61:8080[\r][\n] Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.Wire wire FINE: Content-Length: 31[\r][\n] Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.Wire wire FINE: Content-Type: application/x-www-form-urlencoded[\r][\n] Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.Wire wire FINE: [\r][\n] Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.Wire wire FINE: command=indexversionwt=javabin Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.methods.EntityEnclosingMethod writeRequestBody FINE: Request body sent Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.Wire wire FINE: HTTP/1.1 200 OK[\r][\n] Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.Wire wire FINE: HTTP/1.1 200 OK[\r][\n] Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.Wire wire FINE: Content-Type: application/octet-stream[\r][\n] Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.Wire wire FINE: Content-Length: 72[\r][\n] Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.Wire wire FINE: Date: Tue, 02 Feb 2010 19:14:04 GMT[\r][\n] Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.Wire wire FINE: Server: Apache[\r][\n] Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.Wire wire FINE: [\r][\n] Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.Wire wire FINE: [0x1][0xa3][0xe0].responseHeader[0xa2][0xe0]status[0x6][0x0][0x0][0x0][0x0][0xe0]%QTime[0x6][0x0][0x0][0x0][0x0][0xe0],indexversion`[0xe0]*generation` Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.HttpMethodBase shouldCloseConnection FINE: Resorting to protocol version default close connection policy Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.HttpMethodBase shouldCloseConnection FINE: Should NOT close connection, using HTTP/1.1 Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.HttpConnection releaseConnection FINE: Releasing connection back to connection manager. Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool freeConnection FINE: Freeing connection, hostConfig=HostConfiguration[host=http://10.10.0.61:8080] Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.util.IdleConnectionHandler add FINE: Adding connection at: 1265138044622 Feb 2, 2010 1:14:04 PM org.apache.commons.httpclient.MultiThreadedHttpConnectionManager$ConnectionPool notifyWaitingThread FINE
Issues with stale searchers.
I have Solr 1.3 running on Apache Tomcat 5.5.27. I'm running into an issue where searchers are opened up right away when tomcat starts, and never goes away. This is causing read locks on the Lucene index holding open deleted files during merges. This causes our server to run out of disk space in our index. Wondering what is causing this issue as I have been searching for two days without any real answers. Thanks, LSOF Output java 7322 tomcat 70r REG 253,0 2569538 2883610 /opt/solr/data/index/_m5n.cfs (deleted) java 7322 tomcat 71r REG 253,0 2338291 2883609 /opt/solr/data/index/_m5m.cfs (deleted) java 7322 tomcat 72r REG 253,0 13398930 2883608 /opt/solr/data/index/_m5l.cfs (deleted) java 7322 tomcat 73r REG 253,0 2692917 2883598 /opt/solr/data/index/_m5k.cfs (deleted) java 7322 tomcat 74r REG 253,0 32324600 2883592 /opt/solr/data/index/_m5j.cfx (deleted) java 7322 tomcat 75r REG 253,0 6767344 2883603 /opt/solr/data/index/_m5j.cfs (deleted) java 7322 tomcat 76r REG 253,0 32324600 2883592 /opt/solr/data/index/_m5j.cfx (deleted) java 7322 tomcat 77r REG 253,0 15937346 2883600 /opt/solr/data/index/_m5i.cfs (deleted) Stats page on Solr Admin searc...@66952905 main class: org.apache.solr.search.SolrIndexSearcher version:1.0 description:index searcher stats: searcherName : searc...@66952905 main caching : true numDocs : 187169908 maxDoc : 187169908 readerImpl : ReadOnlyMultiSegmentReader readerDir : org.apache.lucene.store.FSDirectory@/opt/solr/data/index indexVersion : 1224609883675 openedAt : Thu Mar 12 17:13:15 CDT 2009 registeredAt : Thu Mar 12 17:13:23 CDT 2009 warmupTime : 0 name: core class: version:1.0 description:SolrCore stats: coreName : startTime : Thu Mar 12 17:13:15 CDT 2009 refCount : 2 aliases : [] name: searcher class: org.apache.solr.search.SolrIndexSearcher version:1.0 description:index searcher stats: searcherName : searc...@66952905 main caching : true numDocs : 187169908 maxDoc : 187169908 readerImpl : ReadOnlyMultiSegmentReader readerDir : org.apache.lucene.store.FSDirectory@/opt/solr/data/index indexVersion : 1224609883675 openedAt : Thu Mar 12 17:13:15 CDT 2009 registeredAt : Thu Mar 12 17:13:23 CDT 2009 warmupTime : 0 Jeremy Carroll Sr. Network Engineer Networked Insights
RE: Issues with stale searchers.
If that's the case it is causing out of disk issues with Solr. We have a 187m document count index which is about ~200Gb in size. Over a period of about a week after optimizations, etc... the open file but deleted count grows very large. Causing the system to not be able to optimize due to lack of disk space. Also new documents that are indexed are not showing up in search results. -Original Message- From: ysee...@gmail.com [mailto:ysee...@gmail.com] On Behalf Of Yonik Seeley Sent: Thursday, March 12, 2009 7:43 PM To: solr-user@lucene.apache.org Subject: Re: Issues with stale searchers. On Thu, Mar 12, 2009 at 6:29 PM, Jeremy Carroll jeremy.carr...@networkedinsights.com wrote: I have Solr 1.3 running on Apache Tomcat 5.5.27. I'm running into an issue where searchers are opened up right away when tomcat starts, and never goes away. This is causing read locks on the Lucene index holding open deleted files during merges. Deleted files being held open can be normal - that's the current IndexSearcher serving requests (even though those files may have been deleted by the IndexWriter already). Looking at your Stats, I only see one Searcher, so things look fine there too. -Yonik http://www.lucidimagination.com This causes our server to run out of disk space in our index. Wondering what is causing this issue as I have been searching for two days without any real answers. Thanks, LSOF Output java 7322 tomcat 70r REG 253,0 2569538 2883610 /opt/solr/data/index/_m5n.cfs (deleted) java 7322 tomcat 71r REG 253,0 2338291 2883609 /opt/solr/data/index/_m5m.cfs (deleted) java 7322 tomcat 72r REG 253,0 13398930 2883608 /opt/solr/data/index/_m5l.cfs (deleted) java 7322 tomcat 73r REG 253,0 2692917 2883598 /opt/solr/data/index/_m5k.cfs (deleted) java 7322 tomcat 74r REG 253,0 32324600 2883592 /opt/solr/data/index/_m5j.cfx (deleted) java 7322 tomcat 75r REG 253,0 6767344 2883603 /opt/solr/data/index/_m5j.cfs (deleted) java 7322 tomcat 76r REG 253,0 32324600 2883592 /opt/solr/data/index/_m5j.cfx (deleted) java 7322 tomcat 77r REG 253,0 15937346 2883600 /opt/solr/data/index/_m5i.cfs (deleted) Stats page on Solr Admin searc...@66952905 main class: org.apache.solr.search.SolrIndexSearcher version: 1.0 description: index searcher stats: searcherName : searc...@66952905 main caching : true numDocs : 187169908 maxDoc : 187169908 readerImpl : ReadOnlyMultiSegmentReader readerDir : org.apache.lucene.store.FSDirectory@/opt/solr/data/index indexVersion : 1224609883675 openedAt : Thu Mar 12 17:13:15 CDT 2009 registeredAt : Thu Mar 12 17:13:23 CDT 2009 warmupTime : 0 name: core class: version: 1.0 description: SolrCore stats: coreName : startTime : Thu Mar 12 17:13:15 CDT 2009 refCount : 2 aliases : [] name: searcher class: org.apache.solr.search.SolrIndexSearcher version: 1.0 description: index searcher stats: searcherName : searc...@66952905 main caching : true numDocs : 187169908 maxDoc : 187169908 readerImpl : ReadOnlyMultiSegmentReader readerDir : org.apache.lucene.store.FSDirectory@/opt/solr/data/index indexVersion : 1224609883675 openedAt : Thu Mar 12 17:13:15 CDT 2009 registeredAt : Thu Mar 12 17:13:23 CDT 2009 warmupTime : 0 Jeremy Carroll Sr. Network Engineer Networked Insights