Hi guys,
We use Solr 1.3 for indexing large amounts of data (50G avg) on Linux
environment and use the replication scripts to make replicas those live in load
balancing slaves.
The issue we face quite often (only in Linux servers) is that they tend to not
been able to find the segment file (segment_x etc) after the replicating
completed. As this has become quite common, we started hitting a serious issue.
Below is a stack trace, if that helps and any help on this matter is greatly
appreciated.
--------------------------------
Nov 5, 2009 11:34:46 PM org.apache.solr.util.plugin.AbstractPluginLoader load
INFO: created /admin/: org.apache.solr.handler.admin.AdminHandlers
Nov 5, 2009 11:34:46 PM org.apache.solr.util.plugin.AbstractPluginLoader load
INFO: created /admin/ping: org.apache.solr.handler.PingRequestHandler
Nov 5, 2009 11:34:46 PM org.apache.solr.util.plugin.AbstractPluginLoader load
INFO: created /debug/dump: org.apache.solr.handler.DumpRequestHandler
Nov 5, 2009 11:34:46 PM org.apache.solr.util.plugin.AbstractPluginLoader load
INFO: created gap: org.apache.solr.highlight.GapFragmenter
Nov 5, 2009 11:34:46 PM org.apache.solr.util.plugin.AbstractPluginLoader load
INFO: created regex: org.apache.solr.highlight.RegexFragmenter
Nov 5, 2009 11:34:46 PM org.apache.solr.util.plugin.AbstractPluginLoader load
INFO: created html: org.apache.solr.highlight.HtmlFormatter
Nov 5, 2009 11:34:46 PM org.apache.solr.servlet.SolrDispatchFilter init
SEVERE: Could not start SOLR. Check solr/home property
java.lang.RuntimeException: java.io.FileNotFoundException:
/solrinstances/solrhome01/data/index/segments_v (No such file or directory)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:960)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:470)
at
org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:119)
at
org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:69)
at
org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:275)
at
org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:397)
at
org.apache.catalina.core.ApplicationFilterConfig.<init>(ApplicationFilterConfig.java:108)
at
org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:3709)
at
org.apache.catalina.core.StandardContext.start(StandardContext.java:4363)
at
org.apache.catalina.core.StandardContext.reload(StandardContext.java:3099)
at
org.apache.catalina.manager.ManagerServlet.reload(ManagerServlet.java:916)
at
org.apache.catalina.manager.HTMLManagerServlet.reload(HTMLManagerServlet.java:536)
at
org.apache.catalina.manager.HTMLManagerServlet.doGet(HTMLManagerServlet.java:114)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.jamonapi.JAMonFilter.doFilter(JAMonFilter.java:57)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:525)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:845)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.FileNotFoundException:
/solrinstances/solrhome01/data/index/segments_v (No such file or directory)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
at
org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
at
org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:482)
at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:214)
at
org.apache.lucene.index.DirectoryIndexReader$1.doBody(DirectoryIndexReader.java:94)
at
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:653)
at
org.apache.lucene.index.DirectoryIndexReader.open(DirectoryIndexReader.java:111)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:316)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:237)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:951)
... 30 more
Nov 5, 2009 11:34:46 PM org.apache.solr.common.SolrException log
SEVERE: java.lang.RuntimeException: java.io.FileNotFoundException:
/solrinstances/solrhome01/data/index/segments_v (No such file or directory)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:960)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:470)
at
org.apache.solr.core.CoreContainer$Initializer.initialize(CoreContainer.java:119)
at
org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:69)
at
org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:275)
at
org.apache.catalina.core.ApplicationFilterConfig.setFilterDef(ApplicationFilterConfig.java:397)
at
org.apache.catalina.core.ApplicationFilterConfig.<init>(ApplicationFilterConfig.java:108)
at
org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:3709)
at
org.apache.catalina.core.StandardContext.start(StandardContext.java:4363)
at
org.apache.catalina.core.StandardContext.reload(StandardContext.java:3099)
at
org.apache.catalina.manager.ManagerServlet.reload(ManagerServlet.java:916)
at
org.apache.catalina.manager.HTMLManagerServlet.reload(HTMLManagerServlet.java:536)
at
org.apache.catalina.manager.HTMLManagerServlet.doGet(HTMLManagerServlet.java:114)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at com.jamonapi.JAMonFilter.doFilter(JAMonFilter.java:57)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:525)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:128)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:286)
at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:845)
at
org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:583)
at
org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:447)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.io.FileNotFoundException:
/solrinstances/solrhome01/data/index/segments_v (No such file or directory)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:212)
at
org.apache.lucene.store.FSDirectory$FSIndexInput$Descriptor.<init>(FSDirectory.java:552)
at
org.apache.lucene.store.FSDirectory$FSIndexInput.<init>(FSDirectory.java:582)
at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:488)
at org.apache.lucene.store.FSDirectory.openInput(FSDirectory.java:482)
at org.apache.lucene.index.SegmentInfos.read(SegmentInfos.java:214)
at
org.apache.lucene.index.DirectoryIndexReader$1.doBody(DirectoryIndexReader.java:94)
at
org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:653)
at
org.apache.lucene.index.DirectoryIndexReader.open(DirectoryIndexReader.java:111)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:316)
at org.apache.lucene.index.IndexReader.open(IndexReader.java:237)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:951)
... 30 more
Nov 5, 2009 11:34:46 PM org.apache.solr.servlet.SolrDispatchFilter init
INFO: SolrDispatchFilter.init() done
Nov 5, 2009 11:34:46 PM org.apache.solr.servlet.SolrServlet init
INFO: SolrServlet.init()
--------------------------------
Steps to re-produce the error (However, for me did not work in my local box.
Also remote server is too far away to remote-debug!).
- Post some new data to the master server (Usually about 1Gb worth text files)
- Run the replicate script in slave Solr instance
- Try to login to admin in slave Solr instance
And you should see above stack trace even in the Tomcat output.
Thanks in advance.
Madu
