Hi,
I’m seeing the following exception in Solr that gets automatically resolved
eventually.
2017-09-22 00:18:17.243 ERROR (qtp1702660825-17) [ x:spedamallu1-core-1]
o.a.s.c.CoreContainer Error creating core [spedamallu1-core-1]: Error opening
new searcher
org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:952)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:816)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:890)
at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1167)
at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:252)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:418)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:296)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134)
at org.eclipse.jetty.server.Server.handle(Server.java:534)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95)
at
org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148)
at
org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1891)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2011)
at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1041)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:925)
... 32 more
Caused by: org.apache.lucene.store.LockObtainFailedException: Lock held by this
virtual machine:
/Users/spedamallu/Desktop/mount-1/spedamallu1-core-1/data/index/write.lock
at
org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:127)
at
org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:41)
at
org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45)
at
org.apache.lucene.store.FilterDirectory.obtainLock(FilterDirectory.java:104)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:804)
at
org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:125)
at
org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:100)
at
org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:240)
at
org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:114)
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1852)
I kind of have a theory of why this is happening. Can someone please confirm if
this is indeed the case.
* I was trying to run a long running operation on a transient core and it
was getting evicted because of LRU
* So, to ensure my operation completion, I have added a solrCore.open()
call before the beginning of the operation that would increment the core
refCount and decrementing the refCount again in the finally block.
* Now, I was successful in completing the operation, but that caused above
Exception.
* Based on the logs and the point that this exception disappears when my
operation is completed, my theory is that:
* Incrementing the refCount does not prevent or delay the
solrCore.close() call for eviction based on LRU. However, while the refCount is
decreased and Solr assumes that this core has been evicted, it is still indeed
loaded performing the long-running operation.
* So, now when a new request comes to this core, Solr assumes that it
has to be loaded anew and tries to obtain the lock which is actually held by
the long running operation and throws this error.
* Can someone, please confirm if this could be the possibility. Is there a
way I can get out of it?
Thanks in advance,
Shashank
