Hi Shashank, There is an open issue about this exception [1]. Can you take a look and test the patch to see if it works in your case?
[1] https://issues.apache.org/jira/browse/SOLR-11297 On Sep 21, 2017 10:19 PM, "Shashank Pedamallu" <spedama...@vmware.com> wrote: Hi, I’m seeing the following exception in Solr that gets automatically resolved eventually. 2017-09-22 00:18:17.243 ERROR (qtp1702660825-17) [ x:spedamallu1-core-1] o.a.s.c.CoreContainer Error creating core [spedamallu1-core-1]: Error opening new searcher org.apache.solr.common.SolrException: Error opening new searcher at org.apache.solr.core.SolrCore.<init>(SolrCore.java:952) at org.apache.solr.core.SolrCore.<init>(SolrCore.java:816) at org.apache.solr.core.CoreContainer.create(CoreContainer.java:890) at org.apache.solr.core.CoreContainer.getCore( CoreContainer.java:1167) at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:252) at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:418) at org.apache.solr.servlet.SolrDispatchFilter.doFilter( SolrDispatchFilter.java:345) at org.apache.solr.servlet.SolrDispatchFilter.doFilter( SolrDispatchFilter.java:296) at org.eclipse.jetty.servlet.ServletHandler$CachedChain. doFilter(ServletHandler.java:1691) at org.eclipse.jetty.servlet.ServletHandler.doHandle( ServletHandler.java:582) at org.eclipse.jetty.server.handler.ScopedHandler.handle( ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle( SecurityHandler.java:548) at org.eclipse.jetty.server.session.SessionHandler. doHandle(SessionHandler.java:226) at org.eclipse.jetty.server.handler.ContextHandler. doHandle(ContextHandler.java:1180) at org.eclipse.jetty.servlet.ServletHandler.doScope( ServletHandler.java:512) at org.eclipse.jetty.server.session.SessionHandler. doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler. doScope(ContextHandler.java:1112) at org.eclipse.jetty.server.handler.ScopedHandler.handle( ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle( ContextHandlerCollection.java:213) at org.eclipse.jetty.server.handler.HandlerCollection. handle(HandlerCollection.java:119) at org.eclipse.jetty.server.handler.HandlerWrapper.handle( HandlerWrapper.java:134) at org.eclipse.jetty.server.Server.handle(Server.java:534) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) at org.eclipse.jetty.server.HttpConnection.onFillable( HttpConnection.java:251) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded( AbstractConnection.java:273) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) at org.eclipse.jetty.io.SelectChannelEndPoint$2.run( SelectChannelEndPoint.java:93) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume. executeProduceConsume(ExecuteProduceConsume.java:303) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume. produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run( ExecuteProduceConsume.java:136) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob( QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run( QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.solr.common.SolrException: Error opening new searcher at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1891) at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2011) at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1041) at org.apache.solr.core.SolrCore.<init>(SolrCore.java:925) ... 32 more Caused by: org.apache.lucene.store.LockObtainFailedException: Lock held by this virtual machine: /Users/spedamallu/Desktop/mount-1/spedamallu1-core-1/ data/index/write.lock at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock( NativeFSLockFactory.java:127) at org.apache.lucene.store.FSLockFactory.obtainLock( FSLockFactory.java:41) at org.apache.lucene.store.BaseDirectory.obtainLock( BaseDirectory.java:45) at org.apache.lucene.store.FilterDirectory.obtainLock( FilterDirectory.java:104) at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:804) at org.apache.solr.update.SolrIndexWriter.<init>( SolrIndexWriter.java:125) at org.apache.solr.update.SolrIndexWriter.create( SolrIndexWriter.java:100) at org.apache.solr.update.DefaultSolrCoreState. createMainIndexWriter(DefaultSolrCoreState.java:240) at org.apache.solr.update.DefaultSolrCoreState.getIndexWriter( DefaultSolrCoreState.java:114) at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1852) I kind of have a theory of why this is happening. Can someone please confirm if this is indeed the case. * I was trying to run a long running operation on a transient core and it was getting evicted because of LRU * So, to ensure my operation completion, I have added a solrCore.open() call before the beginning of the operation that would increment the core refCount and decrementing the refCount again in the finally block. * Now, I was successful in completing the operation, but that caused above Exception. * Based on the logs and the point that this exception disappears when my operation is completed, my theory is that: * Incrementing the refCount does not prevent or delay the solrCore.close() call for eviction based on LRU. However, while the refCount is decreased and Solr assumes that this core has been evicted, it is still indeed loaded performing the long-running operation. * So, now when a new request comes to this core, Solr assumes that it has to be loaded anew and tries to obtain the lock which is actually held by the long running operation and throws this error. * Can someone, please confirm if this could be the possibility. Is there a way I can get out of it? Thanks in advance, Shashank
