Hello, Occasionally we are seeing errors opening new searcher for certain solr cores. Whenever this happens, we are unable to query or ingest new data into these cores. It seems to clear up after some time though. The root cause seems to be - *"org.apache.lucene.store.LockObtainFailedException: Lock held by this virtual machine: /opt/solr/volumes/data9/7d50b38e114af075-core-24/data/index/write.lock"*
Below is the full stack trace. Any ideas on what could be going on that causes such an exception and how to mitigate this? thanks a lot for your help! Unable to create core [7d50b38e114af075-core-24],trace=org.apache.solr.common.SolrException: Unable to create core [7d50b38e114af075-core-24] at org.apache.solr.core.CoreContainer.create(CoreContainer.java:903) at org.apache.solr.core.CoreContainer.getCore(CoreContainer.java:1167) at org.apache.solr.servlet.HttpSolrCall.init(HttpSolrCall.java:252) at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:418) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:345) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:296) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1691) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:582) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1180) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:512) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1112) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:119) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:134) at org.eclipse.jetty.server.Server.handle(Server.java:534) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:320) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:273) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:95) at org.eclipse.jetty.io.SelectChannelEndPoint$2.run(SelectChannelEndPoint.java:93) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.executeProduceConsume(ExecuteProduceConsume.java:303) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:148) at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:136) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:671) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:589) at java.lang.Thread.run(Thread.java:748) Caused by: org.apache.solr.common.SolrException: Error opening new searcher at org.apache.solr.core.SolrCore.<init>(SolrCore.java:952) at org.apache.solr.core.SolrCore.<init>(SolrCore.java:816) at org.apache.solr.core.CoreContainer.create(CoreContainer.java:890) ... 30 more Caused by: org.apache.solr.common.SolrException: Error opening new searcher at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1891) at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:2011) at org.apache.solr.core.SolrCore.initSearcher(SolrCore.java:1041) at org.apache.solr.core.SolrCore.<init>(SolrCore.java:925) ... 32 more Caused by: org.apache.lucene.store.LockObtainFailedException: Lock held by this virtual machine: /opt/solr/volumes/data9/7d50b38e114af075-core-24/data/index/write.lock at org.apache.lucene.store.NativeFSLockFactory.obtainFSLock(NativeFSLockFactory.java:127) at org.apache.lucene.store.FSLockFactory.obtainLock(FSLockFactory.java:41) at org.apache.lucene.store.BaseDirectory.obtainLock(BaseDirectory.java:45) at org.apache.lucene.store.FilterDirectory.obtainLock(FilterDirectory.java:104) at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:804) at org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:125) at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:100) at org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:240) at org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:114) at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1852) ... 35 more ,code=500}```
