I have a core which ceased responding either to select or to admin/core:
restarting the Solr instance resolved the problem, but I am wondering whether
there is some configuration which may need to be tweaked. Below is a portion of
solr.log from the time when the problem began. Please note that the directory
/data/solr/subportal1/run_sel_cache_shard1_replica_n3/data/snapshot_metadata
does indeed exist
2022-03-25 08:47:58.526 INFO (qtp1847637306-38437) [c:run_sel_index s:shard1
r:core_node4 x:run_sel_index_shard1_replica_n3] o.a.s.c.S.Request
[run_sel_index_shard1_replica_n3] webapp=/solr path=/select
params={q={!join+from%3Dacc_ref+to%3Dacc_s+fromIndex%3Drun_sel_cache}list_guid:6ccb6d6731f557a9fd3edb34ad637add&facet.limit=2&facet.field=datastore_provider_ss&facet.field=datastore_region_ss&facet.field=datastore_filetype_ss&facet.field=acc_s&facet.field=sra_study_s&facet.field=experiment_s&facet.field=bioproject_s&facet.field=biosample_s&facet.field=sample_acc_s&facet.field=sra_sample_s&facet.field=consent_s&facet.field=gap_accession_sam_ss&facet.field=libraryselection_s&facet.field=librarysource_s&facet.field=librarylayout_s&facet.field=platform_s&facet.field=submission_id_s&facet.field=assemblyname_s&facet.field=submission_id_run_s&facet.field=instrument_s&facet.field=bytes_l&facet.field=bases_l&facet.field=mbytes_l&facet.field=mbases_l&start=0&facet.mincount=1&rows=0&wt=json&facet=on}
hits=94 status=0 QTime=2610
2022-03-25 08:47:58.526 INFO (qtp1847637306-38437) [c:run_sel_index s:shard1
r:core_node4 x:run_sel_index_shard1_replica_n3] o.a.s.c.SolrCore
[run_sel_cache_shard1_replica_n3] CLOSING SolrCore
org.apache.solr.core.SolrCore@59c22833<mailto:org.apache.solr.core.SolrCore@59c22833>
2022-03-25 08:47:58.527 INFO (qtp1847637306-38437) [c:run_sel_index s:shard1
r:core_node4 x:run_sel_index_shard1_replica_n3] o.a.s.m.SolrMetricManager
Closing metric reporters for
registry=solr.core.run_sel_cache.shard1.replica_n3, tag=SolrCore@59c22833
2022-03-25 08:47:58.531 INFO (qtp1847637306-38437) [c:run_sel_index s:shard1
r:core_node4 x:run_sel_index_shard1_replica_n3] o.a.s.m.r.SolrJmxReporter
Closing reporter [org.apache.solr.metrics.reporters.SolrJmxReporter@2b7f5d5e:
rootName = null, domain = solr.core.run_sel_cache.shard1.replica_n3, service
url = null, agent id = null] for registry
solr.core.run_sel_cache.shard1.replica_n3 /
com.codahale.metrics.MetricRegistry@5211d667<mailto:com.codahale.metrics.MetricRegistry@5211d667>
2022-03-25 08:47:58.568 INFO (qtp1847637306-38090) [c:run_sel_index s:shard1
r:core_node4 x:run_sel_index_shard1_replica_n3] o.a.s.c.S.Request
[run_sel_index_shard1_replica_n3] webapp=/solr path=/select
params={q={!join+from%3Dacc_ref+to%3Dacc_s+fromIndex%3Drun_sel_cache}list_guid:6ccb6d6731f557a9fd3edb34ad637add&facet.limit=2&facet.field=datastore_provider_ss&facet.field=datastore_region_ss&facet.field=datastore_filetype_ss&facet.field=acc_s&facet.field=sra_study_s&facet.field=experiment_s&facet.field=bioproject_s&facet.field=biosample_s&facet.field=sample_acc_s&facet.field=sra_sample_s&facet.field=consent_s&facet.field=gap_accession_sam_ss&facet.field=libraryselection_s&facet.field=librarysource_s&facet.field=librarylayout_s&facet.field=platform_s&facet.field=submission_id_s&facet.field=assemblyname_s&facet.field=submission_id_run_s&facet.field=instrument_s&facet.field=bytes_l&facet.field=bases_l&facet.field=mbytes_l&facet.field=mbases_l&start=0&facet.mincount=1&rows=0&wt=json&facet=on}
hits=94 status=0 QTime=593
2022-03-25 08:47:58.593 INFO (qtp1847637306-38437) [c:run_sel_index s:shard1
r:core_node4 x:run_sel_index_shard1_replica_n3] o.a.s.m.SolrMetricManager
Closing metric reporters for
registry=solr.collection.run_sel_cache.shard1.leader, tag=SolrCore@59c22833
2022-03-25 08:47:58.596 INFO (qtp1847637306-38437) [c:run_sel_index s:shard1
r:core_node4 x:run_sel_index_shard1_replica_n3] o.a.s.u.DirectUpdateHandler2
Committing on IndexWriter.close() ... SKIPPED (unnecessary).
2022-03-25 08:47:59.257 INFO (qtp1847637306-38404) [ ] o.a.s.c.SolrCore
[run_sel_cache_shard1_replica_n3] CLOSING SolrCore
org.apache.solr.core.SolrCore@59c22833<mailto:org.apache.solr.core.SolrCore@59c22833>
2022-03-25 08:47:59.257 INFO (qtp1847637306-38404) [ ]
o.a.s.m.SolrMetricManager Closing metric reporters for
registry=solr.core.run_sel_cache.shard1.replica_n3, tag=SolrCore@59c22833
2022-03-25 08:47:59.257 INFO (qtp1847637306-38404) [ ]
o.a.s.m.SolrMetricManager Closing metric reporters for
registry=solr.collection.run_sel_cache.shard1.leader, tag=SolrCore@59c22833
2022-03-25 08:47:59.258 WARN (qtp1847637306-38404) [ ] o.a.s.c.ZkController
/configs/run_sel_cache has no more registered listeners, but a live one
attempted to unregister!
2022-03-25 08:47:59.259 ERROR (qtp1847637306-38404) [ ] o.a.s.c.SolrCore
java.lang.IllegalArgumentException: Unknown directory:
NRTCachingDirectory(MMapDirectory@/data/solr/subportal1/run_sel_cache_shard1_replica_n3/data/snapshot_metadata
lockFactory=org.apache.lucene.store.NativeFSLockFactory@2bebbe92<mailto:lockFactory=org.apache.lucene.store.NativeFSLockFactory@2bebbe92>;
maxCacheMB=48.0 maxMergeSizeMB=4.0) {}
at
org.apache.solr.core.CachingDirectoryFactory.release(CachingDirectoryFactory.java:427)
at org.apache.solr.core.SolrCore.close(SolrCore.java:1674)
at
org.apache.solr.handler.admin.CoreAdminOperation.getCoreStatus(CoreAdminOperation.java:353)
at org.apache.solr.handler.admin.StatusOp.execute(StatusOp.java:46)
at
org.apache.solr.handler.admin.CoreAdminOperation.execute(CoreAdminOperation.java:362)
at
org.apache.solr.handler.admin.CoreAdminHandler$CallInfo.call(CoreAdminHandler.java:397)
at
org.apache.solr.handler.admin.CoreAdminHandler.handleRequestBody(CoreAdminHandler.java:181)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:211)
at
org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:842)
at
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:808)
at org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:559)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:420)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:352)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1596)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:545)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:590)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1607)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1297)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:485)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1577)
at
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1212)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:221)
at
org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:177)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:322)
at
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
at org.eclipse.jetty.server.Server.handle(Server.java:500)
at
org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:547)
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
at
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:270)
at
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
at
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
at
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:388)
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
at
org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
at java.lang.Thread.run(Thread.java:748)
2022-03-25 08:47:59.260 INFO (qtp1847637306-38404) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/cores
params={action=STATUS&indexInfo=false&wt=json} status=0 QTime=3
2022-03-25 08:47:59.262 INFO (qtp1847637306-37771) [ ]
o.a.s.h.a.CollectionsHandler Invoked Collection Action :clusterstatus with
params action=clusterstatus&wt=json and sendToOCPQueue=true
2022-03-25 08:47:59.275 INFO (qtp1847637306-37771) [ ] o.a.s.s.HttpSolrCall
[admin] webapp=null path=/admin/collections
params={action=clusterstatus&wt=json} status=0 QTime=13
2022-03-25 08:48:09.579 INFO (qtp1847637306-38348) [ ] o.a.s.c.SolrCore
[run_sel_cache_shard1_replica_n3] CLOSING SolrCore
org.apache.solr.core.SolrCore@59c22833<mailto:org.apache.solr.core.SolrCore@59c22833>
2022-03-25 08:48:09.579 INFO (qtp1847637306-38348) [ ]
o.a.s.m.SolrMetricManager Closing metric reporters for
registry=solr.core.run_sel_cache.shard1.replica_n3, tag=SolrCore@59c22833
2022-03-25 08:48:09.579 INFO (qtp1847637306-38348) [ ]
o.a.s.m.SolrMetricManager Closing metric reporters for
registry=solr.collection.run_sel_cache.shard1.leader, tag=SolrCore@59c22833
2022-03-25 08:48:09.579 WARN (qtp1847637306-38348) [ ] o.a.s.c.ZkController
/configs/run_sel_cache has no more registered listeners, but a live one
attempted to unregister!
