Hello everyone I use a 3-node solrCloud with 3 Zookeeper nodes (all on VMWare at one physical machine) and got stuck while creating new collections. From time to time it works via API or GUI, but mostly i receive a BaseHttpSolrClient$RemoteSolrException: Error from server
Here is what I see: [cid:bc5bb68a-15af-4c1f-aece-4929283ff29f] These are my already created Collections: [cid:5dd51a50-a540-4041-919d-a655530e9d2e] In security.json is set "blockUnknown": False in authentication and at the end of the file, which is already copied to zookeeper. This is how the zookeeper status looks like: [cid:d7da20d2-1ba3-4c49-bcc4-9ac3158e55bb] When trying to create a new collection i use either the GUI or (Python code which works fine e.g. to get something like the Clusterstatus): createCollection = "http://s03:8983/solr/admin/collections?action=CREATE&name=GPL_COLLECTION_10&replicationFactor=2&numShards=1"; status = requests.post(createCollection, auth=HTTPBasicAuth(user, password)) and receive status 500: {'responseHeader': {'status': 500, 'QTime': 180017}, 'error': {'metadata': ['error-class', 'org.apache.solr.common.SolrException', 'root-error-class', 'org.apache.solr.common.SolrException'], 'msg': 'create the collection time out:180s', 'trace': 'org.apache.solr.common.SolrException: create the collection time out:180s\n\tat org.apache.solr.handler.admin.CollectionsHandler.submitCollectionApiCommand(CollectionsHandler.java:415)\n\tat org.apache.solr.handler.admin.api.CreateCollectionAPI.createCollection(CreateCollectionAPI.java:133)\n\tat org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.lambda$static$0(CollectionsHandler.java:513)\n\tat org.apache.solr.handler.admin.CollectionsHandler$CollectionOperation.execute(CollectionsHandler.java:1265)\n\tat org.apache.solr.handler.admin.CollectionsHandler.invokeAction(CollectionsHandler.java:315)\n\tat org.apache.solr.handler.admin.CollectionsHandler.handleRequestBody(CollectionsHandler.java:293)\n\tat org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:226)\n\tat org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:932)\n\tat org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:880)\n\tat org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:541)\n\tat org.apache.solr.servlet.SolrDispatchFilter.dispatch(SolrDispatchFilter.java:262)\n\tat org.apache.solr.servlet.SolrDispatchFilter.lambda$doFilter$0(SolrDispatchFilter.java:219)\n\tat org.apache.solr.servlet.ServletUtils.traceHttpRequestExecution2(ServletUtils.java:246)\n\tat org.apache.solr.servlet.ServletUtils.rateLimitRequest(ServletUtils.java:215)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:213)\n\tat org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:195)\n\tat org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:210)\n\tat org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)\n\tat org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131)\n\tat org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:598)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223)\n\tat org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1570)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)\n\tat org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)\n\tat org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1543)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)\n\tat org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306)\n\tat org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)\n\tat org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:149)\n\tat org.eclipse.jetty.server.handler.InetAccessHandler.handle(InetAccessHandler.java:228)\n\tat org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:141)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)\n\tat org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:301)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)\n\tat org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:822)\n\tat org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)\n\tat org.eclipse.jetty.server.Server.handle(Server.java:563)\n\tat org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598)\n\tat org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753)\n\tat org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501)\n\tat org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:287)\n\tat org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)\n\tat org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)\n\tat org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)\n\tat org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421)\n\tat org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390)\n\tat org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277)\n\tat org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199)\n\tat org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)\n\tat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)\n\tat java.base/java.lang.Thread.run(Thread.java:829)\n', 'code': 500}} the logfile of the leader says further: <head> org.apache.solr.client.solrj.impl.BaseHttpSolrClient$RemoteSolrException: Error from server at http://s01:8983/solr/admin/cores: Expected mime type in [application/octet-stream, application/vnd.apache.solr.javabin] but got text/html. <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/> <title>Error 403 Forbidden</title> </head> <body><h2>HTTP ERROR 403 Forbidden</h2> <table> <tr><th>URI:</th><td>/solr/admin/cores</td></tr> <tr><th>STATUS:</th><td>403</td></tr> <tr><th>MESSAGE:</th><td>Forbidden</td></tr> <tr><th>SERVLET:</th><td>-</td></tr> </table> </body> </html> at org.apache.solr.client.solrj.impl.Http2SolrClient.checkContentType(Http2SolrClient.java:974) ~[?:?] at org.apache.solr.client.solrj.impl.Http2SolrClient.processErrorsAndResponse(Http2SolrClient.java:863) ~[?:?] at org.apache.solr.client.solrj.impl.Http2SolrClient.processErrorsAndResponse(Http2SolrClient.java:576) ~[?:?] at org.apache.solr.client.solrj.impl.Http2SolrClient$1.lambda$onHeaders$0(Http2SolrClient.java:481) ~[?:?] at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:294) ~[?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?] at java.lang.Thread.run(Thread.java:829) [?:?] 2024-05-12 17:04:03.874 INFO (zkCallback-11-thread-11) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/GPL_COLLECTION_10/state.json zxid: 438086664484] for collection [GPL_CO> 2024-05-12 17:04:03.875 INFO (zkCallback-11-thread-12) [] o.a.s.c.c.ZkStateReader A cluster state change: [WatchedEvent state:SyncConnected type:NodeDeleted path:/collections/GPL_COLLECTION_10/state.json zxid: 438086664484] for collection [GPL_CO> 2024-05-12 17:04:03.904 INFO (OverseerThreadFactory-17-thread-2) [c:GPL_COLLECTION_10] o.a.s.c.a.c.CreateCollectionCmd Cleaned up artifacts for failed create collection for [GPL_COLLECTION_10] 2024-05-12 17:04:03.904 ERROR (OverseerThreadFactory-17-thread-2) [c:GPL_COLLECTION_10] o.a.s.c.a.c.OverseerCollectionMessageHandler Collection GPL_COLLECTION_10}, operation create failed => org.apache.solr.common.SolrException: Underlying core cr> I am trying to solve this since days. Could you please help me out? Thanks in andvance, please let me know if I should provide some more information. Regards, Ulli
