Hi, Chris I had the same messages in solr log while testing 7.4 and 7.5 The only remedy I've found - increasing header size: /opt/solr/server/etc/jetty.xml <Set name="requestHeaderSize"><Property name="solr.jetty.request.header.size" default="81920" /></Set> <Set name="responseHeaderSize"><Property name="solr.jetty.response.header.size" default="81920" /></Set> After solr restart - no more annoying messages
> -----Original Message----- > From: Chris Ulicny [mailto:culicny@iq.media] > Sent: Wednesday, October 31, 2018 7:40 PM > To: solr-user > Subject: Re: Overseer could not get tags > > I've managed to replicate this issue with the 7.5.0 release as well by > starting up a single instance of solr in cloud mode (on windows) and > uploading the security.json file below to it. > > After a short while, the "could not get tags from node..." messages start > coming through every 60 seconds. The accompanying logged error and > expecting stacktrace are also included below. > > Is there a JIRA ticket for this issue (or a directly related one)? I > couldn't seem to find one. > > Thanks, > Chris > > *security.json:* > { > "authentication":{"blockUnknown":true,"class":"solr.BasicAuthPlugin", > "credentials":{ > "solradmin":"...", > "solrreader":"...", > "solrwriter":"..."} > }, > "authorization":{"class":"solr.RuleBasedAuthorizationPlugin", > "permissions":[ > {"name":"read","role":"reader"}, > {"name":"security-read","role":"reader"}, > {"name":"schema-read","role":"reader"}, > {"name":"config-read","role":"reader"}, > {"name":"core-admin-read","role":"reader"}, > {"name":"collection-admin-read","role":"reader"}, > {"name":"update","role":"writer"}, > {"name":"security-edit","role":"admin"}, > {"name":"schema-edit","role":"admin"}, > {"name":"config-edit","role":"admin"}, > {"name":"core-admin-edit","role":"admin"}, > {"name":"collection-admin-edit","role":"admin"}, > {"name":"all","role":"admin"}], > "user-role":{ > "solradmin":["reader","writer","admin"], > "solrreader":["reader"], > "solrwriter":["reader","writer"]} > } > } > > *StackTrace:* > 2018-10-31 16:20:01.994 WARN (MetricsHistoryHandler-12-thread-1) [ ] > o.a.s.c.s.i.SolrClientNodeStateProvider could not get tags from node > ip:8080_solr > org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error > from server at http://ip:8080/solr: Expected mime type > application/octet-stream but got text/html. <html> > <head> > <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> > <title>Error 401 require authentication</title> > </head> > <body><h2>HTTP ERROR 401</h2> > <p>Problem accessing /solr/admin/metrics. Reason: > <pre> require authentication</pre></p> > </body> > </html> > > at > org.apache.solr.client.solrj.impl.HttpSolrClient.executeMethod(HttpSolrClient.ja > va:607) > ~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - > jimczi - 2018-09-18 13:07:58] > at > org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:255) > ~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - > jimczi - 2018-09-18 13:07:58] > at > org.apache.solr.client.solrj.impl.HttpSolrClient.request(HttpSolrClient.java:244) > ~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - > jimczi - 2018-09-18 13:07:58] > at > org.apache.solr.client.solrj.SolrClient.request(SolrClient.java:1219) > ~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - > jimczi - 2018-09-18 13:07:58] > at > org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider$ClientSnitchCtx.in > voke(SolrClientNodeStateProvider.java:342) > ~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - > jimczi - 2018-09-18 13:07:58] > at > org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.fetchReplicaMetri > cs(SolrClientNodeStateProvider.java:195) > ~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - > jimczi - 2018-09-18 13:07:58] > at > org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider$AutoScalingSnitc > h.getRemoteInfo(SolrClientNodeStateProvider.java:241) > ~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - > jimczi - 2018-09-18 13:07:58] > at > org.apache.solr.common.cloud.rule.ImplicitSnitch.getTags(ImplicitSnitch.java:7 > 6) > ~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - > jimczi - 2018-09-18 13:07:58] > at > org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.fetchTagValues(S > olrClientNodeStateProvider.java:139) > ~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - > jimczi - 2018-09-18 13:07:58] > at > org.apache.solr.client.solrj.impl.SolrClientNodeStateProvider.getNodeValues(S > olrClientNodeStateProvider.java:128) > ~[solr-solrj-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - > jimczi - 2018-09-18 13:07:58] > at > org.apache.solr.handler.admin.MetricsHistoryHandler.collectGlobalMetrics(Me > tricsHistoryHandler.java:498) > ~[solr-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - > jimczi - 2018-09-18 13:07:55] > at > org.apache.solr.handler.admin.MetricsHistoryHandler.collectMetrics(MetricsHi > storyHandler.java:371) > ~[solr-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - > jimczi - 2018-09-18 13:07:55] > at > org.apache.solr.handler.admin.MetricsHistoryHandler.lambda$new$0(Metrics > HistoryHandler.java:231) > ~[solr-core-7.5.0.jar:7.5.0 b5bf70b7e32d7ddd9742cc821d471c5fabd4e3df - > jimczi - 2018-09-18 13:07:55] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [?:1.8.0_121] > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) > [?:1.8.0_121] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.acces > s$301(ScheduledThreadPoolExecutor.java:180) > [?:1.8.0_121] > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(S > cheduledThreadPoolExecutor.java:294) > [?:1.8.0_121] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java: > 1142) > [?:1.8.0_121] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java > :617) > [?:1.8.0_121] > at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121] > > > On Wed, Oct 17, 2018 at 8:32 AM Chris Ulicny <culicny@iq.media> wrote: > > > Hi all, > > > > Recently in a 7.4.0 test cluster, we ran into SOLR-12814 > > <https://issues.apache.org/jira/browse/SOLR-12814> which we fixed by > > slightly increasing the request header size. However, there were some other > > log messages along with the "URI size >8192" message which we thought > were > > related, but have not abated since increasing the header size. A full > > shutdown of the solr processes and bringing them back up one at a time did > > not solve the issue. > > > > The overseer node seems to not be authenticating any of the requests to > > /solr/admin/metrics on any node (including itself). Every minute, there are > > two warning per node > > > > 10/17/2018, 7:53:45 AM WARN SolrClientNodeStateProvider could not > > get tags from node host1:port1_solr > > 10/17/2018, 7:53:45 AM WARN SolrClientNodeStateProvider could not > > get tags from node host1:port1_solr > > 10/17/2018, 7:53:46 AM WARN SolrClientNodeStateProvider could not > > get tags from node host2:port2_solr > > 10/17/2018, 7:53:46 AM WARN SolrClientNodeStateProvider could not > > get tags from node host2:port2_solr > > 10/17/2018, 7:53:46 AM WARN SolrClientNodeStateProvider could not > > get tags from node host3:port3_solr > > 10/17/2018, 7:53:46 AM WARN SolrClientNodeStateProvider could not > > get tags from node host3:port3_solr > > > > There are two slightly different stack traces that appear with each pair: > > https://pastebin.com/2Z1C5rXr > > > > The warning message possibly comes from > > solrj.impl.SolrClientNodeStateProvider.fetchMetrics which both of the > > attempted requests call in their stack trace. > > > > However, we already have a 7.4.0 production cluster running that also has > > security enabled with similar replica density where we have not seen this > > issue. > > > > *Test:* > > -- 10 collections (9 with 2 shards, 1 with 43 shards) > > -- replication factor of 2 for all collections > > -- 3 hosts with 40 or 41 replicas each > > > > *Production:* > > -- 9 collections with 14 shards > > -- replication factor of 2 for all collections > > -- 7 hosts with 36 replicas each > > > > I've enabled TRACE logging in our test environment on most options related > > to metrics and authentication. So far the only new message I've gotten is > > the challenge from the target server for the necessary credentials right > > before the warning and stack trace. > > > > 2018-10-17 12:20:46.368 DEBUG (MetricsHistoryHandler-8-thread-1) [ ] > > o.a.h.i.a.HttpAuthenticator Authentication required > > 2018-10-17 12:20:46.368 DEBUG (MetricsHistoryHandler-8-thread-1) [ ] > > o.a.h.i.a.HttpAuthenticator host3:port3 requested authentication > > > > I suspect the creation and balancing of the large collection on test might > > have something to do with it since the problem started happening after > > that. > > > > Are there any other specific log settings I should turn on that might > > produce some useful information? > > > > Thanks, > > Chris > >
