Hello all I hit another problem in moving from Solr 6 to 8.
We secure our ZooKeeper entirely (there's a restrictive ACL for every znode) To pass the ZooKeeper credentials to Solr we implemented ZkCredentialsProvider and ZkACLProvider to load the credentials from a file on disk, which has the credentials in an encoded format (the details don't really matter, just that we have our own implementations of the providers) In Solr 6 we were able to specify the implementation classes in the /solr.xml (zkCredentialsProvider & zkACLProvider) in the home directory for each node and provided the implementation in a jar in the /lib directory of the node, and this worked without issue. In Solr 8.1.1, this still works for *some* of the requests that are made to ZooKeeper, but other requests made by the SolrZkClient (via SolrDispatchFilter) do not use the ZkCredentialsProvider specified in solr.xml, instead falling back to the DefaultZkCredentialsProvider, or a provider specified by the system property *-DzkCredentialsProvider.* - If Solr makes requests using the DefaultZkCredentialsProvider, they will fail with *NoAuth for /clusterprops.json.* - If I specify the *-DzkCredentialsProvider* then the jars in /lib are not visible to the SolrZkClient making the requests, resulting in a ClassNotFoundException (log attached). - If I add the jar directly to solr\server\solr-webapp\webapp\WEB-INF\lib then it is loadable, but I don't want to mess around with the lib directory Solr provides. - I know that the solrconfig.xml allows you to configure the <lib> directive for the classpath, but Solr won't be able to retrieve that solrconfig.xml from ZooKeeper without authenticating... Catch-22. I've not had a chance to properly read the Solr 8 Reference Guide yet, but it does still refer to the zkCredentialsProvider & zkACLProvider properties being supported https://lucene.apache.org/solr/guide/8_0/format-of-solr-xml.html#the-solr-element I didn't see anything particular in the upgrade notes for Solr 7 or 8 about changes to zkCredentialsProvider So is there something wrong with what we were doing in Solr 6? Or has there been a regression? Thanks for any advice Colvin
2019-06-11 10:04:42.621 INFO (main) [ ] o.a.s.s.SolrDispatchFilter / __| ___| |_ _ Starting in cloud mode on port 8983 2019-06-11 10:04:42.621 INFO (main) [ ] o.a.s.s.SolrDispatchFilter \__ \/ _ \ | '_| Install dir: C:\solr 2019-06-11 10:04:42.622 INFO (main) [ ] o.a.s.s.SolrDispatchFilter |___/\___/_|_| Start time: 2019-06-11T10:04:42.621Z 2019-06-11 10:04:42.644 INFO (main) [ ] o.a.s.c.SolrResourceLoader Using system property solr.solr.home: C:/data/solr/clusters/is_cluster/nodes/node1 2019-06-11 10:04:42.656 INFO (main) [ ] o.a.s.c.c.SolrZkClient Using ZkCredentialsProvider: xxx.yyy.zzz.EncodedZkCredentialsProvider 2019-06-11 10:04:42.657 WARN (main) [ ] o.a.s.c.c.SolrZkClient VM param zkCredentialsProvider does not point to a class implementing ZkCredentialsProvider and with a non-arg constructor => java.lang.ClassNotFoundException: xxx.yyy.zzz.EncodedZkCredentialsProvider at java.net.URLClassLoader.findClass(URLClassLoader.java:382) java.lang.ClassNotFoundException: xxx.yyy.zzz.EncodedZkCredentialsProvider at java.net.URLClassLoader.findClass(URLClassLoader.java:382) ~[?:1.8.0_211] at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_211] at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_211] at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:556) ~[jetty-webapp-9.4.14.v20181114.jar:9.4.14.v20181114] at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_211] at java.lang.Class.forName0(Native Method) ~[?:1.8.0_211] at java.lang.Class.forName(Class.java:264) ~[?:1.8.0_211] at org.apache.solr.common.cloud.SolrZkClient.createZkCredentialsToAddAutomatically(SolrZkClient.java:225) ~[?:?] at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:144) ~[?:?] at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:126) ~[?:?] at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:121) ~[?:?] at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:108) ~[?:?] at org.apache.solr.servlet.SolrDispatchFilter.loadNodeConfig(SolrDispatchFilter.java:276) ~[?:?] at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:253) ~[?:?] at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:175) ~[?:?] at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:136) ~[jetty-servlet-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.servlet.ServletHandler.lambda$initialize$0(ServletHandler.java:750) ~[jetty-servlet-9.4.14.v20181114.jar:9.4.14.v20181114] at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) ~[?:1.8.0_211] at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742) ~[?:1.8.0_211] at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742) ~[?:1.8.0_211] at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580) ~[?:1.8.0_211] at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:744) ~[jetty-servlet-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:368) ~[jetty-servlet-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1497) ~[jetty-webapp-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1459) ~[jetty-webapp-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:852) ~[jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:278) ~[jetty-servlet-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545) ~[jetty-webapp-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:46) ~[jetty-deploy-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:192) ~[jetty-deploy-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:505) ~[jetty-deploy-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:151) ~[jetty-deploy-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:180) ~[jetty-deploy-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:453) ~[jetty-deploy-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:64) ~[jetty-deploy-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:610) ~[jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:529) ~[jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.util.Scanner.scan(Scanner.java:392) ~[jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:313) ~[jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:150) ~[jetty-deploy-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:579) ~[jetty-deploy-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:240) ~[jetty-deploy-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138) ~[jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.server.Server.start(Server.java:415) ~[jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) ~[jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) ~[jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.server.Server.doStart(Server.java:382) ~[jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1572) ~[jetty-xml-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1512) ~[jetty-xml-9.4.14.v20181114.jar:9.4.14.v20181114] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_211] at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1511) ~[jetty-xml-9.4.14.v20181114.jar:9.4.14.v20181114] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_211] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_211] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_211] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_211] at org.eclipse.jetty.start.Main.invokeMain(Main.java:220) ~[start.jar:9.4.14.v20181114] at org.eclipse.jetty.start.Main.start(Main.java:490) ~[start.jar:9.4.14.v20181114] at org.eclipse.jetty.start.Main.main(Main.java:77) ~[start.jar:9.4.14.v20181114] 2019-06-11 10:04:47.240 INFO (zkConnectionManagerCallback-2-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected 2019-06-11 10:04:47.248 INFO (main) [ ] o.a.s.s.SolrDispatchFilter Loading solr.xml from SolrHome (not found in ZooKeeper) 2019-06-11 10:04:47.251 INFO (main) [ ] o.a.s.c.SolrXmlConfig Loading container configuration from C:\data\solr\clusters\is_cluster\nodes\node1\solr.xml 2019-06-11 10:04:47.347 INFO (main) [ ] o.a.s.c.SolrXmlConfig MBean server found: com.sun.jmx.mbeanserver.JmxMBeanServer@161b062a, but no JMX reporters were configured - adding default JMX reporter. 2019-06-11 10:04:47.875 INFO (main) [ ] o.a.s.c.SolrResourceLoader [null] Added 7 libs to classloader, from paths: [/C:/data/solr/clusters/is_cluster/nodes/node1/lib] 2019-06-11 10:04:48.004 INFO (main) [ ] o.a.s.h.c.HttpShardHandlerFactory Host whitelist initialized: WhitelistHostChecker [whitelistHosts=null, whitelistHostCheckingEnabled=true] 2019-06-11 10:04:48.597 WARN (main) [ ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@45c8d09f[provider=null,keyStore=null,trustStore=null] 2019-06-11 10:04:48.706 WARN (main) [ ] o.e.j.u.s.S.config No Client EndPointIdentificationAlgorithm configured for SslContextFactory@1536602f[provider=null,keyStore=null,trustStore=null] 2019-06-11 10:04:48.716 INFO (main) [ ] o.a.s.c.ZkContainer Zookeeper client=localhost:9983 2019-06-11 10:04:48.745 INFO (zkConnectionManagerCallback-9-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected 2019-06-11 10:04:48.863 INFO (main) [ ] o.a.s.c.ZkController Register node as live in ZooKeeper:/live_nodes/localhost:8983_solr 2019-06-11 10:04:48.877 INFO (zkCallback-8-thread-1) [ ] o.a.s.c.c.ZkStateReader Updated live nodes from ZooKeeper... (0) -> (1) 2019-06-11 10:04:48.888 INFO (main) [ ] o.a.s.c.CoreContainer Initializing authorization plugin: solr.RuleBasedAuthorizationPlugin 2019-06-11 10:04:48.900 INFO (main) [ ] o.a.s.c.CoreContainer Initializing authentication plugin: solr.BasicAuthPlugin 2019-06-11 10:04:53.558 INFO (main) [ ] o.a.s.c.c.SolrZkClient Using ZkCredentialsProvider: xxx.yyy.zzz.EncodedZkCredentialsProvider 2019-06-11 10:04:53.558 WARN (main) [ ] o.a.s.c.c.SolrZkClient VM param zkCredentialsProvider does not point to a class implementing ZkCredentialsProvider and with a non-arg constructor => java.lang.ClassNotFoundException: xxx.yyy.zzz.EncodedZkCredentialsProvider at java.net.URLClassLoader.findClass(URLClassLoader.java:382) java.lang.ClassNotFoundException: xxx.yyy.zzz.EncodedZkCredentialsProvider at java.net.URLClassLoader.findClass(URLClassLoader.java:382) ~[?:1.8.0_211] at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_211] at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_211] at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:556) ~[jetty-webapp-9.4.14.v20181114.jar:9.4.14.v20181114] at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_211] at java.lang.Class.forName0(Native Method) ~[?:1.8.0_211] at java.lang.Class.forName(Class.java:264) ~[?:1.8.0_211] at org.apache.solr.common.cloud.SolrZkClient.createZkCredentialsToAddAutomatically(SolrZkClient.java:225) ~[?:?] at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:144) ~[?:?] at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:126) ~[?:?] at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:116) ~[?:?] at org.apache.solr.common.cloud.ZkStateReader.<init>(ZkStateReader.java:309) ~[?:?] at org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.getZkStateReader(ZkClientClusterStateProvider.java:175) ~[?:?] at org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.connect(ZkClientClusterStateProvider.java:160) ~[?:?] at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.getZkStateReader(BaseCloudSolrClient.java:279) ~[?:?] at org.apache.solr.client.solrj.impl.SolrClientCloudManager.<init>(SolrClientCloudManager.java:69) ~[?:?] at org.apache.solr.cloud.ZkController.getSolrCloudManager(ZkController.java:729) ~[?:?] at org.apache.solr.core.CoreContainer.createMetricsHistoryHandler(CoreContainer.java:820) ~[?:?] at org.apache.solr.core.CoreContainer.load(CoreContainer.java:641) ~[?:?] at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:255) ~[?:?] at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:175) ~[?:?] at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:136) ~[jetty-servlet-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.servlet.ServletHandler.lambda$initialize$0(ServletHandler.java:750) ~[jetty-servlet-9.4.14.v20181114.jar:9.4.14.v20181114] at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) ~[?:1.8.0_211] at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742) ~[?:1.8.0_211] at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742) ~[?:1.8.0_211] at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580) ~[?:1.8.0_211] at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:744) ~[jetty-servlet-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:368) ~[jetty-servlet-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1497) ~[jetty-webapp-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1459) ~[jetty-webapp-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:852) ~[jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:278) ~[jetty-servlet-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545) ~[jetty-webapp-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:46) ~[jetty-deploy-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:192) ~[jetty-deploy-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:505) ~[jetty-deploy-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:151) ~[jetty-deploy-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:180) ~[jetty-deploy-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:453) ~[jetty-deploy-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:64) ~[jetty-deploy-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:610) ~[jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:529) ~[jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.util.Scanner.scan(Scanner.java:392) ~[jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:313) ~[jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:150) ~[jetty-deploy-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:579) ~[jetty-deploy-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:240) ~[jetty-deploy-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138) ~[jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.server.Server.start(Server.java:415) ~[jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) ~[jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) ~[jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.server.Server.doStart(Server.java:382) ~[jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1572) ~[jetty-xml-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1512) ~[jetty-xml-9.4.14.v20181114.jar:9.4.14.v20181114] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_211] at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1511) ~[jetty-xml-9.4.14.v20181114.jar:9.4.14.v20181114] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_211] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_211] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_211] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_211] at org.eclipse.jetty.start.Main.invokeMain(Main.java:220) ~[start.jar:9.4.14.v20181114] at org.eclipse.jetty.start.Main.start(Main.java:490) ~[start.jar:9.4.14.v20181114] at org.eclipse.jetty.start.Main.main(Main.java:77) ~[start.jar:9.4.14.v20181114] 2019-06-11 10:04:53.566 INFO (zkConnectionManagerCallback-16-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected 2019-06-11 10:04:53.572 ERROR (main) [ ] o.a.s.c.c.ZkStateReader Error reading cluster properties from zookeeper => org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /clusterprops.json at org.apache.zookeeper.KeeperException.create(KeeperException.java:116) org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /clusterprops.json at org.apache.zookeeper.KeeperException.create(KeeperException.java:116) ~[?:?] at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[?:?] at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1221) ~[?:?] at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:358) ~[?:?] at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) ~[?:?] at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:358) ~[?:?] at org.apache.solr.common.cloud.ZkStateReader.loadClusterProperties(ZkStateReader.java:1054) ~[?:?] at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:463) ~[?:?] at org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.getZkStateReader(ZkClientClusterStateProvider.java:176) ~[?:?] at org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.connect(ZkClientClusterStateProvider.java:160) ~[?:?] at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.getZkStateReader(BaseCloudSolrClient.java:279) ~[?:?] at org.apache.solr.client.solrj.impl.SolrClientCloudManager.<init>(SolrClientCloudManager.java:69) ~[?:?] at org.apache.solr.cloud.ZkController.getSolrCloudManager(ZkController.java:729) ~[?:?] at org.apache.solr.core.CoreContainer.createMetricsHistoryHandler(CoreContainer.java:820) ~[?:?] at org.apache.solr.core.CoreContainer.load(CoreContainer.java:641) ~[?:?] at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:255) ~[?:?] at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:175) ~[?:?] at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:136) ~[jetty-servlet-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.servlet.ServletHandler.lambda$initialize$0(ServletHandler.java:750) ~[jetty-servlet-9.4.14.v20181114.jar:9.4.14.v20181114] at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) ~[?:1.8.0_211] at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742) ~[?:1.8.0_211] at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742) ~[?:1.8.0_211] at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580) ~[?:1.8.0_211] at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:744) ~[jetty-servlet-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:368) ~[jetty-servlet-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1497) ~[jetty-webapp-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1459) ~[jetty-webapp-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:852) ~[jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:278) ~[jetty-servlet-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545) ~[jetty-webapp-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:46) ~[jetty-deploy-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:192) ~[jetty-deploy-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:505) ~[jetty-deploy-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:151) ~[jetty-deploy-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:180) ~[jetty-deploy-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:453) ~[jetty-deploy-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:64) ~[jetty-deploy-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:610) ~[jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:529) ~[jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.util.Scanner.scan(Scanner.java:392) ~[jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:313) ~[jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:150) ~[jetty-deploy-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:579) ~[jetty-deploy-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:240) ~[jetty-deploy-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138) ~[jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.server.Server.start(Server.java:415) ~[jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) ~[jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) ~[jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.server.Server.doStart(Server.java:382) ~[jetty-server-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) ~[jetty-util-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1572) ~[jetty-xml-9.4.14.v20181114.jar:9.4.14.v20181114] at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1512) ~[jetty-xml-9.4.14.v20181114.jar:9.4.14.v20181114] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_211] at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1511) ~[jetty-xml-9.4.14.v20181114.jar:9.4.14.v20181114] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_211] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_211] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_211] at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_211] at org.eclipse.jetty.start.Main.invokeMain(Main.java:220) ~[start.jar:9.4.14.v20181114] at org.eclipse.jetty.start.Main.start(Main.java:490) ~[start.jar:9.4.14.v20181114] at org.eclipse.jetty.start.Main.main(Main.java:77) ~[start.jar:9.4.14.v20181114] 2019-06-11 10:04:53.579 ERROR (main) [ ] o.a.s.s.SolrDispatchFilter Could not start Solr. Check solr/home property and the logs 2019-06-11 10:04:53.598 ERROR (main) [ ] o.a.s.c.SolrCore null:org.apache.solr.common.cloud.ZooKeeperException: at org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.getZkStateReader(ZkClientClusterStateProvider.java:185) at org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.connect(ZkClientClusterStateProvider.java:160) at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.getZkStateReader(BaseCloudSolrClient.java:279) at org.apache.solr.client.solrj.impl.SolrClientCloudManager.<init>(SolrClientCloudManager.java:69) at org.apache.solr.cloud.ZkController.getSolrCloudManager(ZkController.java:729) at org.apache.solr.core.CoreContainer.createMetricsHistoryHandler(CoreContainer.java:820) at org.apache.solr.core.CoreContainer.load(CoreContainer.java:641) at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:255) at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:175) at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:136) at org.eclipse.jetty.servlet.ServletHandler.lambda$initialize$0(ServletHandler.java:750) at java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948) at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742) at java.util.stream.Streams$ConcatSpliterator.forEachRemaining(Streams.java:742) at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580) at org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:744) at org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:368) at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1497) at org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1459) at org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:852) at org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:278) at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:545) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:46) at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:192) at org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:505) at org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:151) at org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:180) at org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:453) at org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:64) at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:610) at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:529) at org.eclipse.jetty.util.Scanner.scan(Scanner.java:392) at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:313) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:150) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:579) at org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:240) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:138) at org.eclipse.jetty.server.Server.start(Server.java:415) at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) at org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:113) at org.eclipse.jetty.server.Server.doStart(Server.java:382) at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68) at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1572) at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1512) at java.security.AccessController.doPrivileged(Native Method) at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1511) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.eclipse.jetty.start.Main.invokeMain(Main.java:220) at org.eclipse.jetty.start.Main.start(Main.java:490) at org.eclipse.jetty.start.Main.main(Main.java:77) Caused by: org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /live_nodes at org.apache.zookeeper.KeeperException.create(KeeperException.java:116) at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1541) at org.apache.solr.common.cloud.SolrZkClient.lambda$getChildren$4(SolrZkClient.java:346) at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:346) at org.apache.solr.common.cloud.ZkStateReader.refreshLiveNodes(ZkStateReader.java:778) at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:464) at org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.getZkStateReader(ZkClientClusterStateProvider.java:176) ... 56 more 2019-06-11 10:04:53.613 INFO (main) [ ] o.e.j.s.h.ContextHandler Started o.e.j.w.WebAppContext@2cd2a21f{/solr,file:///C:/solr/server/solr-webapp/webapp/,AVAILABLE}{C:\solr\server/solr-webapp/webapp} 2019-06-11 10:04:53.622 INFO (main) [ ] o.e.j.s.AbstractConnector Started ServerConnector@9257031{HTTP/1.1,[http/1.1, h2c]}{0.0.0.0:8983} 2019-06-11 10:04:53.623 INFO (main) [ ] o.e.j.s.Server Started @12983ms 2019-06-11 10:04:58.006 INFO (zkCallback-8-thread-1) [ ] o.a.s.c.OverseerElectionContext I am going to be the leader localhost:8983_solr 2019-06-11 10:04:58.014 INFO (zkCallback-8-thread-1) [ ] o.a.s.c.Overseer Overseer (id=102252244093960205-localhost:8983_solr-n_0000000041) starting 2019-06-11 10:04:58.036 INFO (zkCallback-8-thread-1) [ ] o.a.s.c.c.SolrZkClient Using ZkCredentialsProvider: xxx.yyy.zzz.EncodedZkCredentialsProvider 2019-06-11 10:04:58.036 WARN (zkCallback-8-thread-1) [ ] o.a.s.c.c.SolrZkClient VM param zkCredentialsProvider does not point to a class implementing ZkCredentialsProvider and with a non-arg constructor => java.lang.ClassNotFoundException: xxx.yyy.zzz.EncodedZkCredentialsProvider at java.net.URLClassLoader.findClass(URLClassLoader.java:382) java.lang.ClassNotFoundException: xxx.yyy.zzz.EncodedZkCredentialsProvider at java.net.URLClassLoader.findClass(URLClassLoader.java:382) ~[?:1.8.0_211] at java.lang.ClassLoader.loadClass(ClassLoader.java:424) ~[?:1.8.0_211] at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_211] at org.eclipse.jetty.webapp.WebAppClassLoader.loadClass(WebAppClassLoader.java:556) ~[jetty-webapp-9.4.14.v20181114.jar:9.4.14.v20181114] at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ~[?:1.8.0_211] at java.lang.Class.forName0(Native Method) ~[?:1.8.0_211] at java.lang.Class.forName(Class.java:264) ~[?:1.8.0_211] at org.apache.solr.common.cloud.SolrZkClient.createZkCredentialsToAddAutomatically(SolrZkClient.java:225) ~[?:?] at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:144) ~[?:?] at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:126) ~[?:?] at org.apache.solr.common.cloud.SolrZkClient.<init>(SolrZkClient.java:116) ~[?:?] at org.apache.solr.common.cloud.ZkStateReader.<init>(ZkStateReader.java:309) ~[?:?] at org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.getZkStateReader(ZkClientClusterStateProvider.java:175) ~[?:?] at org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.connect(ZkClientClusterStateProvider.java:160) ~[?:?] at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.getZkStateReader(BaseCloudSolrClient.java:279) ~[?:?] at org.apache.solr.client.solrj.impl.SolrClientCloudManager.<init>(SolrClientCloudManager.java:69) ~[?:?] at org.apache.solr.cloud.ZkController.getSolrCloudManager(ZkController.java:729) ~[?:?] at org.apache.solr.cloud.Overseer.getSolrCloudManager(Overseer.java:724) ~[?:?] at org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.<init>(OverseerCollectionMessageHandler.java:207) ~[?:?] at org.apache.solr.cloud.OverseerCollectionConfigSetProcessor.getOverseerMessageHandlerSelector(OverseerCollectionConfigSetProcessor.java:87) ~[?:?] at org.apache.solr.cloud.OverseerCollectionConfigSetProcessor.<init>(OverseerCollectionConfigSetProcessor.java:70) ~[?:?] at org.apache.solr.cloud.OverseerCollectionConfigSetProcessor.<init>(OverseerCollectionConfigSetProcessor.java:41) ~[?:?] at org.apache.solr.cloud.Overseer.start(Overseer.java:575) ~[?:?] at org.apache.solr.cloud.OverseerElectionContext.runLeaderProcess(ElectionContext.java:739) ~[?:?] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:171) ~[?:?] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:136) ~[?:?] at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:57) ~[?:?] at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:349) ~[?:?] at org.apache.solr.common.cloud.SolrZkClient$3.lambda$process$1(SolrZkClient.java:287) ~[?:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_211] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_211] at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) ~[?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_211] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_211] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_211] 2019-06-11 10:04:58.042 INFO (zkConnectionManagerCallback-21-thread-1) [ ] o.a.s.c.c.ConnectionManager zkClient has connected 2019-06-11 10:04:58.045 ERROR (zkCallback-8-thread-1) [ ] o.a.s.c.c.ZkStateReader Error reading cluster properties from zookeeper => org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /clusterprops.json at org.apache.zookeeper.KeeperException.create(KeeperException.java:116) org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /clusterprops.json at org.apache.zookeeper.KeeperException.create(KeeperException.java:116) ~[?:?] at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[?:?] at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1221) ~[?:?] at org.apache.solr.common.cloud.SolrZkClient.lambda$getData$5(SolrZkClient.java:358) ~[?:?] at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) ~[?:?] at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:358) ~[?:?] at org.apache.solr.common.cloud.ZkStateReader.loadClusterProperties(ZkStateReader.java:1054) ~[?:?] at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:463) ~[?:?] at org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.getZkStateReader(ZkClientClusterStateProvider.java:176) ~[?:?] at org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.connect(ZkClientClusterStateProvider.java:160) ~[?:?] at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.getZkStateReader(BaseCloudSolrClient.java:279) ~[?:?] at org.apache.solr.client.solrj.impl.SolrClientCloudManager.<init>(SolrClientCloudManager.java:69) ~[?:?] at org.apache.solr.cloud.ZkController.getSolrCloudManager(ZkController.java:729) ~[?:?] at org.apache.solr.cloud.Overseer.getSolrCloudManager(Overseer.java:724) ~[?:?] at org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.<init>(OverseerCollectionMessageHandler.java:207) ~[?:?] at org.apache.solr.cloud.OverseerCollectionConfigSetProcessor.getOverseerMessageHandlerSelector(OverseerCollectionConfigSetProcessor.java:87) ~[?:?] at org.apache.solr.cloud.OverseerCollectionConfigSetProcessor.<init>(OverseerCollectionConfigSetProcessor.java:70) ~[?:?] at org.apache.solr.cloud.OverseerCollectionConfigSetProcessor.<init>(OverseerCollectionConfigSetProcessor.java:41) ~[?:?] at org.apache.solr.cloud.Overseer.start(Overseer.java:575) ~[?:?] at org.apache.solr.cloud.OverseerElectionContext.runLeaderProcess(ElectionContext.java:739) ~[?:?] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:171) ~[?:?] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:136) ~[?:?] at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:57) ~[?:?] at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:349) ~[?:?] at org.apache.solr.common.cloud.SolrZkClient$3.lambda$process$1(SolrZkClient.java:287) ~[?:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_211] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_211] at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) ~[?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_211] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_211] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_211] 2019-06-11 10:04:58.048 WARN (zkCallback-8-thread-1) [ ] o.a.s.c.LeaderElector => org.apache.solr.common.cloud.ZooKeeperException: at org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.getZkStateReader(ZkClientClusterStateProvider.java:185) org.apache.solr.common.cloud.ZooKeeperException: at org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.getZkStateReader(ZkClientClusterStateProvider.java:185) ~[?:?] at org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.connect(ZkClientClusterStateProvider.java:160) ~[?:?] at org.apache.solr.client.solrj.impl.BaseCloudSolrClient.getZkStateReader(BaseCloudSolrClient.java:279) ~[?:?] at org.apache.solr.client.solrj.impl.SolrClientCloudManager.<init>(SolrClientCloudManager.java:69) ~[?:?] at org.apache.solr.cloud.ZkController.getSolrCloudManager(ZkController.java:729) ~[?:?] at org.apache.solr.cloud.Overseer.getSolrCloudManager(Overseer.java:724) ~[?:?] at org.apache.solr.cloud.api.collections.OverseerCollectionMessageHandler.<init>(OverseerCollectionMessageHandler.java:207) ~[?:?] at org.apache.solr.cloud.OverseerCollectionConfigSetProcessor.getOverseerMessageHandlerSelector(OverseerCollectionConfigSetProcessor.java:87) ~[?:?] at org.apache.solr.cloud.OverseerCollectionConfigSetProcessor.<init>(OverseerCollectionConfigSetProcessor.java:70) ~[?:?] at org.apache.solr.cloud.OverseerCollectionConfigSetProcessor.<init>(OverseerCollectionConfigSetProcessor.java:41) ~[?:?] at org.apache.solr.cloud.Overseer.start(Overseer.java:575) ~[?:?] at org.apache.solr.cloud.OverseerElectionContext.runLeaderProcess(ElectionContext.java:739) ~[?:?] at org.apache.solr.cloud.LeaderElector.runIamLeaderProcess(LeaderElector.java:171) ~[?:?] at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:136) ~[?:?] at org.apache.solr.cloud.LeaderElector.access$200(LeaderElector.java:57) ~[?:?] at org.apache.solr.cloud.LeaderElector$ElectionWatcher.process(LeaderElector.java:349) ~[?:?] at org.apache.solr.common.cloud.SolrZkClient$3.lambda$process$1(SolrZkClient.java:287) ~[?:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_211] at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_211] at org.apache.solr.common.util.ExecutorUtil$MDCAwareThreadPoolExecutor.lambda$execute$0(ExecutorUtil.java:209) ~[?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_211] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_211] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_211] Caused by: org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode = NoAuth for /live_nodes at org.apache.zookeeper.KeeperException.create(KeeperException.java:116) ~[?:?] at org.apache.zookeeper.KeeperException.create(KeeperException.java:54) ~[?:?] at org.apache.zookeeper.ZooKeeper.getChildren(ZooKeeper.java:1541) ~[?:?] at org.apache.solr.common.cloud.SolrZkClient.lambda$getChildren$4(SolrZkClient.java:346) ~[?:?] at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:71) ~[?:?] at org.apache.solr.common.cloud.SolrZkClient.getChildren(SolrZkClient.java:346) ~[?:?] at org.apache.solr.common.cloud.ZkStateReader.refreshLiveNodes(ZkStateReader.java:778) ~[?:?] at org.apache.solr.common.cloud.ZkStateReader.createClusterStateWatchersAndUpdate(ZkStateReader.java:464) ~[?:?] at org.apache.solr.client.solrj.impl.ZkClientClusterStateProvider.getZkStateReader(ZkClientClusterStateProvider.java:176) ~[?:?] ... 22 more