It looks as though all four nodes are trying to register with ZK using the same
hostname and port number - possibly they're all connecting as 'localhost'?
Alan Woodward
www.flax.co.uk
On 31 Aug 2016, at 09:34, Chris Rogers wrote:
> Just pinging this again as I sent it late last night. Would be great if
> someone could help with this. It's got me totally stumped...
>
>
> Chris Rogers
> Digital Projects Manager
> Bodleian Digital Library Systems and Services
> chris.rog...@bodleian.ox.ac.uk
>
> ________________________________________
> From: Chris Rogers [chris.rog...@bodleian.ox.ac.uk]
> Sent: 30 August 2016 21:34
> To: solr-user@lucene.apache.org
> Subject: SolrCloud: A previous ephemeral live node still exists
>
> Hi all,
>
> I'm trying to create a SolrCloud setup with Vagrant boxes using Solr 6.2.0
> and Zookeeper 3.4.8
>
> I managed to get this to work perfectly with Solr 6.1.0, but I'm not able to
> start more than one node in cloud mode with Solr 6.2.0.
>
> I have four VMs connected on a private network of Vagrant boxes, one running
> Zookeeper, and three Solr nodes.
>
> The first Solr node connects to Zookeeper as expected. I can access the admin
> and see the Cloud info there.
>
> But when I try and connect with a second node, something goes wrong.
>
> I get the following error in the Solr log:
>
>
> 2016-08-30 20:12:12.292 INFO (main) [ ] o.e.j.u.log Logging initialized
> @490ms
>
> 2016-08-30 20:12:12.551 INFO (main) [ ] o.e.j.s.Server
> jetty-9.3.8.v20160314
>
> 2016-08-30 20:12:12.574 INFO (main) [ ] o.e.j.d.p.ScanningAppProvider
> Deployment monitor [file:///home/vagrant/solr-6.2.0/server/contexts/] at
> interval 0
>
> 2016-08-30 20:12:12.906 INFO (main) [ ]
> o.e.j.w.StandardDescriptorProcessor NO JSP Support for /solr, did not find
> org.apache.jasper.servlet.JspServlet
>
> 2016-08-30 20:12:12.923 WARN (main) [ ] o.e.j.s.SecurityHandler
> ServletContext@o.e.j.w.WebAppContext@5383967b{/solr,file:///home/vagrant/solr-6.2.0/server/solr-webapp/webapp/,STARTING}{/home/vagrant/solr-6.2.0/server/solr-webapp/webapp}
> has uncovered http methods for path: /
>
> 2016-08-30 20:12:12.936 INFO (main) [ ] o.a.s.s.SolrDispatchFilter
> SolrDispatchFilter.init(): WebAppClassLoader=1465085305@57536d79
>
> 2016-08-30 20:12:12.962 INFO (main) [ ] o.a.s.c.SolrResourceLoader JNDI
> not configured for solr (NoInitialContextEx)
>
> 2016-08-30 20:12:12.962 INFO (main) [ ] o.a.s.c.SolrResourceLoader using
> system property solr.solr.home: /home/vagrant/solr-6.2.0/server/solr
>
> 2016-08-30 20:12:12.963 INFO (main) [ ] o.a.s.c.SolrResourceLoader new
> SolrResourceLoader for directory: '/home/vagrant/solr-6.2.0/server/solr'
>
> 2016-08-30 20:12:12.963 INFO (main) [ ] o.a.s.c.SolrResourceLoader JNDI
> not configured for solr (NoInitialContextEx)
>
> 2016-08-30 20:12:12.963 INFO (main) [ ] o.a.s.c.SolrResourceLoader using
> system property solr.solr.home: /home/vagrant/solr-6.2.0/server/solr
>
> 2016-08-30 20:12:12.987 INFO (main) [ ] o.a.s.c.c.SolrZkClient Using
> default ZkCredentialsProvider
>
> 2016-08-30 20:12:13.017 INFO (main) [ ] o.a.s.c.c.ConnectionManager
> Waiting for client to connect to ZooKeeper
>
> 2016-08-30 20:12:13.118 INFO (zkCallback-1-thread-1) [ ]
> o.a.s.c.c.ConnectionManager Watcher
> org.apache.solr.common.cloud.ConnectionManager@25b8acce
> name:ZooKeeperConnection Watcher:172.28.128.3:2181 got event WatchedEvent
> state:SyncConnected type:None path:null path:null type:None
>
> 2016-08-30 20:12:13.119 INFO (main) [ ] o.a.s.c.c.ConnectionManager Client
> is connected to ZooKeeper
>
> 2016-08-30 20:12:13.119 INFO (main) [ ] o.a.s.c.c.SolrZkClient Using
> default ZkACLProvider
>
> 2016-08-30 20:12:13.130 INFO (main) [ ] o.a.s.s.SolrDispatchFilter Loading
> solr.xml from SolrHome (not found in ZooKeeper)
>
> 2016-08-30 20:12:13.133 INFO (main) [ ] o.a.s.c.SolrXmlConfig Loading
> container configuration from /home/vagrant/solr-6.2.0/server/solr/solr.xml
>
> 2016-08-30 20:12:13.233 INFO (main) [ ] o.a.s.c.CorePropertiesLocator
> Config-defined core root directory: /home/vagrant/solr-6.2.0/server/solr
>
> 2016-08-30 20:12:13.264 INFO (main) [ ] o.a.s.c.CoreContainer New
> CoreContainer 182259421
>
> 2016-08-30 20:12:13.265 INFO (main) [ ] o.a.s.c.CoreContainer Loading
> cores into CoreContainer [instanceDir=/home/vagrant/solr-6.2.0/server/solr]
>
> 2016-08-30 20:12:13.266 WARN (main) [ ] o.a.s.c.CoreContainer Couldn't add
> files from /home/vagrant/solr-6.2.0/server/solr/lib to classpath:
> /home/vagrant/solr-6.2.0/server/solr/lib
>
> 2016-08-30 20:12:13.281 INFO (main) [ ] o.a.s.h.c.HttpShardHandlerFactory
> created with socketTimeout : 600000,connTimeout : 60000,maxConnectionsPerHost
> : 20,maxConnections : 10000,corePoolSize : 0,maximumPoolSize :
> 2147483647,maxThreadIdleTime : 5,sizeOfQueue : -1,fairnessPolicy :
> false,useRetries : false,connectionsEvictorSleepDelay :
> 5000,maxConnectionIdleTime : 40000,
>
> 2016-08-30 20:12:13.507 INFO (main) [ ] o.a.s.u.UpdateShardHandler
> Creating UpdateShardHandler HTTP client with params:
> socketTimeout=600000&connTimeout=60000&retry=true
>
> 2016-08-30 20:12:13.511 INFO (main) [ ] o.a.s.l.LogWatcher SLF4J impl is
> org.slf4j.impl.Log4jLoggerFactory
>
> 2016-08-30 20:12:13.512 INFO (main) [ ] o.a.s.l.LogWatcher Registering Log
> Listener [Log4j (org.slf4j.impl.Log4jLoggerFactory)]
>
> 2016-08-30 20:12:13.519 INFO (main) [ ] o.a.s.c.ZkContainer Zookeeper
> client=172.28.128.3:2181
>
> 2016-08-30 20:12:13.535 INFO (main) [ ] o.a.s.c.ZkController Added new
> OnReconnect listener
> org.apache.solr.cloud.ZkController$$Lambda$5/306612792@1aa7ecca
>
> 2016-08-30 20:12:13.538 INFO (main) [ ] o.a.s.c.c.ConnectionManager
> Waiting for client to connect to ZooKeeper
>
> 2016-08-30 20:12:13.549 INFO
> (zkCallback-4-thread-1-processing-n:10.0.2.15:8983_solr) [ ]
> o.a.s.c.c.ConnectionManager Watcher
> org.apache.solr.common.cloud.ConnectionManager@6ece6797
> name:ZooKeeperConnection Watcher:172.28.128.3:2181 got event WatchedEvent
> state:SyncConnected type:None path:null path:null type:None
>
> 2016-08-30 20:12:13.549 INFO (main) [ ] o.a.s.c.c.ConnectionManager Client
> is connected to ZooKeeper
>
> 2016-08-30 20:12:13.620 INFO (main) [ ] o.a.s.c.c.ZkStateReader Updating
> cluster state from ZooKeeper...
>
> 2016-08-30 20:12:13.627 INFO (main) [ ] o.a.s.c.c.ZkStateReader Loaded
> empty cluster properties
>
> 2016-08-30 20:12:13.634 INFO (main) [ ] o.a.s.c.c.ZkStateReader Updated
> live nodes from ZooKeeper... (0) -> (1)
>
> 2016-08-30 20:12:43.674 ERROR (main) [ ] o.a.s.s.SolrDispatchFilter Could
> not start Solr. Check solr/home property and the logs
>
> 2016-08-30 20:12:43.741 ERROR (main) [ ] o.a.s.c.SolrCore
> null:org.apache.solr.common.SolrException: A previous ephemeral live node
> still exists. Solr cannot continue. Please ensure that no other Solr process
> using the same port is running already.
>
> at
> org.apache.solr.cloud.ZkController.checkForExistingEphemeralNode(ZkController.java:741)
>
> at org.apache.solr.cloud.ZkController.init(ZkController.java:675)
>
> at org.apache.solr.cloud.ZkController.<init>(ZkController.java:419)
>
> at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:112)
>
> at org.apache.solr.core.CoreContainer.load(CoreContainer.java:437)
>
> at
> org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:158)
>
> at
> org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:134)
>
> at org.eclipse.jetty.servlet.FilterHolder.initialize(FilterHolder.java:137)
>
> at
> org.eclipse.jetty.servlet.ServletHandler.initialize(ServletHandler.java:856)
>
> at
> org.eclipse.jetty.servlet.ServletContextHandler.startContext(ServletContextHandler.java:348)
>
> at org.eclipse.jetty.webapp.WebAppContext.startWebapp(WebAppContext.java:1379)
>
> at
> org.eclipse.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1341)
>
> at
> org.eclipse.jetty.server.handler.ContextHandler.doStart(ContextHandler.java:772)
>
> at
> org.eclipse.jetty.servlet.ServletContextHandler.doStart(ServletContextHandler.java:261)
>
> at org.eclipse.jetty.webapp.WebAppContext.doStart(WebAppContext.java:517)
>
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>
> at
> org.eclipse.jetty.deploy.bindings.StandardStarter.processBinding(StandardStarter.java:41)
>
> at org.eclipse.jetty.deploy.AppLifeCycle.runBindings(AppLifeCycle.java:188)
>
> at
> org.eclipse.jetty.deploy.DeploymentManager.requestAppGoal(DeploymentManager.java:499)
>
> at
> org.eclipse.jetty.deploy.DeploymentManager.addApp(DeploymentManager.java:147)
>
> at
> org.eclipse.jetty.deploy.providers.ScanningAppProvider.fileAdded(ScanningAppProvider.java:180)
>
> at
> org.eclipse.jetty.deploy.providers.WebAppProvider.fileAdded(WebAppProvider.java:458)
>
> at
> org.eclipse.jetty.deploy.providers.ScanningAppProvider$1.fileAdded(ScanningAppProvider.java:64)
>
> at org.eclipse.jetty.util.Scanner.reportAddition(Scanner.java:610)
>
> at org.eclipse.jetty.util.Scanner.reportDifferences(Scanner.java:529)
>
> at org.eclipse.jetty.util.Scanner.scan(Scanner.java:392)
>
> at org.eclipse.jetty.util.Scanner.doStart(Scanner.java:313)
>
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>
> at
> org.eclipse.jetty.deploy.providers.ScanningAppProvider.doStart(ScanningAppProvider.java:150)
>
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>
> at
> org.eclipse.jetty.deploy.DeploymentManager.startAppProvider(DeploymentManager.java:561)
>
> at
> org.eclipse.jetty.deploy.DeploymentManager.doStart(DeploymentManager.java:236)
>
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:132)
>
> at org.eclipse.jetty.server.Server.start(Server.java:405)
>
> at
> org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:114)
>
> at
> org.eclipse.jetty.server.handler.AbstractHandler.doStart(AbstractHandler.java:61)
>
> at org.eclipse.jetty.server.Server.doStart(Server.java:372)
>
> at
> org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
>
> at org.eclipse.jetty.xml.XmlConfiguration$1.run(XmlConfiguration.java:1510)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at org.eclipse.jetty.xml.XmlConfiguration.main(XmlConfiguration.java:1435)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:498)
>
> at org.eclipse.jetty.start.Main.invokeMain(Main.java:214)
>
> at org.eclipse.jetty.start.Main.start(Main.java:457)
>
> at org.eclipse.jetty.start.Main.main(Main.java:75)
>
>
> 2016-08-30 20:12:43.744 INFO (main) [ ] o.a.s.s.SolrDispatchFilter
> SolrDispatchFilter.init() done
>
> 2016-08-30 20:12:43.762 INFO (main) [ ] o.e.j.s.h.ContextHandler Started
> o.e.j.w.WebAppContext@5383967b{/solr,file:///home/vagrant/solr-6.2.0/server/solr-webapp/webapp/,AVAILABLE}{/home/vagrant/solr-6.2.0/server/solr-webapp/webapp}
>
> 2016-08-30 20:12:43.771 INFO (main) [ ] o.e.j.s.ServerConnector Started
> ServerConnector@305b7c14{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}
>
> 2016-08-30 20:12:43.771 INFO (main) [ ] o.e.j.s.Server Started @31972ms
>
> It does start in the end, but it's not working properly, I just get errors in
> the Solr admin.
>
> The only reference I can find to the the error:
>
> "A previous ephemeral live node still exists. Solr cannot continue. Please
> ensure that no other Solr process using the same port is running already."
>
> ,was this Jira ticket, but it suggests that the issue is resolved in 6.2.0
>
> https://issues.apache.org/jira/browse/SOLR-8777
>
> Has anyone else experienced this issue? Any hints on resolving?
>
>
> Chris Rogers
> Digital Projects Manager
> Bodleian Digital Library Systems and Services
> chris.rog...@bodleian.ox.ac.uk<mailto://chris.rog...@bodleian.ox.ac.uk>
