Jonathan Hurley created AMBARI-22655:
----------------------------------------
Summary: Livy/Livy2 Unable To Start Due to Address Already In Use
Key: AMBARI-22655
URL: https://issues.apache.org/jira/browse/AMBARI-22655
Project: Ambari
Issue Type: Bug
Affects Versions: 2.6.1
Reporter: Jonathan Hurley
Assignee: Jonathan Hurley
Priority: Critical
Fix For: 2.6.2
While restarting Livy and Livy2 on a non-root cluster, the following is seen:
{code}
17/12/14 14:36:23 WARN LivyConf: The configuration key
livy.repl.enableHiveContext has been deprecated as of Livy 0.4 and may be
removed in the future. Please use the new key livy.repl.enable-hive-context
instead.
17/12/14 14:36:23 WARN LivyConf: The configuration key
livy.server.csrf_protection.enabled has been deprecated as of Livy 0.4 and may
be removed in the future. Please use the new key
livy.server.csrf-protection.enabled instead.
17/12/14 14:36:23 INFO AccessManager: AccessControlManager acls disabled;users
with view permission: ;users with modify permission: ;users with super
permission: cstm-zeppelin;other allowed users: *
17/12/14 14:36:28 INFO LineBufferedStream: stdout: Welcome to
17/12/14 14:36:28 INFO LineBufferedStream: stdout: ____ __
17/12/14 14:36:28 INFO LineBufferedStream: stdout: / __/__ ___ _____/ /__
17/12/14 14:36:28 INFO LineBufferedStream: stdout: _\ \/ _ \/ _ `/ __/ '_/
17/12/14 14:36:28 INFO LineBufferedStream: stdout: /___/ .__/\_,_/_/ /_/\_\
version 2.2.0.2.6.4.0-73
17/12/14 14:36:28 INFO LineBufferedStream: stdout: /_/
17/12/14 14:36:28 INFO LineBufferedStream: stdout:
17/12/14 14:36:28 INFO LineBufferedStream: stdout: Using Scala version 2.11.8,
OpenJDK 64-Bit Server VM, 1.8.0_131
17/12/14 14:36:28 INFO LineBufferedStream: stdout: Branch HEAD
17/12/14 14:36:28 INFO LineBufferedStream: stdout: Compiled by user jenkins on
2017-12-13T19:08:32Z
17/12/14 14:36:28 INFO LineBufferedStream: stdout: Revision
a24017869f5450397136ee8b11be818e7cd3facb
17/12/14 14:36:28 INFO LineBufferedStream: stdout: Url
[email protected]:hortonworks/spark2.git
17/12/14 14:36:28 INFO LineBufferedStream: stdout: Type --help for more
information.
17/12/14 14:36:29 WARN NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
17/12/14 14:36:31 INFO AHSProxy: Connecting to Application History server at
nat-yc-r7-ovvs-ambari-autostart-4-re-2.openstacklocal/172.22.121.144:10200
17/12/14 14:36:32 WARN DomainSocketFactory: The short-circuit local reads
feature cannot be used because libhadoop cannot be loaded.
17/12/14 14:36:33 INFO StateStore$: Using FileSystemStateStore for recovery.
17/12/14 14:36:33 INFO BatchSessionManager: Recovered 0 batch sessions. Next
session id: 0
17/12/14 14:36:33 INFO InteractiveSessionManager: Recovered 0 interactive
sessions. Next session id: 0
17/12/14 14:36:33 INFO InteractiveSessionManager: Heartbeat watchdog thread
started.
17/12/14 14:36:33 INFO LivyServer: SPNEGO auth enabled (principal =
HTTP/nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklo...@example.com)
17/12/14 14:36:33 INFO LivyServer: CSRF protection is enabled.
17/12/14 14:36:34 INFO KerberosAuthenticationHandler: Login using keytab
/etc/security/keytabs/spnego.service.keytab, for principal
HTTP/nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklo...@example.com
17/12/14 14:36:34 INFO KerberosAuthenticationHandler: Map server:
nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklocal to principal:
[HTTP/nat-yc-r7-ovvs-ambari-autostart-4-re-3.openstacklo...@example.com], added
= true
17/12/14 14:36:34 WARN AbstractLifeCycle: FAILED
ServerConnector@df1cff6{SSL-http/1.1}{0.0.0.0:8999}: java.net.BindException:
Address already in use
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at
org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:321)
at
org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
at
org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:236)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.server.Server.doStart(Server.java:366)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.apache.livy.server.WebServer.start(WebServer.scala:92)
at org.apache.livy.server.LivyServer.start(LivyServer.scala:259)
at org.apache.livy.server.LivyServer$.main(LivyServer.scala:339)
at org.apache.livy.server.LivyServer.main(LivyServer.scala)
17/12/14 14:36:34 WARN AbstractLifeCycle: FAILED
org.eclipse.jetty.server.Server@60c16548: java.net.BindException: Address
already in use
java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at
org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:321)
at
org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
at
org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:236)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.server.Server.doStart(Server.java:366)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.apache.livy.server.WebServer.start(WebServer.scala:92)
at org.apache.livy.server.LivyServer.start(LivyServer.scala:259)
at org.apache.livy.server.LivyServer$.main(LivyServer.scala:339)
at org.apache.livy.server.LivyServer.main(LivyServer.scala)
Exception in thread "main" java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at
sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at
org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:321)
at
org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
at
org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:236)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.server.Server.doStart(Server.java:366)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.apache.livy.server.WebServer.start(WebServer.scala:92)
at org.apache.livy.server.LivyServer.start(LivyServer.scala:259)
at org.apache.livy.server.LivyServer$.main(LivyServer.scala:339)
at org.apache.livy.server.LivyServer.main(LivyServer.scala)
{code}
This occurs because the PID file cannot be accessed by the non-root agent and
returns an exit code of 1:
{code}
call returned (1, 'cat: /var/run/livy/livy-cstm-livy-server.pid: Permission
denied')
{code}
This tricks out PID detection into thinking that there is no process running
(or a missing PID file) and therefore, we do not need to stop Livy/Livy2.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
