Boris Gitline created SENTRY-2506:
-------------------------------------
Summary: Sentry starts but dysfunctional when Web Port is
unavailable
Key: SENTRY-2506
URL: https://issues.apache.org/jira/browse/SENTRY-2506
Project: Sentry
Issue Type: Bug
Components: Sentry
Reporter: Boris Gitline
Sentry clients (e.g., HMS) show errors like:
2019-02-26 14:04:13,451 ERROR
org.apache.sentry.core.common.transport.SentryTransportPool:
[pool-9-thread-131]: Failed to obtain transport for
bdtcstr96n16.svr.us.jpmchase.net:8038: null
2019-02-26 14:04:40,815 ERROR
org.apache.sentry.core.common.transport.SentryTransportPool:
[pool-9-thread-130]: Failed to obtain transport for
bdtcstr96n16.svr.us.jpmchase.net:8038: null
2019-02-26 14:04:43,816 ERROR
org.apache.sentry.binding.metastore.SentrySyncHMSNotificationsPostEventListener:
[pool-9-thread-130]: Encountered failure while notifying notification for
CREATE_DATABASE (id: 19)
org.apache.sentry.core.common.exception.SentryUserException:
java.net.SocketException: Connection reset
at
org.apache.sentry.core.common.transport.RetryClientInvocationHandler.connect(RetryClientInvocationHandler.java:166)
~[sentry-core-common-2.1.0-cdh6.1.1.jar:?]
at
org.apache.sentry.core.common.transport.RetryClientInvocationHandler.invokeImpl(RetryClientInvocationHandler.java:90)
~[sentry-core-common-2.1.0-cdh6.1.1.jar:?]
at
org.apache.sentry.core.common.transport.SentryClientInvocationHandler.invoke(SentryClientInvocationHandler.java:41)
~[sentry-core-common-2.1.0-cdh6.1.1.jar:?]
at com.sun.proxy.$Proxy34.notifyHmsEvent(Unknown Source) ~[?:?]
::::::
Sentry stderr log shows:
Exception in thread "main" java.util.concurrent.ExecutionException:
java.lang.Exception: Error starting server
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at
org.apache.sentry.service.thrift.SentryService$CommandImpl.run(SentryService.java:596)
at org.apache.sentry.SentryMain.main(SentryMain.java:120)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:313)
at org.apache.hadoop.util.RunJar.main(RunJar.java:227)
Caused by: java.lang.Exception: Error starting server
at org.apache.sentry.service.thrift.SentryService.call(SentryService.java:207)
at org.apache.sentry.service.thrift.SentryService.call(SentryService.java:81)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.security.PrivilegedActionException: java.net.BindException:
Address already in use
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.sentry.service.thrift.SentryService.call(SentryService.java:195)
... 5 more
Caused by: java.net.BindException: Address already in use
at sun.nio.ch.Net.bind0(Native Method)
at sun.nio.ch.Net.bind(Net.java:433)
at sun.nio.ch.Net.bind(Net.java:425)
at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74)
at org.eclipse.jetty.server.ServerConnector.open(ServerConnector.java:317)
at
org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:235)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.eclipse.jetty.server.Server.doStart(Server.java:401)
at
org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at
org.apache.sentry.api.service.thrift.SentryWebServer.start(SentryWebServer.java:190)
at
org.apache.sentry.service.thrift.SentryService.startSentryWebServer(SentryService.java:428)
at
org.apache.sentry.service.thrift.SentryService.runServer(SentryService.java:273)
at
org.apache.sentry.service.thrift.SentryService.access$000(SentryService.java:81)
at org.apache.sentry.service.thrift.SentryService$1.run(SentryService.java:198)
at org.apache.sentry.service.thrift.SentryService$1.run(SentryService.java:195)
... 8 more
port 29000 turns out to be taken by another application:
$ netstat -tulpn | grep 29000
tcp 0 0 0.0.0.0:29000 0.0.0.0:*
LISTEN 9510/aim-agent
. But Sentry still starts as if normal although the clients (Hive, Impala,
etc.) cannot run queries. That is extremely confusing. Sentry startup should be
corrected to fail to start in those situations with a clear error pointing at
the cause.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
