Ethan Li created STORM-3728:
-------------------------------
Summary: Workers are not able to connect to Pacemaker if
pacemaker.auth.method is KERBEROS
Key: STORM-3728
URL: https://issues.apache.org/jira/browse/STORM-3728
Project: Apache Storm
Issue Type: Bug
Reporter: Ethan Li
When pacemaker.auth.method is KERBEROS, worker will fail to connect to
KERBEROS because of exceptions like the following:
{code:java}
2020-12-21 20:07:00.786 o.a.s.c.PaceMakerStateStorage executor-heartbeat-timer
[ERROR] Timed out waiting for channel ready. Failed to set_worker_hb. Will make
2 more attempts.
2020-12-21 20:07:00.902 o.a.s.m.n.KerberosSaslClientHandler
openstorm3blue-n10.blue.ygrid.yahoo.com-pm-1 [INFO] Connection established from
/10.215.73.209:45548 to
openstorm3blue-n10.blue.ygrid.yahoo.com/10.215.79.152:6699
2020-12-21 20:07:00.903 o.a.s.m.n.KerberosSaslNettyClient
openstorm3blue-n10.blue.ygrid.yahoo.com-pm-1 [INFO] Creating Kerberos Client.
2020-12-21 20:07:00.906 o.a.s.m.n.KerberosSaslNettyClient
openstorm3blue-n10.blue.ygrid.yahoo.com-pm-1 [INFO] Kerberos Client Callback
Handler got callback: class javax.security.auth.callback.PasswordCallback
2020-12-21 20:07:00.906 o.a.s.m.n.Login
openstorm3blue-n10.blue.ygrid.yahoo.com-pm-1 [ERROR] Login using jaas conf
/home/y/lib/storm/current/conf/storm_jaas.conf failed
2020-12-21 20:07:00.906 o.a.s.m.n.KerberosSaslNettyClient
openstorm3blue-n10.blue.ygrid.yahoo.com-pm-1 [ERROR] Client failed to login in
principal:javax.security.auth.login.LoginException: No password provided
javax.security.auth.login.LoginException: No password provided
at
com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:923)
~[?:1.8.0_262]
at
com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:764)
~[?:1.8.0_262]
at
com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:618)
~[?:1.8.0_262]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_262]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_262]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_262]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_262]
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
~[?:1.8.0_262]
at
javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
~[?:1.8.0_262]
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
~[?:1.8.0_262]
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
~[?:1.8.0_262]
at java.security.AccessController.doPrivileged(Native Method)
~[?:1.8.0_262]
at
javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
~[?:1.8.0_262]
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
~[?:1.8.0_262]
at org.apache.storm.messaging.netty.Login.login(Login.java:301)
~[storm-client-2.3.0.y.jar:2.3.0-SNAPSHOT]
at org.apache.storm.messaging.netty.Login.<init>(Login.java:83)
~[storm-client-2.3.0.y.jar:2.3.0-SNAPSHOT]
at
org.apache.storm.messaging.netty.KerberosSaslNettyClient.<init>(KerberosSaslNettyClient.java:66)
[storm-client-2.3.0.y.jar:2.3.0-SNAPSHOT]
at
org.apache.storm.messaging.netty.KerberosSaslClientHandler.channelActive(KerberosSaslClientHandler.java:59)
[storm-client-2.3.0.y.jar:2.3.0-SNAPSHOT]
at
org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:213)
[storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
at
org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:199)
[storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
at
org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:192)
[storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
at
org.apache.storm.shade.io.netty.channel.ChannelInboundHandlerAdapter.channelActive(ChannelInboundHandlerAdapter.java:64)
[storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
at
org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:213)
[storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
at
org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:199)
[storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
at
org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:192)
[storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
at
org.apache.storm.shade.io.netty.channel.DefaultChannelPipeline$HeadContext.channelActive(DefaultChannelPipeline.java:1422)
[storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
at
org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:213)
[storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
at
org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:199)
[storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
at
org.apache.storm.shade.io.netty.channel.DefaultChannelPipeline.fireChannelActive(DefaultChannelPipeline.java:941)
[storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
at
org.apache.storm.shade.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:311)
[storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
at
org.apache.storm.shade.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:341)
[storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
at
org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632)
[storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
at
org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
[storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
at
org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
[storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
at
org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
[storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
at
org.apache.storm.shade.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
[storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_262]
2020-12-21 20:07:00.907 o.a.s.m.n.KerberosSaslClientHandler
openstorm3blue-n10.blue.ygrid.yahoo.com-pm-1 [ERROR] Failed to authenticate
with server due to error:
java.lang.RuntimeException: javax.security.auth.login.LoginException: No
password provided
at
org.apache.storm.messaging.netty.KerberosSaslNettyClient.<init>(KerberosSaslNettyClient.java:71)
~[storm-client-2.3.0.y.jar:2.3.0-SNAPSHOT]
at
org.apache.storm.messaging.netty.KerberosSaslClientHandler.channelActive(KerberosSaslClientHandler.java:59)
[storm-client-2.3.0.y.jar:2.3.0-SNAPSHOT]
at
org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:213)
[storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
at
org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:199)
[storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
at
org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:192)
[storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
at
org.apache.storm.shade.io.netty.channel.ChannelInboundHandlerAdapter.channelActive(ChannelInboundHandlerAdapter.java:64)
[storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
at
org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:213)
[storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
at
org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:199)
[storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
at
org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.fireChannelActive(AbstractChannelHandlerContext.java:192)
[storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
at
org.apache.storm.shade.io.netty.channel.DefaultChannelPipeline$HeadContext.channelActive(DefaultChannelPipeline.java:1422)
[storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
at
org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:213)
[storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
at
org.apache.storm.shade.io.netty.channel.AbstractChannelHandlerContext.invokeChannelActive(AbstractChannelHandlerContext.java:199)
[storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
at
org.apache.storm.shade.io.netty.channel.DefaultChannelPipeline.fireChannelActive(DefaultChannelPipeline.java:941)
[storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
at
org.apache.storm.shade.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:311)
[storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
at
org.apache.storm.shade.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:341)
[storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
at
org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:632)
[storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
at
org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579)
[storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
at
org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496)
[storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
at
org.apache.storm.shade.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458)
[storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
at
org.apache.storm.shade.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
[storm-shaded-deps-2.3.0.y.jar:2.3.0-SNAPSHOT]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_262]
Caused by: javax.security.auth.login.LoginException: No password provided
at
com.sun.security.auth.module.Krb5LoginModule.promptForPass(Krb5LoginModule.java:923)
~[?:1.8.0_262]
at
com.sun.security.auth.module.Krb5LoginModule.attemptAuthentication(Krb5LoginModule.java:764)
~[?:1.8.0_262]
at
com.sun.security.auth.module.Krb5LoginModule.login(Krb5LoginModule.java:618)
~[?:1.8.0_262]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_262]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_262]
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_262]
at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_262]
at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
~[?:1.8.0_262]
at
javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
~[?:1.8.0_262]
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
~[?:1.8.0_262]
at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
~[?:1.8.0_262]
at java.security.AccessController.doPrivileged(Native Method)
~[?:1.8.0_262]
at
javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
~[?:1.8.0_262]
at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
~[?:1.8.0_262]
at org.apache.storm.messaging.netty.Login.login(Login.java:301)
~[storm-client-2.3.0.y.jar:2.3.0-SNAPSHOT]
at org.apache.storm.messaging.netty.Login.<init>(Login.java:83)
~[storm-client-2.3.0.y.jar:2.3.0-SNAPSHOT]
at
org.apache.storm.messaging.netty.KerberosSaslNettyClient.<init>(KerberosSaslNettyClient.java:66)
~[storm-client-2.3.0.y.jar:2.3.0-SNAPSHOT]
... 20 more
2020-12-21 20:07:01.802 o.a.s.p.PacemakerClient executor-heartbeat-timer
[ERROR] Error attempting to write to a channel to host
openstorm3blue-n10.blue.ygrid.yahoo.com - Timed out waiting for channel ready.
2020-12-21 20:07:01.803 o.a.s.p.PacemakerClient executor-heartbeat-timer [WARN]
Not getting response or getting null response. Making 9 more attempts for
openstorm3blue-n10.blue.ygrid.yahoo.com.
{code}
Currently by design
[https://github.com/apache/storm/blob/master/docs/Pacemaker.md#security]
pacemaker allows writes by anyone (which should be improved in the future).
So a quick fix is to make sure worker always has pacemaker.auth.method set to
NONE
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
