[
https://issues.apache.org/jira/browse/HBASE-28757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17868743#comment-17868743
]
Rushabh Shah commented on HBASE-28757:
--------------------------------------
I am looking at branch-2.6 code and related tests.
I am specifically looking at
+TestSaslTlsIPC#testRpcCallWithEnabledKerberosSaslAuth (serverTlsEnabled=true,
acceptPlainText=false, clientTlsEnabled=true)+
In this test case, since client and server Tls is enabled, I expected it to use
TLS auth but instead it is using kerberos auth.
See the logs below:
{noformat}
2024-07-23T10:05:19,526 INFO [Time-limited test {}] ipc.RpcServerFactory(64):
Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting
hbase.test.pb.TestProtobufRpcProto
2024-07-23T10:05:19,527 INFO [Time-limited test {}] ipc.NettyRpcServer(316):
Using org.apache.hbase.thirdparty.io.netty.buffer.PooledByteBufAllocator for
buffer allocation
2024-07-23T10:05:19,528 INFO [Time-limited test {}] ipc.NettyRpcServer(198):
Bind to /127.0.0.1:60307
2024-07-23T10:05:19,529 INFO [Time-limited test {}] ipc.FifoRpcScheduler(60):
Using FifoRpcScheduler as user call queue; handlerCount=1; maxQueueLength=10
2024-07-23T10:05:19,529 DEBUG [Time-limited test {}]
ipc.AbstractRpcClient(198):
Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@62d9f548, compressor=null,
tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=2000,
writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0,
fallbackAllowed=true, bind address=null
2024-07-23T10:05:19,531 DEBUG [Thread-14 {}] ipc.RpcConnection(159): Using
KERBEROS authentication for service=TestProtobufRpcProto, sasl=true
2024-07-23T10:05:19,532 DEBUG [RPCClient-NioEventLoopGroup-3-11 {}]
tls.X509Util(313): Using default JDK SSL provider because netty-tcnative is not
available
2024-07-23T10:05:19,533 DEBUG [RPCClient-NioEventLoopGroup-3-11 {}]
tls.X509Util(246): Using Java8 optimized cipher suites for Java version 1.8
2024-07-23T10:05:19,533 DEBUG [RPCClient-NioEventLoopGroup-3-11 {}]
ipc.NettyRpcConnection$3(383): SSL handler added with handshake timeout 5000 ms
2024-07-23T10:05:19,535 DEBUG [NettyRpcServer-13-2 {}] tls.X509Util(313): Using
default JDK SSL provider because netty-tcnative is not available
2024-07-23T10:05:19,536 DEBUG [NettyRpcServer-13-2 {}] tls.X509Util(246): Using
Java8 optimized cipher suites for Java version 1.8
2024-07-23T10:05:19,537 DEBUG [NettyRpcServer-13-2 {}] ipc.NettyRpcServer(447):
SSL handler added for channel: [id: 0xd5e8d21d, L:/127.0.0.1:60307 -
R:localhost/127.0.0.1:60308]
2024-07-23T10:05:19,561 DEBUG [RPCClient-NioEventLoopGroup-3-11 {}]
provider.GssSaslClientAuthenticationProvider(47): Setting up Kerberos RPC to
server=hbase/[email protected]
2024-07-23T10:05:19,579 DEBUG [NettyRpcServer-13-2 {}]
ipc.NettyHBaseSaslRpcServerHandler(55): Read input token of size=516 for
processing by saslServer.evaluateResponse()
2024-07-23T10:05:19,579 DEBUG [NettyRpcServer-13-2 {}]
provider.GssSaslServerAuthenticationProvider(50): Server's Kerberos principal
name is hbase/[email protected]
2024-07-23T10:05:19,581 DEBUG [NettyRpcServer-13-2 {}]
ipc.NettyHBaseSaslRpcServerHandler(62): Will send token of size 108 from
saslServer.
2024-07-23T10:05:19,583 DEBUG [NettyRpcServer-13-2 {}]
ipc.NettyHBaseSaslRpcServerHandler(55): Read input token of size=0 for
processing by saslServer.evaluateResponse()
2024-07-23T10:05:19,583 DEBUG [NettyRpcServer-13-2 {}]
ipc.NettyHBaseSaslRpcServerHandler(62): Will send token of size 32 from
saslServer.
2024-07-23T10:05:19,584 DEBUG [RPCClient-NioEventLoopGroup-3-11 {}]
ipc.RpcConnection(322): sasl negotiation done with serverPrincipal =
hbase/[email protected], succeed = true
2024-07-23T10:05:19,584 DEBUG [NettyRpcServer-13-2 {}]
ipc.NettyHBaseSaslRpcServerHandler(55): Read input token of size=32 for
processing by saslServer.evaluateResponse()
2024-07-23T10:05:19,584 DEBUG [NettyRpcServer-13-2 {}]
provider.GssSaslServerAuthenticationProvider$SaslGssCallbackHandler(95): SASL
server GSSAPI callback: setting canonicalized client ID:
hbase/[email protected]
2024-07-23T10:05:19,584 DEBUG [NettyRpcServer-13-2 {}]
ipc.ServerRpcConnection(356): SASL server context established. Authenticated
client: hbase/[email protected] (auth:KERBEROS). Negotiated QoP is auth
2024-07-23T10:05:19,584 INFO [NettyRpcServer-13-2 {}]
ipc.ServerRpcConnection(359): Auth successful for hbase/[email protected]
(auth:KERBEROS)
2024-07-23T10:05:19,585 INFO [NettyRpcServer-13-2 {}]
ipc.ServerRpcConnection(484): Connection from 127.0.0.1:60308,
version=2.6.1-SNAPSHOT, sasl=true, ugi=hbase/[email protected]
(auth:KERBEROS), service=TestProtobufRpcProto
2024-07-23T10:05:19,597 DEBUG [Time-limited test {}]
ipc.AbstractRpcClient(514): Stopping rpc client
2024-07-23T10:05:19,597 INFO [Time-limited test {}] ipc.NettyRpcServer(352):
Stopping server on localhost/127.0.0.1:60307
{noformat}
> Understand how supportplaintext property works in TLS setup.
> ------------------------------------------------------------
>
> Key: HBASE-28757
> URL: https://issues.apache.org/jira/browse/HBASE-28757
> Project: HBase
> Issue Type: Improvement
> Components: security
> Affects Versions: 2.6.0
> Reporter: Rushabh Shah
> Priority: Major
>
> We are testing TLS feature and I am confused on how
> hbase.server.netty.tls.supportplaintext property works.
> Here is our current setup. This is a fresh cluster deployment.
> hbase.server.netty.tls.enabled --> true
> hbase.client.netty.tls.enabled --> true
> hbase.server.netty.tls.supportplaintext --> false (We don't want to fallback
> on kerberos)
> We still have our kerberos related configuration enabled.
> hbase.security.authentication --> kerberos
> *Our expectation:*
> During regionserver startup, regionserver will use TLS for authentication and
> the communication will succeed.
> *Actual observation*
> During regionserver startup, hmaster authenticates regionserver* via kerberos
> authentication*and *regionserver's reportForDuty RPC fails*.
> RS logs:
> {noformat}
> 2024-07-25 16:59:55,098 INFO [regionserver/regionserver-0:60020]
> regionserver.HRegionServer - reportForDuty to
> master=hmaster-0,60000,1721926791062 with
> isa=regionserver-0/<rs-ip-address>:60020, startcode=1721926793434
> 2024-07-25 16:59:55,548 DEBUG [RS-EventLoopGroup-1-2] ssl.SslHandler - [id:
> 0xa48e3487, L:/<rs-ip-address>:39837 -
> R:hmaster-0/<hmaster-ip-address>:60000] HANDSHAKEN: protocol:TLSv1.2 cipher
> suite:TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
> 2024-07-25 16:59:55,578 DEBUG [RS-EventLoopGroup-1-2]
> security.UserGroupInformation - PrivilegedAction [as: hbase/regionserver-0.
> (auth:KERBEROS)][action:
> org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler$2@3769e55]
> java.lang.Exception
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1896)
> at
> org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler.channelRead0(NettyHBaseSaslRpcClientHandler.java:161)
> at
> org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler.channelRead0(NettyHBaseSaslRpcClientHandler.java:43)
> ...
> ...
> 2024-07-25 16:59:55,581 DEBUG [RS-EventLoopGroup-1-2]
> security.UserGroupInformation - PrivilegedAction [as: hbase/regionserver-0
> (auth:KERBEROS)][action:
> org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler$2@c6f0806]
> java.lang.Exception
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1896)
> at
> org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler.channelRead0(NettyHBaseSaslRpcClientHandler.java:161)
> at
> org.apache.hadoop.hbase.security.NettyHBaseSaslRpcClientHandler.channelRead0(NettyHBaseSaslRpcClientHandler.java:43)
> at
> org.apache.hbase.thirdparty.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
> 2024-07-25 16:59:55,602 WARN [regionserver/regionserver-0:60020]
> regionserver.HRegionServer - error telling master we are up
> org.apache.hbase.thirdparty.com.google.protobuf.ServiceException:
> org.apache.hadoop.hbase.exceptions.ConnectionClosedException: Call to
> address=hmaster-0:60000 failed on local exception:
> org.apache.hadoop.hbase.exceptions.ConnectionClosedException: Connection
> closed
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:340)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:92)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:595)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:16398)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2997)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.lambda$run$2(HRegionServer.java:1084)
> at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:187)
> at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:177)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:1079)
> Caused by: org.apache.hadoop.hbase.exceptions.ConnectionClosedException: Call
> to address=hmaster-0:60000 failed on local exception:
> org.apache.hadoop.hbase.exceptions.ConnectionClosedException: Connection
> closed
> at org.apache.hadoop.hbase.ipc.IPCUtil.wrapException(IPCUtil.java:233)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:391)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:92)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:425)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:420)
> at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:114)
> at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:129)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.cleanupCalls(NettyRpcDuplexHandler.java:231)
> at
> org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelInactive(NettyRpcDuplexHandler.java:239)
> at
> org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:303)
> {noformat}
> Hmaster logs
> {noformat}
> 2024-07-25 16:59:55,378 DEBUG [RS-EventLoopGroup-1-2] ipc.NettyRpcServer -
> SSL handler added for channel: [id: 0xd4071764, L:/<hmaster-ip>:60000 -
> R:regionserver-0/<rs-ip>:39837]
> 2024-07-25 16:59:55,526 DEBUG [RS-EventLoopGroup-1-2] ssl.SslHandler - [id:
> 0xd4071764, L:/<hmaster-ip>:60000 - R:regionserver-0/<rs-ip>:39837]
> HANDSHAKEN: protocol:TLSv1.2 cipher
> suite:TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
> 2024-07-25 16:59:55,583 INFO [RS-EventLoopGroup-1-2] hbase.Server - Auth
> successful for hbase/regionserver-0 (auth:KERBEROS)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
