Greg Senia created HBASE-21982:
----------------------------------
Summary: HBase Kerberos with no Hadoop/HDFS fails on startup
Key: HBASE-21982
URL: https://issues.apache.org/jira/browse/HBASE-21982
Project: HBase
Issue Type: Bug
Components: master, regionserver, rpc
Affects Versions: 1.4.9, 1.4.2
Reporter: Greg Senia
When attempting to Kerberize an HBase Instance that uses the localFS without
Hadoop I noticed that instead of the HBase RegionServer Successfully checking
in with the HBase Master it fails stating that it was using SIMPLE
authentication vs Kerberos. So I think the real question here is does HBase
support running without HDFS/Hadoop for the filesystem in Kerberos Mode or is
HDFS required?
Error on RegionServer:
3-02 13:09:46,314 DEBUG [regionserver/owlms.hdp.senia.org/10.69.68.21:16020]
ipc.BlockingRpcConnection: Connecting to owlms.hdp.senia.org/10.69.68.21:16000
2019-03-02 13:09:46,315 DEBUG
[regionserver/owlms.hdp.senia.org/10.69.68.21:16020]
security.UserGroupInformation: PrivilegedAction as:hbase (auth:SIMPLE)
from:org.apache.hadoop.hbase.ipc.BlockingRpcConnection.setupIOstreams(BlockingRpcConnection.java:452)
2019-03-02 13:09:46,315 DEBUG
[regionserver/owlms.hdp.senia.org/10.69.68.21:16020]
security.AbstractHBaseSaslRpcClient: Creating SASL GSSAPI client. Server's
Kerberos principal name is hbase/[email protected]
2019-03-02 13:09:46,318 DEBUG
[regionserver/owlms.hdp.senia.org/10.69.68.21:16020]
security.HBaseSaslRpcClient: Have sent token of size 635 from initSASLContext.
2019-03-02 13:09:46,318 DEBUG
[regionserver/owlms.hdp.senia.org/10.69.68.21:16020]
security.UserGroupInformation: PrivilegedActionException as:hbase (auth:SIMPLE)
cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.security.AccessDeniedException):
Kerberos principal name does NOT have the expected hostname part: hbase
2019-03-02 13:09:46,319 DEBUG
[regionserver/owlms.hdp.senia.org/10.69.68.21:16020]
security.UserGroupInformation: PrivilegedAction as:hbase (auth:SIMPLE)
from:org.apache.hadoop.hbase.ipc.BlockingRpcConnection.handleSaslConnectionFailure(BlockingRpcConnection.java:374)
2019-03-02 13:09:46,319 WARN
[regionserver/owlms.hdp.senia.org/10.69.68.21:16020] ipc.BlockingRpcConnection:
Exception encountered while connecting to the server :
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.security.AccessDeniedException):
Kerberos principal name does NOT have the expected hostname part: hbase
2019-03-02 13:09:46,319 DEBUG
[regionserver/owlms.hdp.senia.org/10.69.68.21:16020]
security.UserGroupInformation: PrivilegedActionException as:hbase (auth:SIMPLE)
cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.security.AccessDeniedException):
Kerberos principal name does NOT have the expected hostname part: hbase
2019-03-02 13:09:46,319 DEBUG
[regionserver/owlms.hdp.senia.org/10.69.68.21:16020] ipc.FailedServers: Added
failed server with address owlms.hdp.senia.org/10.69.68.21:16000 to list caused
by
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.security.AccessDeniedException):
Kerberos principal name does NOT have the expected hostname part: hbase
2019-03-02 13:09:46,319 WARN
[regionserver/owlms.hdp.senia.org/10.69.68.21:16020]
regionserver.HRegionServer: error telling master we are up
com.google.protobuf.ServiceException:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.security.AccessDeniedException):
Kerberos principal name does NOT have the expected hostname part: hbase
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:335)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:94)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:571)
at
org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.regionServerStartup(RegionServerStatusProtos.java:8982)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.reportForDuty(HRegionServer.java:2431)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:969)
at java.lang.Thread.run(Thread.java:748)
Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hbase.security.AccessDeniedException):
Kerberos principal name does NOT have the expected hostname part: hbase
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:386)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:94)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:409)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:405)
at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:103)
at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:118)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:422)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:327)
... 6 more
Error on HBase Master:
2019-03-02 14:14:13,593 DEBUG
[RpcServer.reader=3,bindAddress=owlms.hdp.senia.org,port=16000] ipc.RpcServer:
RpcServer.listener,port=16000: DISCONNECTING client 10.69.68.21:35620 because
read count=-1. Number of active connections: 1
2019-03-02 14:14:14,615 INFO [owlms:16000.activeMasterManager]
master.ServerManager: Waiting on RegionServer count=0 to settle;
waited=10325215ms, expecting min=1 server(s), max=NO_LIMIT server(s),
timeout=4500ms, lastChange=-10325215ms
2019-03-02 14:14:15,828 DEBUG
[master/owlms.hdp.senia.org/10.69.68.21:16000-SendThread(owlms.hdp.senia.org:2181)]
zookeeper.ClientCnxn: Got ping response for sessionid: 0x1693caa4d83009b after
0ms
2019-03-02 14:14:15,928 DEBUG
[owlms:16000.activeMasterManager-SendThread(owlms.hdp.senia.org:2181)]
zookeeper.ClientCnxn: Got ping response for sessionid: 0x1693caa4d83009c after
0ms
2019-03-02 14:14:16,119 INFO [owlms:16000.activeMasterManager]
master.ServerManager: Waiting on RegionServer count=0 to settle;
waited=10326719ms, expecting min=1 server(s), max=NO_LIMIT server(s),
timeout=4500ms, lastChange=-10326719ms
2019-03-02 14:14:16,590 DEBUG
[owlms:16000.activeMasterManager-SendThread(owlms.hdp.senia.org:2181)]
zookeeper.ClientCnxn: Got ping response for sessionid: 0x1693caa4d83009e after
0ms
2019-03-02 14:14:16,595 DEBUG [RpcServer.listener,port=16000] ipc.RpcServer:
RpcServer.listener,port=16000: connection from 10.69.68.21:52423; # active
connections: 1
2019-03-02 14:14:16,598 DEBUG
[RpcServer.reader=4,bindAddress=owlms.hdp.senia.org,port=16000] ipc.RpcServer:
Kerberos principal name is hbase
2019-03-02 14:14:16,598 DEBUG
[RpcServer.reader=4,bindAddress=owlms.hdp.senia.org,port=16000] ipc.RpcServer:
RpcServer.listener,port=16000: Caught exception while reading:
org.apache.hadoop.hbase.security.AccessDeniedException: Kerberos principal name
does NOT have the expected hostname part: hbase
at
org.apache.hadoop.hbase.ipc.RpcServer$Connection.saslReadAndProcess(RpcServer.java:1468)
at
org.apache.hadoop.hbase.ipc.RpcServer$Connection.process(RpcServer.java:1788)
at
org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1769)
at
org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:955)
at
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:725)
at
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:701)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2019-03-02 14:14:16,598 DEBUG
[RpcServer.reader=4,bindAddress=owlms.hdp.senia.org,port=16000] ipc.RpcServer:
RpcServer.listener,port=16000: DISCONNECTING client 10.69.68.21:52423 because
read count=-1. Number of active connections: 1
2019-03-02 14:14:17,622 INFO [owlms:16000.activeMasterManager]
master.ServerManager: Waiting on RegionServer count=0 to settle;
waited=10328222ms, expecting min=1 server(s), max=NO_LIMIT server(s),
timeout=4500ms, lastChange=-10328222ms
2019-03-02 14:14:19,125 INFO [owlms:16000.activeMasterManager]
master.ServerManager: Waiting on RegionServer count=0 to settle;
waited=10329725ms, expecting min=1 server(s), max=NO_LIMIT server(s),
timeout=4500ms, lastChange=-10329725ms
2019-03-02 14:14:19,602 DEBUG [RpcServer.listener,port=16000] ipc.RpcServer:
RpcServer.listener,port=16000: connection from 10.69.68.21:58029; # active
connections: 1
2019-03-02 14:14:19,608 DEBUG
[RpcServer.reader=5,bindAddress=owlms.hdp.senia.org,port=16000] ipc.RpcServer:
Kerberos principal name is hbase
2019-03-02 14:14:19,608 DEBUG
[RpcServer.reader=5,bindAddress=owlms.hdp.senia.org,port=16000] ipc.RpcServer:
RpcServer.listener,port=16000: Caught exception while reading:
org.apache.hadoop.hbase.security.AccessDeniedException: Kerberos principal name
does NOT have the expected hostname part: hbase
at
org.apache.hadoop.hbase.ipc.RpcServer$Connection.saslReadAndProcess(RpcServer.java:1468)
at
org.apache.hadoop.hbase.ipc.RpcServer$Connection.process(RpcServer.java:1788)
at
org.apache.hadoop.hbase.ipc.RpcServer$Connection.readAndProcess(RpcServer.java:1769)
at
org.apache.hadoop.hbase.ipc.RpcServer$Listener.doRead(RpcServer.java:955)
at
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.doRunLoop(RpcServer.java:725)
at
org.apache.hadoop.hbase.ipc.RpcServer$Listener$Reader.run(RpcServer.java:701)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
