[
https://issues.apache.org/jira/browse/HBASE-12953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14330210#comment-14330210
]
zhangduo commented on HBASE-12953:
----------------------------------
Add some ugly logs in DefaultVisibilityLabelServiceImpl and
VisibilityLabelsCache.
{code:title=DefaultVisibilityLabelServiceImpl.java}
public OperationStatus[] addLabels(List<byte[]> labels) throws IOException {
...
if (mutateLabelsRegion(puts, finalOpStatus)) {
updateZk(true);
}
for (byte[] label : labels) {
String labelStr = Bytes.toString(label);
LOG.info(labelStr + "=====" + this.labelsCache.getLabelOrdinal(labelStr));
}
...
}
{code}
{code:title=VisibilityLabelsCache.java}
public void refreshLabelsCache(byte[] data) throws IOException {
LOG.info("========refresh", new Exception());
...
}
{code}
And I collected the log for setupBeforeClass
{noformat}
2015-02-21 20:39:16,362 INFO [B.defaultRpcServer.handler=0,queue=0,port=42678]
visibility.DefaultVisibilityLabelServiceImpl(236): secret=====0
2015-02-21 20:39:16,362 INFO [B.defaultRpcServer.handler=0,queue=0,port=42678]
visibility.DefaultVisibilityLabelServiceImpl(236): topsecret=====0
2015-02-21 20:39:16,362 INFO [B.defaultRpcServer.handler=0,queue=0,port=42678]
visibility.DefaultVisibilityLabelServiceImpl(236): confidential=====0
2015-02-21 20:39:16,362 INFO [B.defaultRpcServer.handler=0,queue=0,port=42678]
visibility.DefaultVisibilityLabelServiceImpl(236): public=====0
2015-02-21 20:39:16,362 INFO [B.defaultRpcServer.handler=0,queue=0,port=42678]
visibility.DefaultVisibilityLabelServiceImpl(236): private=====0
2015-02-21 20:39:16,362 INFO [B.defaultRpcServer.handler=0,queue=0,port=42678]
visibility.DefaultVisibilityLabelServiceImpl(236): ©ABC=====0
2015-02-21 20:39:16,362 INFO [B.defaultRpcServer.handler=0,queue=0,port=42678]
visibility.DefaultVisibilityLabelServiceImpl(236): ु=====0
2015-02-21 20:39:16,362 INFO [B.defaultRpcServer.handler=0,queue=0,port=42678]
visibility.DefaultVisibilityLabelServiceImpl(236): ©ABC"ु\secret"'&\=====0
2015-02-21 20:39:16,363 INFO [B.defaultRpcServer.handler=0,queue=0,port=42678]
visibility.DefaultVisibilityLabelServiceImpl(236): '"+=====0
2015-02-21 20:39:16,363 INFO [B.defaultRpcServer.handler=0,queue=0,port=42678]
visibility.DefaultVisibilityLabelServiceImpl(236): -?=====0
2015-02-21 20:39:16,363 INFO [main-EventThread]
visibility.VisibilityLabelsCache(115): ========refresh
java.lang.Exception
at
org.apache.hadoop.hbase.security.visibility.VisibilityLabelsCache.refreshLabelsCache(VisibilityLabelsCache.java:115)
at
org.apache.hadoop.hbase.security.visibility.ZKVisibilityLabelWatcher.refreshVisibilityLabelsCache(ZKVisibilityLabelWatcher.java:76)
at
org.apache.hadoop.hbase.security.visibility.ZKVisibilityLabelWatcher.nodeDataChanged(ZKVisibilityLabelWatcher.java:115)
at
org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:412)
at
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:522)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:498)
{noformat}
We can see that, refresh is triggered from a background zk watcher, and it is
possible that the labels are not in labelsCache after addLabels being called.
I think this is the reason why TestShell is flakey, but I do not know what is
the right way to fix it(make test wait or modify addLabels?). [~stack]
> RegionServer is not functionally working with AysncRpcClient in secure mode
> ---------------------------------------------------------------------------
>
> Key: HBASE-12953
> URL: https://issues.apache.org/jira/browse/HBASE-12953
> Project: HBase
> Issue Type: Bug
> Components: security
> Affects Versions: 2.0.0, 1.1.0
> Reporter: Ashish Singhi
> Assignee: zhangduo
> Priority: Critical
> Fix For: 2.0.0, 1.1.0
>
> Attachments: HBASE-12953.patch, HBASE-12953_1.patch,
> HBASE-12953_2.patch, HBASE-12953_2.patch, HBASE-12953_2.patch,
> HBASE-12953_2.patch, HBASE-12953_2.patch, HBASE-12953_3 (2).patch,
> HBASE-12953_3 (2).patch, HBASE-12953_3.patch, HBASE-12953_3.patch,
> HBASE-12953_3.patch, testcase.patch
>
>
> HBase version 2.0.0
> Default value for {{hbase.rpc.client.impl}} is set to AsyncRpcClient.
> When trying to install HBase with Kerberos, RegionServer is not working
> functionally.
> The following log is logged in its log file
> {noformat}
> 2015-02-02 14:59:05,407 WARN [AsyncRpcChannel-pool1-t1]
> channel.DefaultChannelPipeline: An exceptionCaught() event was fired, and it
> reached at the tail of the pipeline. It usually means the last handler in the
> pipeline did not handle the exception.
> io.netty.channel.ChannelPipelineException:
> org.apache.hadoop.hbase.security.SaslClientHandler.handlerAdded() has thrown
> an exception; removed.
> at
> io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:499)
> at
> io.netty.channel.DefaultChannelPipeline.callHandlerAdded(DefaultChannelPipeline.java:481)
> at
> io.netty.channel.DefaultChannelPipeline.addFirst0(DefaultChannelPipeline.java:114)
> at
> io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:97)
> at
> io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:235)
> at
> io.netty.channel.DefaultChannelPipeline.addFirst(DefaultChannelPipeline.java:214)
> at
> org.apache.hadoop.hbase.ipc.AsyncRpcChannel$2.operationComplete(AsyncRpcChannel.java:194)
> at
> org.apache.hadoop.hbase.ipc.AsyncRpcChannel$2.operationComplete(AsyncRpcChannel.java:157)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:680)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:603)
> at
> io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:563)
> at
> io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:406)
> at
> io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:82)
> at
> io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:253)
> at
> io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:288)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:528)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:468)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:382)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:354)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)]
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212)
> at
> org.apache.hadoop.hbase.security.SaslClientHandler.handlerAdded(SaslClientHandler.java:154)
> at
> io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:486)
> ... 20 more
> Caused by: GSSException: No valid credentials provided (Mechanism level:
> Failed to find any Kerberos tgt)
> at
> sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
> at
> sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121)
> at
> sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)
> at
> sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223)
> at
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212)
> at
> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193)
> {noformat}
> When set hbase.rpc.client.impl to RpcClientImpl, there seems to be no issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
