[
https://issues.apache.org/jira/browse/HBASE-14605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957568#comment-14957568
]
Devaraj Das commented on HBASE-14605:
-------------------------------------
Yeah [~jinghe] this is a regression introduced by HBASE-14475... The patch
14605-v1.txt that Ted uploaded is based on the premise that the coprocessors'
initialization happens as the user RegionServer runs as. This should be fine in
any case (it was already implicitly happening as the Regionserver user; this
patch makes it explicit)...
One issue that still remains is that if the split-request/compaction-request
implementation changes in the future where, for example, some HDFS operations
are done synchronously (not suggesting this though), then the patch in
HBASE-14475 will cause issues since the HDFS operations to do with splitting
will happen as the end-user instead of the RegionServer user... But I guess, we
can tackle those issues as and when they show up.
> Split fails due to 'No valid credentials' error when
> SecureBulkLoadEndpoint#start tries to access hdfs
> ------------------------------------------------------------------------------------------------------
>
> Key: HBASE-14605
> URL: https://issues.apache.org/jira/browse/HBASE-14605
> Project: HBase
> Issue Type: Bug
> Reporter: Ted Yu
> Assignee: Ted Yu
> Attachments: 14605-v1.txt, 14605.alt
>
>
> During recent testing in secure cluster (with HBASE-14475), we found the
> following when user X (non-super user) split a table with region replica:
> {code}
> 2015-10-12 10:58:18,955 ERROR [FifoRpcScheduler.handler1-thread-9]
> master.HMaster: Region server hbase-4-4.novalocal,60020,1444645588137
> reported a fatal error:
> ABORTING region server hbase-4-4.novalocal,60020,1444645588137: The
> coprocessor org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> threw an unexpected exception
> Cause:
> java.lang.IllegalStateException: Failed to get FileSystem instance
> at
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.start(SecureBulkLoadEndpoint.java:148)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment.startup(CoprocessorHost.java:415)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:257)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadSystemCoprocessors(CoprocessorHost.java:160)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:192)
> at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:701)
> at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:608)
> ...
> Caused by: java.io.IOException: Failed on local exception:
> java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed
> [Caused by GSSException: No valid credentials provided (Mechanism
> level: Failed to find any Kerberos tgt)]; Host Details : local host is:
> "hbase-4-4/172.22.66.186"; destination host is: "os-r6-
> okarus-hbase-4-2.novalocal":8020;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
> at org.apache.hadoop.ipc.Client.call(Client.java:1473)
> at org.apache.hadoop.ipc.Client.call(Client.java:1400)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
> at com.sun.proxy.$Proxy18.mkdirs(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:555)
> at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy19.mkdirs(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2775)
> at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2746)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:967)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:963)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> {code}
> The cause was that SecureBulkLoadEndpoint#start tried to create staging dir
> in hdfs as user X but didn't pass authentication.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
