[
https://issues.apache.org/jira/browse/HBASE-14605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957755#comment-14957755
]
Jerry He commented on HBASE-14605:
----------------------------------
The reason the original HBASE-14475 didn't have a HDFS permission issue is
probably because of these:
1. HRegionFileSystem is already instantiated with a FileSystem instance by the
region server process with 'hbase' id. DFSClient id.
2. Later requestUser.doAs() will do the work as request user. But if the work
uses the existing HRegionFileSystem (e.g. create region files, etc) the real
user to HDFS namenode is still 'hbase'.
> Split fails due to 'No valid credentials' error when
> SecureBulkLoadEndpoint#start tries to access hdfs
> ------------------------------------------------------------------------------------------------------
>
> Key: HBASE-14605
> URL: https://issues.apache.org/jira/browse/HBASE-14605
> Project: HBase
> Issue Type: Bug
> Reporter: Ted Yu
> Assignee: Ted Yu
> Attachments: 14605-v1.txt, 14605-v2.txt, 14605.alt
>
>
> During recent testing in secure cluster (with HBASE-14475), we found the
> following when user X (non-super user) split a table with region replica:
> {code}
> 2015-10-12 10:58:18,955 ERROR [FifoRpcScheduler.handler1-thread-9]
> master.HMaster: Region server hbase-4-4.novalocal,60020,1444645588137
> reported a fatal error:
> ABORTING region server hbase-4-4.novalocal,60020,1444645588137: The
> coprocessor org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint
> threw an unexpected exception
> Cause:
> java.lang.IllegalStateException: Failed to get FileSystem instance
> at
> org.apache.hadoop.hbase.security.access.SecureBulkLoadEndpoint.start(SecureBulkLoadEndpoint.java:148)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost$Environment.startup(CoprocessorHost.java:415)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadInstance(CoprocessorHost.java:257)
> at
> org.apache.hadoop.hbase.coprocessor.CoprocessorHost.loadSystemCoprocessors(CoprocessorHost.java:160)
> at
> org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.<init>(RegionCoprocessorHost.java:192)
> at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:701)
> at org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:608)
> ...
> Caused by: java.io.IOException: Failed on local exception:
> java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed
> [Caused by GSSException: No valid credentials provided (Mechanism
> level: Failed to find any Kerberos tgt)]; Host Details : local host is:
> "hbase-4-4/172.22.66.186"; destination host is: "os-r6-
> okarus-hbase-4-2.novalocal":8020;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
> at org.apache.hadoop.ipc.Client.call(Client.java:1473)
> at org.apache.hadoop.ipc.Client.call(Client.java:1400)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
> at com.sun.proxy.$Proxy18.mkdirs(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.mkdirs(ClientNamenodeProtocolTranslatorPB.java:555)
> at sun.reflect.GeneratedMethodAccessor13.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy19.mkdirs(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.primitiveMkdir(DFSClient.java:2775)
> at org.apache.hadoop.hdfs.DFSClient.mkdirs(DFSClient.java:2746)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:967)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:963)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> {code}
> The cause was that SecureBulkLoadEndpoint#start tried to create staging dir
> in hdfs as user X but didn't pass authentication.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
