Just for kicks I tried applying the patch in that ticket and it didn't have any effect. It makes sense because my issue is on CREATE, and the ticket only has to do with OPEN.
Note that I don't have these issues using WebHDFS, only using httpfs, so it definitely seems like we're on the right track... Thanks in advance, Ben ________________________________ From: Benjamin Ross Sent: Tuesday, October 11, 2016 12:02 PM To: Wei-Chiu Chuang Cc: u...@hadoop.apache.org; user@ranger.incubator.apache.org Subject: RE: Authentication Failure talking to Ranger KMS That seems promising. But shouldn't I be able to work around it by just ensuring that httpfs has all necessary privileges in the KMS service under Ranger? Thanks, Ben ________________________________ From: Wei-Chiu Chuang [weic...@cloudera.com] Sent: Tuesday, October 11, 2016 11:57 AM To: Benjamin Ross Cc: u...@hadoop.apache.org; user@ranger.incubator.apache.org Subject: Re: Authentication Failure talking to Ranger KMS Somes to me you encountered this bug? HDFS-10481<https://issues.apache.org/jira/browse/HDFS-10481> If you’re using CDH, this is fixed in CDH5.5.5, CDH5.7.2 and CDH5.8.2 Wei-Chiu Chuang A very happy Clouderan On Oct 11, 2016, at 8:38 AM, Benjamin Ross <br...@lattice-engines.com<mailto:br...@lattice-engines.com>> wrote: All, I'm trying to use httpfs to write to an encryption zone with security off. I can read from an encryption zone, but I can't write to one. Here's the applicable namenode logs. httpfs and root both have all possible privileges in the KMS. What am I missing? 2016-10-07 15:48:16,164 DEBUG ipc.Server (Server.java:authorizeConnection(2095)) - Successfully authorized userInfo { effectiveUser: "root" realUser: "httpfs" } protocol: "org.apache.hadoop.hdfs.protocol.ClientProtocol" 2016-10-07 15:48:16,164 DEBUG ipc.Server (Server.java:processOneRpc(1902)) - got #2 2016-10-07 15:48:16,164 DEBUG ipc.Server (Server.java:run(2179)) - IPC Server handler 9 on 8020: org.apache.hadoop.hdfs.protocol.ClientProtocol.create from 10.41.1.64:47622 Call#2 Retry#0 for RpcKind RPC_PROTOCOL_BUFFER 2016-10-07 15:48:16,165 DEBUG security.UserGroupInformation (UserGroupInformation.java:logPrivilegedAction(1751)) - PrivilegedAction as:root (auth:PROXY) via httpfs (auth:SIMPLE) from:org.apache.hadoop.ipc.Server$Handler.run(Server.java:2205) 2016-10-07 15:48:16,166 DEBUG hdfs.StateChange (NameNodeRpcServer.java:create(699)) - *DIR* NameNode.create: file /tmp/cryptotest/hairyballs for DFSClient_NONMAPREDUCE_-1005188439_28 at 10.41.1.64 2016-10-07 15:48:16,166 DEBUG hdfs.StateChange (FSNamesystem.java:startFileInt(2411)) - DIR* NameSystem.startFile: src=/tmp/cryptotest/hairyballs, holder=DFSClient_NONMAPREDUCE_-1005188439_28, clientMachine=10.41.1.64, createParent=true, replication=3, createFlag=[CREATE , OVERWRITE], blockSize=134217728, supportedVersions=[CryptoProtocolVersion{description='Encryption zones', version=2, unknownValue=null}] 2016-10-07 15:48:16,167 DEBUG security.UserGroupInformation (UserGroupInformation.java:logPrivilegedAction(1751)) - PrivilegedAction as:hdfs (auth:SIMPLE) from:org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:484) 2016-10-07 15:48:16,171 DEBUG client.KerberosAuthenticator (KerberosAuthenticator.java:authenticate(205)) - Using fallback authenticator sequence. 2016-10-07 15:48:16,176 DEBUG security.UserGroupInformation (UserGroupInformation.java:doAs(1728)) - PrivilegedActionException as:hdfs (auth:SIMPLE) cause:org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, status: 403, messag e: Forbidden 2016-10-07 15:48:16,176 DEBUG ipc.Server (ProtobufRpcEngine.java:call(631)) - Served: create queueTime= 2 procesingTime= 10 exception= IOException 2016-10-07 15:48:16,177 DEBUG security.UserGroupInformation (UserGroupInformation.java:doAs(1728)) - PrivilegedActionException as:root (auth:PROXY) via httpfs (auth:SIMPLE) cause:java.io.IOException: java.util.concurrent.ExecutionException: java.io.IOException: org.apach e.hadoop.security.authentication.client.AuthenticationException: Authentication failed, status: 403, message: Forbidden 2016-10-07 15:48:16,177 INFO ipc.Server (Server.java:logException(2299)) - IPC Server handler 9 on 8020, call org.apache.hadoop.hdfs.protocol.ClientProtocol.create from 10.41.1.64:47622 Call#2 Retry#0 java.io.IOException: java.util.concurrent.ExecutionException: java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, status: 403, message: Forbidden at org.apache.hadoop.crypto.key.kms.KMSClientProvider.generateEncryptedKey(KMSClientProvider.java:750) at org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.generateEncryptedKey(KeyProviderCryptoExtension.java:371) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.generateEncryptedDataEncryptionKey(FSNamesystem.java:2352) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2478) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2377) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:716) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:405) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2211) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2207) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2205) Caused by: java.util.concurrent.ExecutionException: java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, status: 403, message: Forbidden at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:289) at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:276) at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:111) at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:132) at com.google.common.cache.LocalCache$Segment.getAndRecordStats(LocalCache.java:2381) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2351) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228) at com.google.common.cache.LocalCache.get(LocalCache.java:3965) at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3969) at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4829) at org.apache.hadoop.crypto.key.kms.ValueQueue.getAtMost(ValueQueue.java:266) at org.apache.hadoop.crypto.key.kms.ValueQueue.getNext(ValueQueue.java:226) at org.apache.hadoop.crypto.key.kms.KMSClientProvider.generateEncryptedKey(KMSClientProvider.java:745) ... 15 more Caused by: java.io.IOException: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, status: 403, message: Forbidden at org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:495) at org.apache.hadoop.crypto.key.kms.KMSClientProvider.access$100(KMSClientProvider.java:84) at org.apache.hadoop.crypto.key.kms.KMSClientProvider$EncryptedQueueRefiller.fillQueueForKey(KMSClientProvider.java:133) at org.apache.hadoop.crypto.key.kms.ValueQueue$1.load(ValueQueue.java:181) at org.apache.hadoop.crypto.key.kms.ValueQueue$1.load(ValueQueue.java:175) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3568) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350) ... 23 more Caused by: org.apache.hadoop.security.authentication.client.AuthenticationException: Authentication failed, status: 403, message: Forbidden at org.apache.hadoop.security.authentication.client.AuthenticatedURL.extractToken(AuthenticatedURL.java:274) This message has been scanned for malware by Websense. www.websense.com<http://www.websense.com/> Click here<https://www.mailcontrol.com/sr/MZbqvYs5QwJvpeaetUwhCQ==> to report this email as spam.
