I think I remember that the NameNode FQDN has to bet set into HDFS and
Ranger properties, but I'm not sure it is automatically set.
This could explain your problem (as I think I encountered the same a year
ago).
Regards,
Loïc
Loïc CHANEL
System & virtualization engineer
TO - XaaS Ind - Worldline (Villeurbanne, France)
2016-05-18 11:01 GMT+02:00 Lune Silver <lunescar.ran...@gmail.com>:
> I enabled the DEBUGLOG in ranger admin and found this when I rpess on the
> test connect button in the repository definition windows in the ranger
> admin UI.
>
> ###
> 2016-05-18 10:42:03,135 [timed-executor-pool-0] DEBUG
> org.apache.hadoop.security.SaslRpcClient (SaslRpcClient.java:264) - Get
> token info proto:interface
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolPB
> info:@org.apache.hadoop.security.token.TokenInfo(value=class
> org.apache.hadoop.hdfs.security.token.delegation.DelegationTokenSelector)
> 2016-05-18 10:42:03,139 [timed-executor-pool-0] DEBUG
> org.apache.hadoop.security.UserGroupInformation
> (UserGroupInformation.java:1681) - PrivilegedAction as:rangerhdfslookup
> (auth:null)
> from:org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:648)
> 2016-05-18 10:42:03,139 [timed-executor-pool-0] WARN
> org.apache.hadoop.ipc.Client$Connection$1 (Client.java:680) - Exception
> encountered while connecting to the server : java.lang.NullPointerException
> 2016-05-18 10:42:03,140 [timed-executor-pool-0] DEBUG
> org.apache.hadoop.security.UserGroupInformation
> (UserGroupInformation.java:1661) - PrivilegedActionException
> as:rangerhdfslookup (auth:null) cause:java.io.IOException:
> java.lang.NullPointerException
> 2016-05-18 10:42:03,143 [timed-executor-pool-0] DEBUG
> org.apache.hadoop.ipc.Client$Connection (Client.java:1180) - closing ipc
> connection to <NAMENODE HOST FQDN>/<namenode host IP>:8020:
> java.lang.NullPointerException
> java.io.IOException: java.lang.NullPointerException
> at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:685)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:648)
> at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:735)
> at
> org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:373)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1493)
> at org.apache.hadoop.ipc.Client.call(Client.java:1397)
> at org.apache.hadoop.ipc.Client.call(Client.java:1358)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
> at com.sun.proxy.$Proxy90.getListing(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getListing(ClientNamenodeProtocolTranslatorPB.java:573)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy91.getListing(Unknown Source)
> at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:2094)
> at org.apache.hadoop.hdfs.DFSClient.listPaths(DFSClient.java:2077)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:791)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$700(DistributedFileSystem.java:106)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:853)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:849)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:849)
> at
> org.apache.ranger.services.hdfs.client.HdfsClient.listFilesInternal(HdfsClient.java:83)
> at
> org.apache.ranger.services.hdfs.client.HdfsClient.access$000(HdfsClient.java:41)
> at
> org.apache.ranger.services.hdfs.client.HdfsClient$1.run(HdfsClient.java:165)
> at
> org.apache.ranger.services.hdfs.client.HdfsClient$1.run(HdfsClient.java:162)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:356)
> at
> org.apache.ranger.services.hdfs.client.HdfsClient.listFiles(HdfsClient.java:169)
> at
> org.apache.ranger.services.hdfs.client.HdfsClient.testConnection(HdfsClient.java:211)
> at
> org.apache.ranger.services.hdfs.client.HdfsResourceMgr.testConnection(HdfsResourceMgr.java:46)
> at
> org.apache.ranger.services.hdfs.RangerServiceHdfs.validateConfig(RangerServiceHdfs.java:57)
> at
> org.apache.ranger.biz.ServiceMgr$ValidateCallable.actualCall(ServiceMgr.java:484)
> at
> org.apache.ranger.biz.ServiceMgr$ValidateCallable.actualCall(ServiceMgr.java:471)
> at
> org.apache.ranger.biz.ServiceMgr$TimedCallable.call(ServiceMgr.java:432)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.security.SaslRpcClient.createSaslClient(SaslRpcClient.java:227)
> at
> org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:159)
> at
> org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:396)
> at
> org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:558)
> at
> org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:373)
> at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:727)
> at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:723)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:722)
> ... 39 more
> 2016-05-18 10:42:03,144 [timed-executor-pool-0] DEBUG
> org.apache.hadoop.ipc.Client$Connection (Client.java:1189) - IPC Client
> (1901255770) connection to <NAMENODE HOST FQDN>/<namenode host IP>:8020
> from rangerhdfslookup: closed
> 2016-05-18 10:42:03,144 [timed-executor-pool-0] TRACE
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker
> (ProtobufRpcEngine.java:235) - 60: Exception <- <NAMENODE HOST
> FQDN>/<namenode host IP>:8020: getListing {java.io.IOException: Failed on
> local exception: java.io.IOException: java.lang.NullPointerException; Host
> Details : local host is: "<ranger host fqdn>/<ranger host IP>"; destination
> host is: "<NAMENODE HOST FQDN>":8020; }
> 2016-05-18 10:42:03,147 [timed-executor-pool-0] DEBUG
> apache.ranger.services.hdfs.client.HdfsClient (HdfsClient.java:140) - <==
> HdfsClient listFilesInternal Error : java.io.IOException: Failed on local
> exception: java.io.IOException: java.lang.NullPointerException; Host
> Details : local host is: "<ranger host fqdn>/<ranger host IP>"; destination
> host is: "<NAMENODE HOST FQDN>":8020;
> 2016-05-18 10:42:03,147 [timed-executor-pool-0] ERROR
> apache.ranger.services.hdfs.client.HdfsResourceMgr
> (HdfsResourceMgr.java:48) - <== HdfsResourceMgr.testConnection Error:
> org.apache.ranger.plugin.client.HadoopException: listFilesInternal: Unable
> to get listing of files for directory /null] from Hadoop environment [<HDFS
> REPO>].
> ###
>
> The user rangerhdfslookup exists in my kerberos db, a kinit
> rangerhdfslookupwith the right password works fine, and this is the same
> password that I put in the repository definition, and in the ambari UI.
>
> BR.
>
> Lune.
>
> On Wed, May 18, 2016 at 10:15 AM, Lune Silver <lunescar.ran...@gmail.com>
> wrote:
>
>> Re Ramesh.
>>
>> So my SSL problem is solved, but I still have this error in my log :
>> ###
>>
>> 2016-05-18 10:07:32,579 [timed-executor-pool-0] ERROR
>> org.apache.ranger.services.hdfs.RangerServiceHdfs
>> (RangerServiceHdfs.java:59) - <== RangerServiceHdfs.validateConfig
>> Error:org.apache.ranger.plugin.client.HadoopException: listFilesInternal:
>> Unable to get listing of files for directory /null] from Hadoop environment
>> [<CLUSTERNAME>_hadoop].
>> ###
>>
>> I already have a log of files and folders in HDFS.
>> What do you mean by create an empty file ? With which user ? In which
>> folder ?
>>
>> BR.
>>
>> Lune.
>>
>>
>>
>> On Wed, May 18, 2016 at 9:52 AM, Lune Silver <lunescar.ran...@gmail.com>
>> wrote:
>>
>>> Pfew.
>>>
>>> Indeed the wrong truststore was my problem.
>>> By using the one of the JDK I managed to get rid of the error.
>>>
>>> To get the JAVA_HOME location :
>>> readlink -f /usr/bin/java | sed "s:bin/java::"
>>>
>>> Then the cacerts file is located in lib/security/.
>>> And the default password is changeit.
>>>
>>> BR.
>>>
>>> Lune.
>>>
>>> On Wed, May 18, 2016 at 9:29 AM, Lune Silver <lunescar.ran...@gmail.com>
>>> wrote:
>>>
>>>> In fact, it uses by default the JDK cacert.
>>>> https://issues.apache.org/jira/browse/AMBARI-15917
>>>>
>>>> So I'm wondering if I'm not using the wrong truststore for ranger admin
>>>> in fact.
>>>>
>>>> BR.
>>>>
>>>> Lune
>>>>
>>>> On Wed, May 18, 2016 at 9:27 AM, Lune Silver <lunescar.ran...@gmail.com
>>>> > wrote:
>>>>
>>>>> In fact, I'm wondering.
>>>>> What is the truststore used by default by Ranger Admin ?
>>>>>
>>>>> I can find a property for the truststore of Ranger User-Sync, but not
>>>>> for Ranger Admin.
>>>>>
>>>>> BR.
>>>>>
>>>>>
>>>>> Lune.
>>>>>
>>>>> On Wed, May 18, 2016 at 9:16 AM, Lune Silver <
>>>>> lunescar.ran...@gmail.com> wrote:
>>>>>
>>>>>> Re Ramesh.
>>>>>>
>>>>>> I investigated more my problem and I'm sorry for the confusion.
>>>>>> I checked the policy cache directory on the namenode, and also the
>>>>>> logs of the namenode.
>>>>>>
>>>>>> The policycache dir contains an empty file.
>>>>>> And the namenode log contains the following error message :
>>>>>> ###
>>>>>> 2016-05-18 08:53:50,129 ERROR client.RangerAdminRESTClient
>>>>>> (RangerAdminRESTClient.java:getServicePoliciesIfUpdated(79)) - Error
>>>>>> getting policies. request=https://<RANGER HOST FQDN>:<RANGER ADMIN
>>>>>> PORT>/service/plugins/policies/download/<HDFS
>>>>>> REPO>?lastKnownVersion=-1&pluginId=hdfs@<NAMENODE HOST FQDN>-<HDFS
>>>>>> REPO>,
>>>>>> response={"httpStatusCode":400,"statusCode":1,"msgDesc":"Unauthorized
>>>>>> access - unable to get client
>>>>>> certificate","messageList":[{"name":"OPER_NOT_ALLOWED_FOR_ENTITY","rbKey":"xa.error.oper_not_allowed_for_state","message":"Operation
>>>>>> not allowed for entity"}]}, serviceName=<HDFS REPO>
>>>>>> 2016-05-18 08:53:50,130 ERROR util.PolicyRefresher
>>>>>> (PolicyRefresher.java:loadPolicyfromPolicyAdmin(228)) -
>>>>>> PolicyRefresher(serviceName=<HDFS REPO>): failed to refresh policies.
>>>>>> Will
>>>>>> continue to use last known version of policies (-1)
>>>>>> java.lang.Exception: Unauthorized access - unable to get client
>>>>>> certificate
>>>>>> at
>>>>>> org.apache.ranger.admin.client.RangerAdminRESTClient.getServicePoliciesIfUpdated(RangerAdminRESTClient.java:81)
>>>>>> at
>>>>>> org.apache.ranger.plugin.util.PolicyRefresher.loadPolicyfromPolicyAdmin(PolicyRefresher.java:205)
>>>>>> at
>>>>>> org.apache.ranger.plugin.util.PolicyRefresher.loadPolicy(PolicyRefresher.java:175)
>>>>>> at
>>>>>> org.apache.ranger.plugin.util.PolicyRefresher.startRefresher(PolicyRefresher.java:132)
>>>>>> at
>>>>>> org.apache.ranger.plugin.service.RangerBasePlugin.init(RangerBasePlugin.java:106)
>>>>>> at
>>>>>> org.apache.ranger.authorization.hadoop.RangerHdfsPlugin.init(RangerHdfsAuthorizer.java:399)
>>>>>> at
>>>>>> org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer.start(RangerHdfsAuthorizer.java:83)
>>>>>> at
>>>>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startCommonServices(FSNamesystem.java:1062)
>>>>>> at
>>>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.startCommonServices(NameNode.java:763)
>>>>>> at
>>>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:687)
>>>>>> at
>>>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:896)
>>>>>> at
>>>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:880)
>>>>>> at
>>>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1586)
>>>>>> at
>>>>>> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1652)
>>>>>> ###
>>>>>>
>>>>>> What does OPER_NOT_ALLOWED_FOR_ENTITY means ?
>>>>>> Which user is the operator for the hdfs plugin ?
>>>>>> Is it the user created for the plugin (in the property Ranger
>>>>>> repository config user) ?
>>>>>>
>>>>>> I enabled the SSL for HDFS plugin following the HW doc here :
>>>>>>
>>>>>> http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_Security_Guide/content/ch04s18s02s04s01.html
>>>>>>
>>>>>> Do you think my problem could come from an error from my SSL
>>>>>> configuration ?
>>>>>>
>>>>>> If I summarize what i did :
>>>>>>
>>>>>> I have :
>>>>>> - one node with the namenode
>>>>>> - one node with ranger (admin + usersync)
>>>>>>
>>>>>> On the namenode host, I created a plugin keystore.
>>>>>> This keystore contains the certificate for the alias rangerHdfsAgent.
>>>>>> ###
>>>>>> cd /etc/hadoop/conf
>>>>>> keytool -genkey -keyalg RSA -alias rangerHdfsAgent -keystore
>>>>>> /etc/hadoop/conf/ranger-plugin-keystore.jks -validity 3600 -keysize 2048
>>>>>> -dname
>>>>>> 'cn=HdfsPlugin,ou=<mycompany>,o=<mycompany>,l=<mycity>,st=<mycountry>,c=<idcountry>'
>>>>>> chown hdfs:hdfs /etc/hadoop/conf/ranger-plugin-keystore.jks
>>>>>> chmod 400 /etc/hadoop/conf/ranger-plugin-keystore.jks
>>>>>> ###
>>>>>>
>>>>>> On the Ranger host, I exported the certificate for the alias
>>>>>> rangeradmin from the admin keystore.
>>>>>> ###
>>>>>> keytool -export -keystore
>>>>>> /etc/ranger/admin/conf/ranger-admin-keystore.jks -alias rangeradmin -file
>>>>>> /etc/ranger/admin/conf/ranger-admin-trust.cer
>>>>>> ###
>>>>>>
>>>>>> Then I transfered the cer file from the ranger host to the namenode
>>>>>> host.
>>>>>>
>>>>>> On the namenode host, I imported the certificate of the alias
>>>>>> rangeradmin into the plugin truststore. (the truststore was not yet
>>>>>> existing)
>>>>>> ###
>>>>>> keytool -import -file /etc/hadoop/conf/ranger-admin-trust.cer -alias
>>>>>> rangeradmintrust -keystore /etc/hadoop/conf/ranger-plugin-truststore.jks
>>>>>> chown hdfs:hdfs /etc/hadoop/conf/ranger-plugin-truststore.jks
>>>>>> chmod 400 /etc/hadoop/conf/ranger-plugin-truststore.jks
>>>>>> ###
>>>>>>
>>>>>> On the namenode host, I exported the certificate for the alias
>>>>>> rangerHdfsAgent from the plugin keystore.
>>>>>> ###
>>>>>> keytool -export -keystore /etc/hadoop/conf/ranger-plugin-keystore.jks
>>>>>> -alias rangerHdfsAgent -file /etc/hadoop/conf/ranger-hdfsAgent-trust.cer
>>>>>> ###
>>>>>>
>>>>>> Then I transfered the ranger-hdfsAgent-trust.cer file from the
>>>>>> namenode host to the ranger host.
>>>>>>
>>>>>> On the ranger host, I imported the certificate for the alias
>>>>>> rangerHdfsAgent in the admin truststore (the truststore was not yet
>>>>>> existing).
>>>>>> ###
>>>>>> keytool -import -file
>>>>>> /etc/ranger/admin/conf/ranger-hdfsAgent-trust.cer -alias
>>>>>> rangerHdfsAgentTrust -keystore
>>>>>> /etc/ranger/admin/conf/ranger-admin-truststore.jks
>>>>>> chown ranger:ranger /etc/ranger/admin/conf/ranger-admin-truststore.jks
>>>>>> chmod 400 /etc/ranger/admin/conf/ranger-admin-truststore.jks
>>>>>> ###
>>>>>>
>>>>>> In the Ambari UI, I added the CN HdfsPlugin in the property "Common
>>>>>> Name For Certificate".
>>>>>>
>>>>>> In the Ranger Admin UI, I checked that, in the repository definition,
>>>>>> there is also this property with the right value.
>>>>>>
>>>>>> Do you think Is there something wrong ?
>>>>>>
>>>>>> BR.
>>>>>>
>>>>>> Lune.
>>>>>>
>>>>>>
>>>>>> On Tue, May 17, 2016 at 3:45 PM, Lune Silver <
>>>>>> lunescar.ran...@gmail.com> wrote:
>>>>>>
>>>>>>> Hello !
>>>>>>>
>>>>>>> I just enabled the HDFS plugin for Ranger.
>>>>>>> The repository was created by Ambari (2.2.1 with HDP cluster 2.3.2).
>>>>>>>
>>>>>>> In the Ranger Admin UI, in the repository edit window, when I check
>>>>>>> on the button "test connection", I have the following error message :
>>>>>>> ###
>>>>>>> Unable to connect repository with given config for <MYCLUSTER>_hadoop
>>>>>>> ###
>>>>>>>
>>>>>>> And I can see this in the logs :
>>>>>>> ###
>>>>>>> 2016-05-17 15:41:49,895 [http-bio-6182-exec-5] ERROR
>>>>>>> org.apache.ranger.biz.ServiceMgr (ServiceMgr.java:120) - ==>
>>>>>>> ServiceMgr.validateConfig Error:java.util.concurrent.ExecutionException:
>>>>>>> org.apache.ranger.plugin.client.HadoopException: listFilesInternal:
>>>>>>> Unable
>>>>>>> to get listing of files for directory /null] from Hadoop environment
>>>>>>> [<MYCLUSTER>_hadoop].
>>>>>>> ###
>>>>>>>
>>>>>>> Any idea about why this test connection fails ?
>>>>>>>
>>>>>>> BR.
>>>>>>>
>>>>>>> Lune.
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
