Having active name node in repo config should work just fine. Only resource lookup is not available during failover cases, until the repo config is updated.
For HA configuration to work, need to add the below properties in repo config (I.e. additional entries in the advanced section). They can be copied from hdfs-site.xml. dfs.nameservices = <ha_name> dfs.ha.namenodes.<ha_name> = <nn1,nn2> dfs.namenode.rpc-address.<nn1> = <nn1_host:8020> dfs.namenode.rpc-address.<nn2> = <nn2_host:8020> dfs.client.failover.proxy.provider.<nn2> = org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider From: Dale Bradman <da...@profusion.com<mailto:da...@profusion.com>> Reply-To: "user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>" <user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>> Date: Wednesday, June 15, 2016 at 10:51 AM To: "user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>" <user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>> Subject: RE: HDFS Plugin - Unable to get listing of files for directory [/] from Hadoop environment That did not work. It works when I set: Hadoop.rpc.protection = - Then in HDFS plugin: Namenode URL = hdfs://hdpmaster01:8020 RPC Protection Type = Authentication The above works. It seems it is the HA configuration that is a problem. Will it work with NameNode HA? Is there any risk for it not being configured to HA? Thanks. From: Velmurugan Periasamy [mailto:vperias...@hortonworks.com] Sent: 15 June 2016 14:31 To: user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org> Subject: Re: HDFS Plugin - Unable to get listing of files for directory [/] from Hadoop environment Dale: Could you set hadoop.rpc.protection to authentication and try? Thank you, Vel From: Dale Bradman <da...@profusion.com<mailto:da...@profusion.com>> Reply-To: "user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>" <user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>> Date: Wednesday, June 15, 2016 at 9:28 AM To: "user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>" <user@ranger.incubator.apache.org<mailto:user@ranger.incubator.apache.org>> Subject: HDFS Plugin - Unable to get listing of files for directory [/] from Hadoop environment Trying to configure the HDFS plugin for Keberised, HA, HDP 2.4.2. I have followed this guide http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.4.2/bk_Security_Guide/content/hdfs_plugin_kerberos.html I have created a “rangerrepouser” in AD and is visible in the Ranger UI. Advanced ranger-hdfs-pluging properties: Ranger repository config user = rangerrepouser@AD.EXAMPLE<mailto:rangerrepouser@AD.EXAMPLE> Ranger repository config password = password set in AD Hadoop.rpc.protection = HDFS Service props: Username: rangerrepouser@AD.EXAMPLE<mailto:rangerrepouser@MAILTRACK.LOCAL> Namenode URL: hdfs://tatooine Authorization enabled: Yes Authentication type: Kerberos hadoop.security.auth_to_local : RULE:[1:$1@$0](ambari-qa-Tatooine@AD.EXAMPLE)s/.*/ambari-qa/RULE:[1:$1@$0](hdfs-Tatooine@AD.EXAMPLE)s/.*/hdfs/RULE:[1:$1@$0](.*@AD.EXAMPLE)s/@.*//RULE:[2:$1@$0](amshbase@AD.EXAMPLE)s/.*/ams/RULE:[2:$1@$0](amszk@AD.EXAMPLE)s/.*/ams/RULE:[2:$1@$0](dn@AD.EXAMPLE)s/.*/hdfs/RULE:[2:$1@$0](hive@AD.EXAMPLE)s/.*/hive/RULE:[2:$1@$0](jhs@AD.EXAMPLE)s/.*/mapred/RULE:[2:$1@$0](jn@AD.EXAMPLE)s/.*/hdfs/RULE:[2:$1@$0](nm@AD.EXAMPLE)s/.*/yarn/RULE:[2:$1@$0](nn@AD.EXAMPLE)s/.*/hdfs/RULE:[2:$1@$0](rm@AD.EXAMPLE)s/.*/yarn/RULE:[2:$1@$0](yarn@AD.EXAMPLE)s/.*/yarn/DEFAULT<mailto:ambari-qa-Tatooine@MAILTRACK.LOCAL)s/.*/ambari-qa/RULE:%5b1:$1@$0%5d(hdfs-Tatooine@MAILTRACK.LOCAL)s/.*/hdfs/RULE:%5b1:$1@$0%5d(.*@MAILTRACK.LOCAL)s/@.*//RULE:%5b2:$1@$0%5d(amshbase@MAILTRACK.LOCAL)s/.*/ams/RULE:%5b2:$1@$0%5d(amszk@MAILTRACK.LOCAL)s/.*/ams/RULE:%5b2:$1@$0%5d(dn@MAILTRACK.LOCAL)s/.*/hdfs/RULE:%5b2:$1@$0%5d(hive@MAILTRACK.LOCAL)s/.*/hive/RULE:%5b2:$1@$0%5d(jhs@MAILTRACK.LOCAL)s/.*/mapred/RULE:%5b2:$1@$0%5d(jn@MAILTRACK.LOCAL)s/.*/hdfs/RULE:%5b2:$1@$0%5d(nm@MAILTRACK.LOCAL)s/.*/yarn/RULE:%5b2:$1@$0%5d(nn@MAILTRACK.LOCAL)s/.*/hdfs/RULE:%5b2:$1@$0%5d(rm@MAILTRACK.LOCAL)s/.*/yarn/RULE:%5b2:$1@$0%5d(yarn@MAILTRACK.LOCAL)s/.*/yarn/DEFAULT> Dfs.datanode.kerberos.principal=dn/hdpnode01.hadoop.local@AD.EXAMPLE<mailto:Dfs.datanode.kerberos.principal=dn/hdpnode01.hadoop.local@AD.EXAMPLE> Dfs.namenode.kerberos.principal= nn/hdpmaster01.hadoop.local@ AD.EXAMPLE Dfs.secondary.namenode.kerberos.principal nn/hdpmaster01.hadoop.local@ AD.EXAMPLE RPC Protection Type = Here is the xa_portal.log: 2016-06-15 14:21:05,037 [timed-executor-pool-0] INFO org.apache.ranger.plugin.client.BaseClient (BaseClient.java:100) - Init Login: using username/password 2016-06-15 14:21:05,194 [timed-executor-pool-0] ERROR apache.ranger.services.hdfs.client.HdfsResourceMgr (HdfsResourceMgr.java:48) - <== HdfsResourceMgr.testConnection Error: org.apache.ranger.plugin.client.HadoopException: Unable to get listing of files for directory [/] from Hadoop environment [Tatooine_hadoop]. 2016-06-15 14:21:05,194 [timed-executor-pool-0] ERROR org.apache.ranger.services.hdfs.RangerServiceHdfs (RangerServiceHdfs.java:59) - <== RangerServiceHdfs.validateConfig Error:org.apache.ranger.plugin.client.HadoopException: Unable to get listing of files for directory [/] from Hadoop environment [Tatooine_hadoop]. 2016-06-15 14:21:05,195 [timed-executor-pool-0] ERROR org.apache.ranger.biz.ServiceMgr$TimedCallable (ServiceMgr.java:434) - TimedCallable.call: Error:org.apache.ranger.plugin.client.HadoopException: Unable to get listing of files for directory [/] from Hadoop environment [Tatooine_hadoop]. 2016-06-15 14:21:05,195 [http-bio-6080-exec-3] ERROR org.apache.ranger.biz.ServiceMgr (ServiceMgr.java:120) - ==> ServiceMgr.validateConfig Error:java.util.concurrent.ExecutionException: org.apache.ranger.plugin.client.HadoopException: Unable to get listing of files for directory [/] from Hadoop environment [Tatooine_hadoop]. 1. Any ideas as to why this is not working? Everything seems consistent. 2. Does the rangerrepouser have to be set up on the Ranger Admin server? It is visible on Ranger UI but is only synchronised with my edge node and not the Admin server 3. Does it matter that the namenode and secondary namenode are pointing to the same Kerberos principal? Doesn’t work if I point them to their respective principals either. Thanks, Dale
