I encountered an error "Class org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer not found" when starting the name node. I added the patch for RANGER-1412 to my enable-pdfs-plugin.sh script and got past this error.
Now when I start the namenode, I see this callstack: https://github.com/apache/ranger/blob/ranger-0.7/agents-common/src/main/java/org/apache/ranger/admin/client/RangerAdminRESTClient.java#L141 2017-09-21 20:43:33,571 [main] ERROR util.PolicyRefresher: PolicyRefresher(serviceName=openqe79blue): failed to refresh policies. Will continue to use last known version of policies (-1) com.sun.jersey.api.client.ClientHandlerException: A message body reader for Java class org.apache.ranger.plugin.util.ServicePolicies, and Java type class org.apache.ranger.plugin.util.ServicePolicies, and MIME media type text/html; charset=ISO-8859-1 was not found at com.sun.jersey.api.client.ClientResponse.getEntity(ClientResponse.java:549) at com.sun.jersey.api.client.ClientResponse.getEntity(ClientResponse.java:506) at org.apache.ranger.admin.client.RangerAdminRESTClient.getServicePoliciesIfUpdated(RangerAdminRESTClient.java:141) at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicyfromPolicyAdmin(PolicyRefresher.java:264) at org.apache.ranger.plugin.util.PolicyRefresher.loadPolicy(PolicyRefresher.java:202) at org.apache.ranger.plugin.util.PolicyRefresher.startRefresher(PolicyRefresher.java:149) at org.apache.ranger.plugin.service.RangerBasePlugin.init(RangerBasePlugin.java:157) at org.apache.ranger.authorization.hadoop.RangerHdfsPlugin.init(RangerHdfsAuthorizer.java:613) at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer.start(RangerHdfsAuthorizer.java:98) at org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer.start(RangerHdfsAuthorizer.java:86) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startCommonServices(FSNamesystem.java:1131) at org.apache.hadoop.hdfs.server.namenode.NameNode.startCommonServices(NameNode.java:760) at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:711) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:905) at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:884) at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1610) at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1678) On Wed, Sep 20, 2017 at 4:59 PM, Ramesh Mani <rm...@hortonworks.com> wrote: > Aron, > > Also make sure that the ranger admin conf files > /etc/ranger/admin/conf/ranger-admin-site.xml > has these rangerlookup kerberos principal. Should be there after > installation. > > Regards, > Ramesh > > From: Ramesh Mani <rm...@hortonworks.com> > Date: Wednesday, September 20, 2017 at 1:32 PM > > To: "user@ranger.apache.org" <user@ranger.apache.org> > Subject: Re: HDFS Kerberos documentation/setup > > Aron, > > TestConnection is just used for lookup purpose only. (To list the resource > while maintaining policies). There should be steps to create keytab for > rangerlookup, > just make sure that you have policy for that user so it can list the hdfs > directories/files. > > Even if the test connection fails it doesn’t stop you from maintaining > policies and using ranger. > > Regards, > Ramesh > > From: Aaron Gresch <agre...@gmail.com> > Reply-To: "user@ranger.apache.org" <user@ranger.apache.org> > Date: Wednesday, September 20, 2017 at 1:10 PM > To: "user@ranger.apache.org" <user@ranger.apache.org> > Subject: Re: HDFS Kerberos documentation/setup > > > When I am in the Ranger Admin Service Manager -> Edit Service for HDFS, > there is a Test Connection button. When I press it, it tries to login with > Username and Password. We use keytabs. Tracing the ranger_admin.log, in > BaseClient.java, the lookupPrincipal and lookupKeytab are not set. If I > force these to be set in the code, it then uses a keytab. > > I'm not certain how the keytabs are to be specified other than the > install.properties file. Clearly I must not have specified them properly. > > https://github.com/apache/ranger/blob/688807cf74fc434e246a2f7d6c0e71 > e941178421/agents-common/src/main/java/org/apache/ranger/ > plugin/client/BaseClient.java#L79-L80 > > > <https://github.com/apache/ranger/blob/688807cf74fc434e246a2f7d6c0e71e941178421/agents-common/src/main/java/org/apache/ranger/plugin/client/BaseClient.java#L79-L80> > > > On Wed, Sep 20, 2017 at 3:00 PM, Ramesh Mani <rm...@hortonworks.com> > wrote: > >> Aron, >> >> When you say Login in with user and password is that Ranger Admin UI >> login? Or is the hdfs plugin login into ranger to fetch the policy? >> >> Looks like the NPE is not related to Ranger, but please check >> namenode.log what is there. Please enable debug on namenode and check it >> out. >> >> Thanks, >> Ramesh >> >> From: Aaron Gresch <agre...@gmail.com> >> Reply-To: "user@ranger.apache.org" <user@ranger.apache.org> >> Date: Wednesday, September 20, 2017 at 12:21 PM >> To: "user@ranger.apache.org" <user@ranger.apache.org> >> Subject: Re: HDFS Kerberos documentation/setup >> >> >> Thanks. >> >> Having lots of issues trying to get this to work. >> >> Issue 1 - Admin Server >> >> I'm not exactly sure what I am doing right or wrong so far, but it is >> still trying to login with a user and password rather than a keytab. In >> BaseClient.login(), I hard-coded the keytab and principal, and then I see a >> proper HDFS file listing occurring. This however is failing (see issue >> 2). Looks like it is expecting some xalogin.xml that does not exist to set >> these properties. I still need to dig into why this does not exist. >> >> I'm not clear what authentication mode means exactly. I don't think >> UNIX/LDAP/AD fit anything we do based on questioning the Hadoop team here. >> I'm not exactly clear what this setting is used for or which setting we >> should specify. >> >> Issue 2 - Namenode >> >> I installed the plugin and was able to restart the name node, but no >> policy data was in the cache directory, it appears unable to sync. When I >> do a "hadoop ls" on the command line, I get a NPE: >> >> Caused by: >> org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): >> java.lang.NullPointerException >> >> at org.apache.hadoop.hdfs.DFSUtil.bytes2String(DFSUtil.java:238) >> >> at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.g >> etINodeAttrs(FSPermissionChecker.java:243) >> >> at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.c >> heckPermission(FSPermissionChecker.java:182) >> >> at org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.c >> heckTraverse(FSPermissionChecker.java:499) >> >> at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkTrav >> erse(FSDirectory.java:1605) >> >> at org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkTrav >> erse(FSDirectory.java:1623) >> >> at org.apache.hadoop.hdfs.server.namenode.FSDirectory.resolvePa >> th(FSDirectory.java:544) >> >> at org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp >> .getListingInt(FSDirStatAndListingOp.java:55) >> >> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getListi >> ng(FSNamesystem.java:3695) >> >> >> On Tue, Sep 19, 2017 at 2:53 PM, Ramesh Mani <rm...@hortonworks.com> >> wrote: >> >>> Hi Aaron >>> >>> Please check this out https://cwiki.apache.org/c >>> onfluence/display/RANGER/Ranger+installation+in+Kerberized++Environment >>> >>> Regards, >>> Ramesh >>> >>> From: Aaron Gresch <agre...@gmail.com> >>> Reply-To: "user@ranger.apache.org" <user@ranger.apache.org> >>> Date: Tuesday, September 19, 2017 at 11:58 AM >>> To: "user@ranger.apache.org" <user@ranger.apache.org> >>> Subject: HDFS Kerberos documentation/setup >>> >>> 1) What documentation should I be following to install Ranger manually >>> for a Kerberos Hadoop cluster? I am interested in the HDFS Plugin. >>> >>> This is what I found linked from the apache site, but is very old: >>> >>> https://cwiki.apache.org/confluence/display/RANGER/Ranger+In >>> stallation+Guide >>> >>> >>> 2) Following those instructions, I see "Create a repository in Ranger >>> Policy Manager. E.g. "local_hdfs"." >>> >>> Is this the same as creating a Service? I see Services under HDFS on >>> the Ranger admin server. >>> >>> >>> 3) Creating an HDFS service lists a Username and Password. We don't use >>> passwords for our clusters, but have keytabs. What should this mandatory >>> field be? What is it used for? >>> >>> >>> 4) How is this supposed to be setup for secure clusters? Is there any >>> manually setup example I can be pointed to? >>> >>> >>> Thanks, >>> Aaron >>> >> >> >
