Hello Madhan, Thanks a lot for the prompt reply. This answers how does ranger authenticate itself to the service where it wants to do lookup. I am still confused about the other part.
If I have HBase and HDFS policy how does Ranger makes sure that the service talking to ranger is one and not another for performing authorization checks ? Thanks. On Friday, April 8, 2016, Madhan Neethiraj <[email protected]> wrote: > Rohit, > > Your question at the end is the answer you are looking for! > > The username and password given while adding a service is used by Ranger > Admin > while performing lookup. In case of secure cluster, the username should be > the > Kerberos principal. This user must have enough permissions in the component > (via a Ranger policy?) to perform the lookup. > > > > > Hope this help. > > Madhan > > On 4/8/16, 5:31 PM, "rohit sinha" <[email protected] <javascript:;>> > wrote: > > >Hello Madhan, > > > >I am having some difficulty understanding how ranger and services identify > >each other on a secure cluster. > > > >For example, if I have a resource lookup plugin running in ranger for > HBase > >and a ranger authorization plugin running in HBase and when they make > >request to each other how does ranger make sure that the talking service > in > >HBase and also how does HBase identifies the talking service in ranger. > > > >Also, for every service we take username and password while the service is > >being added. What is it used for ? > > > >Thanks. > > > > > >Thanks, > >Rohit Sinha > > > > > >On Fri, Apr 8, 2016 at 4:38 PM, Madhan Neethiraj <[email protected] > <javascript:;>> wrote: > > > >> Rohit, > >> > >> >> 1. Can you point me to some resource where I can see how to configure > >> Ranger on Kerberos without KMS. > >> Ranger configuration does not depend on presence or absence of KMS. It > will > >> help if you can provide more details on what you are trying to do or > what > >> difference/difficulty you see due to presence/absence of KMS. > >> > >> >> 2. Also, how can I list all the policies for a resource in my service > >> > >> > >> > >> RangerBasePlugin is designed for authorizing accesses and don’t provide > >> APIs to > >> search the policies. You can use REST APIs of Ranger Admin to search > >> policies. > >> > >> > >> For example: curl -f -X GET -H "Accept: application/json" -u > >> admin-user:admin-pass > >> > http://ranger-admin-host:6080/service/public/v2/api/service/cl1_hive/policy?resource:table=employee1 > >> > >> For details of the REST APIs, please refer to: > >> > https://cwiki.apache.org/confluence/display/RANGER/REST+APIs+for+Service+Definition%2C+Service+and+Policy+Management > >> > >> Thanks, > >> Madhan > >> > >> On 4/7/16, 9:28 PM, "rohit sinha" <[email protected] > <javascript:;>> wrote: > >> > >> >Hello Madhan, > >> >Thanks a lot for the prompt reply. > >> > > >> >1. Can you point me to some resource where I can see how to configure > >> >Ranger on Kerberos without KMS. > >> > > >> >2. Also, how can I list all the policies for a resource in my service. > I > >> >don't see any api exposed by RangerBasePlugin for this. I was looking > into > >> >the code base and saw that ServiceStore has some APIs for this. How > can I > >> >access this ? > >> > > >> >Thanks. > >> > > >> >Thanks, > >> >Rohit Sinha > >> > > >> > > >> >On Thu, Apr 7, 2016 at 9:18 PM, Madhan Neethiraj <[email protected] > <javascript:;>> > >> wrote: > >> > > >> >> Rohit, > >> >> > >> >> Thanks for sharing your feedback on Ranger integration. Good to know > >> >> that you are able to get Ranger authorization working in your > >> application. > >> >> > >> >> >> 1. Is it possible to have Ranger running on an unencrypted HDFS > >> >> >> with secure Hadoop through Kerberos ? > >> >> Yes. Ranger works with secure Hadoop - either unencrypted or > encrypted. > >> >> > >> >> >> 2. Currently, I see the following error log for policy cache file. > >> >> >> Isn’t the policy cache file created automatically ? > >> >> Yes, the policy cache file is created automatically. I guess the > error > >> >> > >> >> you see could be cause by non-existent directory where the cache file > >> >> would be saved - in this case /etc/ranger/myservicedev/policycache/. > >> >> Please ensure that this directory exists and has write permission for > >> >> the user that runs the application. > >> >> > >> >> The directory location can be specified via configuration > >> >> ranger.plugin.myservice.policy.cache.dir in > >> ranger-myservice-security.xml. > >> >> > >> >> Hope this helps. > >> >> > >> >> Madhan > >> >> > >> >> > >> >> > >> >> > >> >> > >> >> On 4/7/16, 7:38 PM, "rohit sinha" <[email protected] > <javascript:;>> wrote: > >> >> > >> >> >Hello, > >> >> > > >> >> >Thanks a lot for your prompt replies. It was really helpful. > >> >> >Yes, it seems like there was some misconfiguration issue. I was > able to > >> >> >make it work and I have a very basic integration up and running > with my > >> >> >service now. I have worked on integration/analysis of other > >> authorization > >> >> >models and I must say that Ranger so far has been the easiest to > >> integrate > >> >> >with (at the basic level) and the documentation helped a lot. > >> Appreciate > >> >> >your efforts in building it so far. > >> >> > > >> >> >I have two question which I am trying to figure out. Will > appreciate if > >> >> >someone can provide some pointers. > >> >> > > >> >> >1. Kerberos: From the documentation found here > >> >> > > >> >> > >> > https://cwiki.apache.org/confluence/display/RANGER/Apache+Ranger+0.5.0+Installation#ApacheRanger0.5.0Installation-ConfigureKerberosAuthenticationforStorm > >> >> >I understand that Ranger works with KMS on an encrypted HDFS. Is it > >> >> >possible to have Ranger running on an unencrypted HDFS with secure > >> Hadoop > >> >> >through Kerberos ? Something where I can talk to Ranger by giving a > >> >> >principal and keytab ? > >> >> > > >> >> >2. Currently, I see the following error log for policy cache file. > >> Isn't > >> >> >the policy cache file created automatically ? Am I missing some > >> >> >configuration. > >> >> > > >> >> >16/04/07 21:26:18 INFO util.PolicyRefresher: > >> >> >> PolicyRefresher(serviceName=myservice): found updated version. > >> >> >> lastKnownVersion=-1; newVersion=1 > >> >> >> 16/04/07 21:26:18 ERROR util.PolicyRefresher: failed to save > >> policies to > >> >> >> cache file > >> >> '/etc/ranger/myservicedev/policycache/myservice_myservice.json' > >> >> >> java.io.FileNotFoundException: > >> >> >> /etc/ranger/myservicedev/policycache/myservice_myservice.json (No > >> such > >> >> file > >> >> >> or directory) > >> >> >> at java.io.FileOutputStream.open(Native Method) > >> >> >> at > java.io.FileOutputStream.<init>(FileOutputStream.java:221) > >> >> >> at > java.io.FileOutputStream.<init>(FileOutputStream.java:171) > >> >> >> at java.io.FileWriter.<init>(FileWriter.java:90) > >> >> >> at > >> >> >> > >> >> > >> > org.apache.ranger.plugin.util.PolicyRefresher.saveToCache(PolicyRefresher.java:310) > >> >> >> at > >> >> >> > >> >> > >> > org.apache.ranger.plugin.util.PolicyRefresher.loadPolicy(PolicyRefresher.java:191) > >> >> >> at > >> >> >> > >> >> > >> > org.apache.ranger.plugin.util.PolicyRefresher.startRefresher(PolicyRefresher.java:134) > >> >> >> at > >> >> >> > >> >> > >> > org.apache.ranger.plugin.service.RangerBasePlugin.init(RangerBasePlugin.java:105) > >> >> >> .... internal stacktrace.... > >> >> >> at java.lang.Thread.run(Thread.java:745) > >> >> > > >> >> > > >> >> > > >> >> > > >> >> >Thanks. > >> >> > > >> >> >Thanks, > >> >> >Rohit Sinha > >> >> > > >> >> > > >> >> >On Thu, Apr 7, 2016 at 10:04 AM, Madhan Neethiraj < > [email protected] <javascript:;>> > >> >> wrote: > >> >> > > >> >> >> Rohit, > >> >> >> > >> >> >> To download policies from Ranger Admin, Ranger plugins require the > >> URL > >> >> to > >> >> >> Ranger Admin and the name of the service containing the policies. > >> These > >> >> >> values are read from following configurations from a file named > >> >> >> ranger-<pluginType>-security.xml > >> >> >> > >> >> >> ranger.plugin.<pluginType>.policy.rest.url > >> >> >> ranger.plugin.<pluginType>.service.name > >> >> >> > >> >> >> For example, these are specified in > >> conf/ranger-sampleapp-security.xml > >> >> for > >> >> >> the sample application. > >> >> >> > >> >> >> Can you please review the configuration for your plugin for the > >> above? > >> >> >> > >> >> >> Hope this helps. > >> >> >> > >> >> >> Madhan > >> >> >> > >> >> >> > >> >> >> > >> >> >> On 4/7/16, 5:40 AM, "rohit sinha" <[email protected] > <javascript:;>> wrote: > >> >> >> > >> >> >> >Hello Madhan, > >> >> >> >Thanks a lot for your reply. > >> >> >> > > >> >> >> >I am looking into the integration and I working towards > developing > >> the > >> >> >> >components outside of ranger trunk for initial development > purpose. > >> >> After > >> >> >> >taking an overview of the codebase it seems that it's possible to > >> have > >> >> >> >plugins outside of the ranger trunk too. > >> >> >> > > >> >> >> >I was able to add my service to Ranger Amin through the curl > request > >> >> and > >> >> >> >connect to Ranger from my Authorizer but I see the following > error: > >> >> >> > > >> >> >> >com.sun.jersey.api.client.ClientHandlerException: > >> >> >> >> java.lang.IllegalArgumentException: URI is not absolute > >> >> >> >> at > >> >> >> >> > >> >> >> > >> >> > >> > com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:151) > >> >> >> >> at > >> com.sun.jersey.api.client.Client.handle(Client.java:648) > >> >> >> >> at > >> >> >> >> > com.sun.jersey.api.client.WebResource.handle(WebResource.java:680) > >> >> >> >> at > >> >> >> >> > >> com.sun.jersey.api.client.WebResource.access$200(WebResource.java:74) > >> >> >> >> at > >> >> >> >> > >> >> > com.sun.jersey.api.client.WebResource$Builder.get(WebResource.java:507) > >> >> >> >> at > >> >> >> >> > >> >> >> > >> >> > >> > org.apache.ranger.admin.client.RangerAdminRESTClient.getServicePoliciesIfUpdated(RangerAdminRESTClient.java:94) > >> >> >> >> at > >> >> >> >> > >> >> >> > >> >> > >> > org.apache.ranger.plugin.util.PolicyRefresher.loadPolicyfromPolicyAdmin(PolicyRefresher.java:215) > >> >> >> >> at > >> >> >> >> > >> >> >> > >> >> > >> > org.apache.ranger.plugin.util.PolicyRefresher.loadPolicy(PolicyRefresher.java:183) > >> >> >> >> at > >> >> >> >> > >> >> >> > >> >> > >> > org.apache.ranger.plugin.util.PolicyRefresher.run(PolicyRefresher.java:156) > >> >> >> >> Caused by: java.lang.IllegalArgumentException: URI is not > absolute > >> >> >> >> at java.net.URI.toURL(URI.java:1095) > >> >> >> >> at > >> >> >> >> > >> >> >> > >> >> > >> > com.sun.jersey.client.urlconnection.URLConnectionClientHandler._invoke(URLConnectionClientHandler.java:159) > >> >> >> >> at > >> >> >> >> > >> >> >> > >> >> > >> > com.sun.jersey.client.urlconnection.URLConnectionClientHandler.handle(URLConnectionClientHandler.java:149) > >> >> >> >> ... 8 more > >> >> >> >> 16/04/07 12:31:36 WARN util.PolicyRefresher: cache file does > not > >> >> exist > >> >> >> or > >> >> >> >> not readble 'null' > >> >> >> > > >> >> >> > > >> >> >> >A quick search on google pointed towards misconfiguration of > service > >> >> name > >> >> >> >in security.xml which I have double checked. > >> >> >> > > >> >> >> >Any pointers to debug this will be appreciated. > >> >> >> > > >> >> >> >Thanks. > >> >> >> > > >> >> >> >Thanks, > >> >> >> >Rohit Sinha > >> >> >> > > >> >> >> > > >> >> >> >On Wed, Apr 6, 2016 at 10:43 PM, Madhan Neethiraj < > >> >> >> >[email protected] <javascript:;>> wrote: > >> >> >> > > >> >> >> >> Rohit, > >> >> >> >> > >> >> >> >> You are right. REPOSITORY_NAME referenced in the doc is the > name > >> of > >> >> the > >> >> >> >> service instance in Ranger Admin, which contains the policies > for > >> the > >> >> >> >> component (in this case HBase). The plugin reads this value > from a > >> >> >> >> configuration named ranger.plugin.hbase.service.name (in file > >> >> >> >> ranger-hbase-security.xml). > >> >> >> >> > >> >> >> >> >> After doing this I don't see anything in the Audit -> > Plugins > >> >> >> >> An entry will be created here for every policy download form > >> plugins. > >> >> >> >> Plugins download the policies at the following events: > >> >> >> >> - during the component startup (HBase/HiveServer/...) > >> >> >> >> - when there is a policy change in service instance > >> >> >> >> > >> >> >> >> Hope this helps. > >> >> >> >> > >> >> >> >> Madhan > >> >> >> >> > >> >> >> >> > >> >> >> >> > >> >> >> >> On 4/6/16, 7:13 PM, "rohit sinha" <[email protected] > <javascript:;>> > >> wrote: > >> >> >> >> > >> >> >> >> >Thanks for sharing the SampleApp. I was able to run it > understand > >> >> the > >> >> >> >> >integration point. > >> >> >> >> >I also tried to enable HBase ranger plugin. When I added the > >> service > >> >> >> from > >> >> >> >> >the Ranger Admin UI I was able to talk to HBase and the > resource > >> >> >> >> completion > >> >> >> >> >worked. After that I deleted the HBase plugin from the UI and > >> tried > >> >> to > >> >> >> >> >enable it from the the command line following instructions > >> mentioned > >> >> >> here: > >> >> >> >> > > >> >> >> >> > >> >> >> > >> >> > >> > https://cwiki.apache.org/confluence/display/RANGER/Apache+Ranger+0.5.0+Installation#ApacheRanger0.5.0Installation-InstallingApacheHBase(1.1.0.1) > >> >> >> >> > > >> >> >> >> >After doing this I don't see anything in the Audit -> > Plugins. In > >> >> the > >> >> >> >> >instructions I do see warning > >> >> >> >> > > >> >> >> >> >> Make sure the REPOSITORY_NAME service exists in Ranger > Admin. > >> If > >> >> not, > >> >> >> >> the > >> >> >> >> >> hbase-plugin will not be able to communicate with Ranger > admin. > >> >> >> >> > > >> >> >> >> >Does this mean I need to add the service from the Ranger Admin > >> Panel > >> >> >> or I > >> >> >> >> >did something wrong in the enabling the plugin ? > >> >> >> >> > > >> >> >> >> >Any help will be highly appreciated. > >> >> >> >> > > >> >> >> >> >Thanks. > >> >> >> >> > > >> >> >> >> >Thanks, > >> >> >> >> >Rohit Sinha > >> >> >> >> > > >> >> >> >> > > >> >> >> >> >On Wed, Apr 6, 2016 at 12:43 PM, rohit sinha < > >> >> [email protected] <javascript:;>> > >> >> >> >> >wrote: > >> >> >> >> > > >> >> >> >> >> Awesome. Thanks a lot. > >> >> >> >> >> > >> >> >> >> >> Thanks, > >> >> >> >> >> Rohit Sinha > >> >> >> >> >> > >> >> >> >> >> > >> >> >> >> >> On Wed, Apr 6, 2016 at 12:27 PM, Don Bosco Durai < > >> >> [email protected] <javascript:;>> > >> >> >> >> wrote: > >> >> >> >> >> > >> >> >> >> >>> It is optional. It is easy to setup and helps a lot while > >> >> debugging > >> >> >> >> >>> during initial setup. > >> >> >> >> >>> > >> >> >> >> >>> Bosco > >> >> >> >> >>> > >> >> >> >> >>> > >> >> >> >> >>> > >> >> >> >> >>> > >> >> >> >> >>> > >> >> >> >> >>> On 4/6/16, 12:23 PM, "rohit sinha" < > [email protected] <javascript:;>> > >> >> wrote: > >> >> >> >> >>> > >> >> >> >> >>> >Thanks a lot for the prompt replies. Really appreciate it. > >> >> >> >> >>> >The "Ranger Stacks - How to add a custom plugin?" was > really > >> >> >> helpful > >> >> >> >> in > >> >> >> >> >>> >getting some understanding of the integration. I am going > >> >> through > >> >> >> the > >> >> >> >> >>> >SampleApp docs now. > >> >> >> >> >>> > > >> >> >> >> >>> >Is auditing an optional feature ? From the documentation > it > >> >> looks > >> >> >> like > >> >> >> >> >>> it's > >> >> >> >> >>> >not and Solr installation is a requirement. I was > wondering > >> if I > >> >> >> can > >> >> >> >> have > >> >> >> >> >>> >auditing off and skip Solr installation for initial > >> integration > >> >> >> >> purpose. > >> >> >> >> >>> > > >> >> >> >> >>> >Thanks. > >> >> >> >> >>> > > >> >> >> >> >>> >Thanks, > >> >> >> >> >>> >Rohit Sinha > >> >> >> >> >>> > > >> >> >> >> >>> > > >> >> >> >> >>> >On Wed, Apr 6, 2016 at 11:55 AM, Madhan Neethiraj < > >> >> >> [email protected] <javascript:;>> > >> >> >> >> >>> wrote: > >> >> >> >> >>> > > >> >> >> >> >>> >> Rohit, > >> >> >> >> >>> >> > >> >> >> >> >>> >> In addition to the details in the wiki, I would > recommend > >> >> >> reviewing > >> >> >> >> the > >> >> >> >> >>> >> following sample application to understand the details > of > >> >> adding > >> >> >> >> Ranger > >> >> >> >> >>> >> authorization to an application. > >> >> >> >> >>> >> > >> >> >> >> >>> >> - README.txt: > >> >> >> >> >>> >> > >> >> >> >> >>> > >> >> >> >> > >> >> >> > >> >> > >> > https://github.com/apache/incubator-ranger/blob/master/ranger-examples/README.txt > >> >> >> >> >>> >> - Application sources: > >> >> >> >> >>> >> > >> >> >> >> >>> > >> >> >> >> > >> >> >> > >> >> > >> > https://github.com/apache/incubator-ranger/tree/master/ranger-examples/sampleapp > >> >> >> >> >>> >> . > >> >> >> >> >>> >> > >> >> >> >> >>> >> Madhan > >> >> >> >> >>> >> > >> >> >> >> >>> >> > >> >> >> >> >>> >> > >> >> >> >> >>> >> On 4/6/16, 11:22 AM, "Velmurugan Periasamy" < > >> >> >> >> >>> [email protected] <javascript:;> > >> >> >> >> >>> >> on behalf of [email protected] <javascript:;>> wrote: > >> >> >> >> >>> >> > >> >> >> >> >>> >> >Rohit - > >> >> >> >> >>> >> > > >> >> >> >> >>> > >> >> >> >> > >> >> >> > >> >> > >> > https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=53741207 > >> >> >> >> >>> >> >explains how to add a custom plugin for Ranger. > >> >> >> >> >>> >> > > >> >> >> >> >>> >> >On 4/6/16, 10:47 AM, "rohit sinha" < > >> [email protected] <javascript:;>> > >> >> >> >> wrote: > >> >> >> >> >>> >> > > >> >> >> >> >>> >> >>Hello, > >> >> >> >> >>> >> >>I am looking into integrating an external service with > >> >> Apache > >> >> >> >> Ranger > >> >> >> >> >>> for > >> >> >> >> >>> >> >>authorization. > >> >> >> >> >>> >> >>I looked up the wiki but there is no information about > >> >> >> integrating > >> >> >> >> >>> new > >> >> >> >> >>> >> >>services. > >> >> >> >> >>> >> >>Can someone give me some info which might be helpful > in > >> >> >> >> identifying > >> >> >> >> >>> >> >>different components which needs to be developed and > >> other > >> >> >> >> required > >> >> >> >> >>> stuff > >> >> >> >> >>> >> >> > >> >> >> >> >>> >> >>Thanks. > >> >> >> >> >>> >> > > >> >> >> >> >>> >> > > >> >> >> >> >>> >> > > >> >> >> >> >>> >> > >> >> >> >> >>> >> > >> >> >> >> >>> > >> >> >> >> >>> > >> >> >> >> >> > >> >> >> >> > >> >> >> > >> >> >> > >> >> > >> >> > >> > >> > > -- Thanks, Rohit Sinha
