Hi Odon, Could you please put name node in debug and see the logs, you can see if there are any exceptions related to Ranger when name node comes.
Like Bosco and Kevin had mentioned mostly like that configurations are not picked up and plugin is not getting enabled. Thanks, Ramesh From: Kevin Risden <kris...@apache.org<mailto:kris...@apache.org>> Reply-To: "user@ranger.apache.org<mailto:user@ranger.apache.org>" <user@ranger.apache.org<mailto:user@ranger.apache.org>> Date: Friday, January 11, 2019 at 9:27 AM To: "user@ranger.apache.org<mailto:user@ranger.apache.org>" <user@ranger.apache.org<mailto:user@ranger.apache.org>> Subject: Re: Accessing Ranger Policy Manager API from HDFS plugin Do you have the ranger plugin on the hdfs classpath? Could be that nothing is getting picked up because it's not installed in the right location? Kevin Risden On Fri, Jan 11, 2019, 12:19 Odon Copon <odonco...@gmail.com<mailto:odonco...@gmail.com> wrote: Ok, I'm running out of ideas to debug the issue. No logs like Bosco posted before, and I cannot find a reason for this not to work, when the endpoint is accessible and is correctly set in install.properties. On Fri, 11 Jan 2019 at 15:42, Odon Copon <odonco...@gmail.com<mailto:odonco...@gmail.com>> wrote: Thanks Bosco, that's really helpful. Ran the following search "grep -irnw '/' -e 'AuditProviderFactory' --exclude \*.java --exclude \*.class --exclude \*.jar 2>/dev/null" trying to look for "AuditProviderFactory" keyword, but there's nothing with that word in it (exluding java, class and jar files). So no log files like yours. What could be causing this? I'm restarting the namenode executing the following command: "sudo /etc/init.d/hadoop-namenode restart", is this enough or is there any other internal component that needs to be restarted as well? like the nodemanager per example. On Fri, 11 Jan 2019 at 15:34, Don Bosco Durai <bo...@apache.org<mailto:bo...@apache.org>> wrote: You might want to look into the logs. Here are some of mine from the HDFS namenode log .. 2019-01-10 06:52:22,128 INFO provider.AuditProviderFactory (AuditProviderFactory.java:init(150)) - AUDIT PROPERTY: ranger.plugin.hdfs.policy.rest.url=http://orange1.mydomain.com:6080 2019-01-10 06:52:22,200 INFO service.RangerBasePlugin (RangerBasePlugin.java:init(151)) - PolicyEngineOptions: { evaluatorType: auto, cacheAuditResult: true, disableContextEnrichers: false, disableCustomConditions: false, disableTrieLookupPrefilter: false } 2019-01-10 06:52:23,274 INFO util.PolicyRefresher (PolicyRefresher.java:loadPolicyfromPolicyAdmin(277)) - PolicyRefresher(serviceName=orange_hadoop): found updated version. lastKnownVersion=-1; newVersion=3 2019-01-10 06:52:23,337 INFO util.RangerResourceTrie (RangerResourceTrie.java:<init>(112)) - resourceName=path; optIgnoreCase=false; optWildcard=true; wildcardChars=*?{}\; nodeCount=18; leafNodeCount=1; singleChildNodeCount=17; maxDepth=18; evaluatorListCount=0; wildcardEvaluatorListCount=2; evaluatorListRefCount=17; wildcardEvaluatorListRefCount=15 Bosco From: Odon Copon <odonco...@gmail.com<mailto:odonco...@gmail.com>> Reply-To: <user@ranger.apache.org<mailto:user@ranger.apache.org>> Date: Friday, January 11, 2019 at 7:23 AM To: <user@ranger.apache.org<mailto:user@ranger.apache.org>> Subject: Re: Accessing Ranger Policy Manager API from HDFS plugin On the namenode I'm editing install.properties file and then "sudo ./enable-hdfs-plugin.sh". Then I'm restarting the namenode service. On Fri, 11 Jan 2019 at 15:19, Don Bosco Durai <bo...@apache.org<mailto:bo...@apache.org>> wrote: It is every 30 seconds. If you are not seeing anything, then it might be a configuration issue. How are you enabling the HDFS plugin? Bosco From: Odon Copon <odonco...@gmail.com<mailto:odonco...@gmail.com>> Reply-To: <user@ranger.apache.org<mailto:user@ranger.apache.org>> Date: Friday, January 11, 2019 at 7:18 AM To: <user@ranger.apache.org<mailto:user@ranger.apache.org>> Subject: Re: Accessing Ranger Policy Manager API from HDFS plugin 1.- How often is the plugin pulling policies from the API? 2.- I don't see anything in the logs regarding the plugin, how can I ensure is it running correctly? 3.- If I run a manual CURL requesting the policies I can see the UI is showing that attempt to retrieve the policies, with a 200. But just the manual CURL requests, no requests coming from the plugin. On Fri, 11 Jan 2019 at 14:58, Don Bosco Durai <bo...@apache.org<mailto:bo...@apache.org>> wrote: Also make sure you have the correct hostname and port for Ranger Admin. Bosco From: Velmurugan Periasamy <vperias...@hortonworks.com<mailto:vperias...@hortonworks.com>> Reply-To: <user@ranger.apache.org<mailto:user@ranger.apache.org>> Date: Friday, January 11, 2019 at 6:32 AM To: "user@ranger.apache.org<mailto:user@ranger.apache.org>" <user@ranger.apache.org<mailto:user@ranger.apache.org>> Subject: Re: Accessing Ranger Policy Manager API from HDFS plugin You can check namenode log for any errors from HDFS plugin. ________________________________ From: Odon Copon <odonco...@gmail.com<mailto:odonco...@gmail.com>> Sent: Friday, January 11, 2019 9:21 AM To: user@ranger.apache.org<mailto:user@ranger.apache.org> Subject: Re: Accessing Ranger Policy Manager API from HDFS plugin I fired manually a CURL request to "/service/plugins/policies/download/<service_name>" and now the UI is displaying some information in plugin tab. 1. Is Ranger Admin thinking the call was made from the plugin and is trying to list it? 2. If plugin would have executed this request, the UI should have displayed this information earlier, right? 3. Any specific log to check for more information? On Fri, 11 Jan 2019 at 14:07, Velmurugan Periasamy <vperias...@hortonworks.com<mailto:vperias...@hortonworks.com>> wrote: You should see plugin sync'ing policies in plugin tab. If it is not showing up, you need to check the logs for any error messages. ________________________________ From: Odon Copon <odonco...@gmail.com<mailto:odonco...@gmail.com>> Sent: Friday, January 11, 2019 8:47 AM To: user@ranger.apache.org<mailto:user@ranger.apache.org> Subject: Re: Accessing Ranger Policy Manager API from HDFS plugin ok, seems "service/plugins/policies/download/" has public access, so confirms what we have been discussing, no authorization is required to download the policies. Good to know, thanks guys. The question know is how to know the plugin is calling this endpoint, and working properly, because the UI doesn't display this plugin. Any tip on this? On Fri, 11 Jan 2019 at 13:08, Odon Copon <odonco...@gmail.com<mailto:odonco...@gmail.com>> wrote: Yes, makes sense to have a 2-way SSL between the plugin and Ranger Admin, but: - 1. Does it mean there's no authentication at all between them? - 2. If there's no authentication, shouldn't a simple CURL work? At the moment if no user/pass is provided the API returns 401, or is there another different endpoint? If so, which one is it? - 3. What is the best way to debug the plugin is communicating or trying to communicate with Ranger admin? Thanks. On Fri, 11 Jan 2019 at 12:53, Velmurugan Periasamy <vperias...@hortonworks.com<mailto:vperias...@hortonworks.com>> wrote: If there is no kerberos HDFS plugin uses the open Download policies API, so it is recommended to use 2-way SSL between HDFS plugin and Ranger Admin. On Jan 11, 2019, at 5:26 AM, Odon Copon <odonco...@gmail.com<mailto:odonco...@gmail.com>> wrote: I cannot perform a CURL to the API from the namenode without user/password, I get a 401 when doing that. So it might required credentials to do that. If I use the admin/password credentials or rangerusersync credentials the CURL works. So wondering if those credentials need to be setup somewhere. On Fri, 11 Jan 2019 at 10:15, Don Bosco Durai <bo...@apache.org<mailto:bo...@apache.org>> wrote: > In terms of "no authentication", is the HDFS plugin using Policy Manager API > with no credentials at all? No credentials, because there is no user/password for HDFS service user. It’s been a while, I think we used to have admin/password before, but it was taken out eventually. The code might be still there… > What's the first action the plugin is performing to be detected by the UI as > active and 200 response? Abhay or Madhan might be able to give you more specifics. Since the plugins are polling and it knows the previous version number, if there are no changes, then it is not registered in the UI. The plugins primarily pull the policies and tags from Ranger Admin. Rest everything is done by the plugin within the component. Bosco From: Odon Copon <odonco...@gmail.com<mailto:odonco...@gmail.com>> Reply-To: <user@ranger.apache.org<mailto:user@ranger.apache.org>> Date: Friday, January 11, 2019 at 2:03 AM To: <user@ranger.apache.org<mailto:user@ranger.apache.org>> Subject: Re: Accessing Ranger Policy Manager API from HDFS plugin Indeed, I know that at the moment without something like Kerberos, users can impersonate others, but I'm currently building a POC with the basic security to evaluate Ranger, and once is ready, start improving the security and scalability. But thank you for pointing that out. In terms of "no authentication", is the HDFS plugin using Policy Manager API with no credentials at all? or default ones? What's the first action the plugin is performing to be detected by the UI as active and 200 response? Some king of ping/heartbeat? or just a rest petition to download the policies? Is there anywhere where I can see in the logs what kind of actions the plugin is doing? I don't find any log information coming from the plugin. Thanks! On Fri, 11 Jan 2019 at 09:53, Don Bosco Durai <bo...@apache.org<mailto:bo...@apache.org>> wrote: If there is no Kerberos, then you have 2 options: 1. No authentication (default) 2. Two way SSL to authenticate the request from the plugin. Note, if it is non-Kerberos environment, then authorization cannot be enforced, because users can impersonate anyone. Bosco From: Odon Copon <odonco...@gmail.com<mailto:odonco...@gmail.com>> Reply-To: <user@ranger.apache.org<mailto:user@ranger.apache.org>> Date: Friday, January 11, 2019 at 1:22 AM To: <user@ranger.apache.org<mailto:user@ranger.apache.org>> Subject: Re: Accessing Ranger Policy Manager API from HDFS plugin Mmm, but what if the system is not using Kerberos? On Fri, 11 Jan 2019, 04:21 Velmurugan Periasamy <vperias...@hortonworks.com<mailto:vperias...@hortonworks.com> wrote: Yes, that's what I referred to. ________________________________ From: Odon Copon <odonco...@gmail.com<mailto:odonco...@gmail.com>> Sent: Thursday, January 10, 2019 5:07 PM To: user@ranger.apache.org<mailto:user@ranger.apache.org> Subject: Re: Accessing Ranger Policy Manager API from HDFS plugin Are we talking about principal in Kerberos or any other principal I'm not understanding? On Thu, 10 Jan 2019 at 18:05, Odon Copon <odonco...@gmail.com<mailto:odonco...@gmail.com>> wrote: What do you mean by HDFS plugin uses service (Namenode) user's principal ? Could you provide an example? Thanks. On Thu, 10 Jan 2019 at 17:08, Velmurugan Periasamy <vperias...@hortonworks.com<mailto:vperias...@hortonworks.com>> wrote: HDFS plugin uses service (Namenode) user's principal. ________________________________ From: Odon Copon <odonco...@gmail.com<mailto:odonco...@gmail.com>> Sent: Thursday, January 10, 2019 8:59 AM To: user@ranger.apache.org<mailto:user@ranger.apache.org> Subject: Accessing Ranger Policy Manager API from HDFS plugin Hi, How does the Ranger HDFS plugin communicates with the Policy Manager API? Is it using a specific user/password combination? I know the User Sync has rangerusersync user and pass, and all that information is stored in rangerusersync.jceks, but what about the HDFS plugin or any other plugin? I'm having issues with that, my plugin once enabled doesn't get displayed in the UI and would like to check the credentials the plugin is using to use the API. For the User Sync - Policy Manager communication works fine. Thanks.
