How about the configuration files? Are they in the HDFS conf directory? Kevin Risden
On Fri, Jan 11, 2019 at 12:38 PM Odon Copon <odonco...@gmail.com> wrote: > > In the logs I can see the classpath as one of the first messages it prints. > The following jars appear there: > - /usr/share/hadoop/share/hadoop/hdfs/lib/ranger-plugin-classloader-1.2.0.jar > - /usr/share/hadoop/share/hadoop/hdfs/lib/ranger-hdfs-plugin-shim-1.2.0.jar > > On Fri, 11 Jan 2019 at 17:27, Kevin Risden <kris...@apache.org> wrote: >> >> Do you have the ranger plugin on the hdfs classpath? Could be that nothing >> is getting picked up because it's not installed in the right location? >> >> Kevin Risden >> >> On Fri, Jan 11, 2019, 12:19 Odon Copon <odonco...@gmail.com wrote: >>> >>> Ok, I'm running out of ideas to debug the issue. >>> No logs like Bosco posted before, and I cannot find a reason for this not >>> to work, when the endpoint is accessible and is correctly set in >>> install.properties. >>> >>> On Fri, 11 Jan 2019 at 15:42, Odon Copon <odonco...@gmail.com> wrote: >>>> >>>> Thanks Bosco, that's really helpful. >>>> Ran the following search "grep -irnw '/' -e 'AuditProviderFactory' >>>> --exclude \*.java --exclude \*.class --exclude \*.jar 2>/dev/null" trying >>>> to look for "AuditProviderFactory" keyword, but there's nothing with that >>>> word in it (exluding java, class and jar files). So no log files like >>>> yours. >>>> What could be causing this? I'm restarting the namenode executing the >>>> following command: "sudo /etc/init.d/hadoop-namenode restart", is this >>>> enough or is there any other internal component that needs to be restarted >>>> as well? like the nodemanager per example. >>>> >>>> On Fri, 11 Jan 2019 at 15:34, Don Bosco Durai <bo...@apache.org> wrote: >>>>> >>>>> You might want to look into the logs. Here are some of mine from the HDFS >>>>> namenode log .. >>>>> >>>>> >>>>> >>>>> 2019-01-10 06:52:22,128 INFO provider.AuditProviderFactory >>>>> (AuditProviderFactory.java:init(150)) - AUDIT PROPERTY: >>>>> ranger.plugin.hdfs.policy.rest.url=http://orange1.mydomain.com:6080 >>>>> >>>>> >>>>> >>>>> 2019-01-10 06:52:22,200 INFO service.RangerBasePlugin >>>>> (RangerBasePlugin.java:init(151)) - PolicyEngineOptions: { evaluatorType: >>>>> auto, cacheAuditResult: true, disableContextEnrichers: false, >>>>> disableCustomConditions: false, disableTrieLookupPrefilter: false } >>>>> >>>>> 2019-01-10 06:52:23,274 INFO util.PolicyRefresher >>>>> (PolicyRefresher.java:loadPolicyfromPolicyAdmin(277)) - >>>>> PolicyRefresher(serviceName=orange_hadoop): found updated version. >>>>> lastKnownVersion=-1; newVersion=3 >>>>> >>>>> 2019-01-10 06:52:23,337 INFO util.RangerResourceTrie >>>>> (RangerResourceTrie.java:<init>(112)) - resourceName=path; >>>>> optIgnoreCase=false; optWildcard=true; wildcardChars=*?{}\; nodeCount=18; >>>>> leafNodeCount=1; singleChildNodeCount=17; maxDepth=18; >>>>> evaluatorListCount=0; wildcardEvaluatorListCount=2; >>>>> evaluatorListRefCount=17; wildcardEvaluatorListRefCount=15 >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> Bosco >>>>> >>>>> >>>>> >>>>> From: Odon Copon <odonco...@gmail.com> >>>>> Reply-To: <user@ranger.apache.org> >>>>> Date: Friday, January 11, 2019 at 7:23 AM >>>>> To: <user@ranger.apache.org> >>>>> Subject: Re: Accessing Ranger Policy Manager API from HDFS plugin >>>>> >>>>> >>>>> >>>>> On the namenode I'm editing install.properties file and then "sudo >>>>> ./enable-hdfs-plugin.sh". >>>>> >>>>> Then I'm restarting the namenode service. >>>>> >>>>> >>>>> >>>>> On Fri, 11 Jan 2019 at 15:19, Don Bosco Durai <bo...@apache.org> wrote: >>>>> >>>>> It is every 30 seconds. If you are not seeing anything, then it might be >>>>> a configuration issue. How are you enabling the HDFS plugin? >>>>> >>>>> >>>>> >>>>> Bosco >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> From: Odon Copon <odonco...@gmail.com> >>>>> Reply-To: <user@ranger.apache.org> >>>>> Date: Friday, January 11, 2019 at 7:18 AM >>>>> To: <user@ranger.apache.org> >>>>> Subject: Re: Accessing Ranger Policy Manager API from HDFS plugin >>>>> >>>>> >>>>> >>>>> 1.- How often is the plugin pulling policies from the API? >>>>> >>>>> 2.- I don't see anything in the logs regarding the plugin, how can I >>>>> ensure is it running correctly? >>>>> >>>>> 3.- If I run a manual CURL requesting the policies I can see the UI is >>>>> showing that attempt to retrieve the policies, with a 200. But just the >>>>> manual CURL requests, no requests coming from the plugin. >>>>> >>>>> >>>>> >>>>> On Fri, 11 Jan 2019 at 14:58, Don Bosco Durai <bo...@apache.org> wrote: >>>>> >>>>> Also make sure you have the correct hostname and port for Ranger Admin. >>>>> >>>>> >>>>> >>>>> Bosco >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> From: Velmurugan Periasamy <vperias...@hortonworks.com> >>>>> Reply-To: <user@ranger.apache.org> >>>>> Date: Friday, January 11, 2019 at 6:32 AM >>>>> To: "user@ranger.apache.org" <user@ranger.apache.org> >>>>> Subject: Re: Accessing Ranger Policy Manager API from HDFS plugin >>>>> >>>>> >>>>> >>>>> You can check namenode log for any errors from HDFS plugin. >>>>> >>>>> >>>>> >>>>> ________________________________ >>>>> >>>>> From: Odon Copon <odonco...@gmail.com> >>>>> Sent: Friday, January 11, 2019 9:21 AM >>>>> To: user@ranger.apache.org >>>>> Subject: Re: Accessing Ranger Policy Manager API from HDFS plugin >>>>> >>>>> >>>>> >>>>> I fired manually a CURL request to >>>>> "/service/plugins/policies/download/<service_name>" and now the UI is >>>>> displaying some information in plugin tab. >>>>> >>>>> 1. Is Ranger Admin thinking the call was made from the plugin and is >>>>> trying to list it? >>>>> >>>>> 2. If plugin would have executed this request, the UI should have >>>>> displayed this information earlier, right? >>>>> >>>>> 3. Any specific log to check for more information? >>>>> >>>>> >>>>> >>>>> On Fri, 11 Jan 2019 at 14:07, Velmurugan Periasamy >>>>> <vperias...@hortonworks.com> wrote: >>>>> >>>>> You should see plugin sync'ing policies in plugin tab. If it is not >>>>> showing up, you need to check the logs for any error messages. >>>>> >>>>> ________________________________ >>>>> >>>>> From: Odon Copon <odonco...@gmail.com> >>>>> Sent: Friday, January 11, 2019 8:47 AM >>>>> To: user@ranger.apache.org >>>>> Subject: Re: Accessing Ranger Policy Manager API from HDFS plugin >>>>> >>>>> >>>>> >>>>> ok, seems "service/plugins/policies/download/" has public access, so >>>>> confirms what we have been discussing, no authorization is required to >>>>> download the policies. Good to know, thanks guys. >>>>> >>>>> The question know is how to know the plugin is calling this endpoint, and >>>>> working properly, because the UI doesn't display this plugin. Any tip on >>>>> this? >>>>> >>>>> >>>>> >>>>> On Fri, 11 Jan 2019 at 13:08, Odon Copon <odonco...@gmail.com> wrote: >>>>> >>>>> Yes, makes sense to have a 2-way SSL between the plugin and Ranger Admin, >>>>> but: >>>>> >>>>> - 1. Does it mean there's no authentication at all between them? >>>>> >>>>> - 2. If there's no authentication, shouldn't a simple CURL work? At the >>>>> moment if no user/pass is provided the API returns 401, or is there >>>>> another different endpoint? If so, which one is it? >>>>> >>>>> - 3. What is the best way to debug the plugin is communicating or trying >>>>> to communicate with Ranger admin? >>>>> >>>>> >>>>> >>>>> Thanks. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> On Fri, 11 Jan 2019 at 12:53, Velmurugan Periasamy >>>>> <vperias...@hortonworks.com> wrote: >>>>> >>>>> If there is no kerberos HDFS plugin uses the open Download policies API, >>>>> so it is recommended to use 2-way SSL between HDFS plugin and Ranger >>>>> Admin. >>>>> >>>>> >>>>> On Jan 11, 2019, at 5:26 AM, Odon Copon <odonco...@gmail.com> wrote: >>>>> >>>>> I cannot perform a CURL to the API from the namenode without >>>>> user/password, I get a 401 when doing that. So it might required >>>>> credentials to do that. If I use the admin/password credentials or >>>>> rangerusersync credentials the CURL works. So wondering if those >>>>> credentials need to be setup somewhere. >>>>> >>>>> >>>>> >>>>> On Fri, 11 Jan 2019 at 10:15, Don Bosco Durai <bo...@apache.org> wrote: >>>>> >>>>> > In terms of "no authentication", is the HDFS plugin using Policy >>>>> > Manager API with no credentials at all? >>>>> >>>>> No credentials, because there is no user/password for HDFS service user. >>>>> It’s been a while, I think we used to have admin/password before, but it >>>>> was taken out eventually. The code might be still there… >>>>> >>>>> >>>>> >>>>> > What's the first action the plugin is performing to be detected by the >>>>> > UI as active and 200 response? >>>>> >>>>> Abhay or Madhan might be able to give you more specifics. Since the >>>>> plugins are polling and it knows the previous version number, if there >>>>> are no changes, then it is not registered in the UI. The plugins >>>>> primarily pull the policies and tags from Ranger Admin. Rest everything >>>>> is done by the plugin within the component. >>>>> >>>>> >>>>> >>>>> Bosco >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> From: Odon Copon <odonco...@gmail.com> >>>>> Reply-To: <user@ranger.apache.org> >>>>> Date: Friday, January 11, 2019 at 2:03 AM >>>>> To: <user@ranger.apache.org> >>>>> Subject: Re: Accessing Ranger Policy Manager API from HDFS plugin >>>>> >>>>> >>>>> >>>>> Indeed, I know that at the moment without something like Kerberos, users >>>>> can impersonate others, but I'm currently building a POC with the basic >>>>> security to evaluate Ranger, and once is ready, start improving the >>>>> security and scalability. But thank you for pointing that out. >>>>> >>>>> In terms of "no authentication", is the HDFS plugin using Policy Manager >>>>> API with no credentials at all? or default ones? >>>>> >>>>> What's the first action the plugin is performing to be detected by the UI >>>>> as active and 200 response? Some king of ping/heartbeat? or just a rest >>>>> petition to download the policies? >>>>> >>>>> Is there anywhere where I can see in the logs what kind of actions the >>>>> plugin is doing? I don't find any log information coming from the plugin. >>>>> >>>>> >>>>> >>>>> Thanks! >>>>> >>>>> >>>>> >>>>> On Fri, 11 Jan 2019 at 09:53, Don Bosco Durai <bo...@apache.org> wrote: >>>>> >>>>> If there is no Kerberos, then you have 2 options: >>>>> >>>>> No authentication (default) >>>>> Two way SSL to authenticate the request from the plugin. >>>>> >>>>> >>>>> >>>>> Note, if it is non-Kerberos environment, then authorization cannot be >>>>> enforced, because users can impersonate anyone. >>>>> >>>>> >>>>> >>>>> Bosco >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> From: Odon Copon <odonco...@gmail.com> >>>>> Reply-To: <user@ranger.apache.org> >>>>> Date: Friday, January 11, 2019 at 1:22 AM >>>>> To: <user@ranger.apache.org> >>>>> Subject: Re: Accessing Ranger Policy Manager API from HDFS plugin >>>>> >>>>> >>>>> >>>>> Mmm, but what if the system is not using Kerberos? >>>>> >>>>> >>>>> >>>>> On Fri, 11 Jan 2019, 04:21 Velmurugan Periasamy >>>>> <vperias...@hortonworks.com wrote: >>>>> >>>>> Yes, that's what I referred to. >>>>> >>>>> >>>>> >>>>> ________________________________ >>>>> >>>>> From: Odon Copon <odonco...@gmail.com> >>>>> Sent: Thursday, January 10, 2019 5:07 PM >>>>> To: user@ranger.apache.org >>>>> Subject: Re: Accessing Ranger Policy Manager API from HDFS plugin >>>>> >>>>> >>>>> >>>>> Are we talking about principal in Kerberos or any other principal I'm not >>>>> understanding? >>>>> >>>>> >>>>> >>>>> On Thu, 10 Jan 2019 at 18:05, Odon Copon <odonco...@gmail.com> wrote: >>>>> >>>>> What do you mean by HDFS plugin uses service (Namenode) user's principal ? >>>>> >>>>> Could you provide an example? >>>>> Thanks. >>>>> >>>>> >>>>> >>>>> On Thu, 10 Jan 2019 at 17:08, Velmurugan Periasamy >>>>> <vperias...@hortonworks.com> wrote: >>>>> >>>>> HDFS plugin uses service (Namenode) user's principal. >>>>> >>>>> >>>>> >>>>> ________________________________ >>>>> >>>>> From: Odon Copon <odonco...@gmail.com> >>>>> Sent: Thursday, January 10, 2019 8:59 AM >>>>> To: user@ranger.apache.org >>>>> Subject: Accessing Ranger Policy Manager API from HDFS plugin >>>>> >>>>> >>>>> >>>>> Hi, >>>>> >>>>> How does the Ranger HDFS plugin communicates with the Policy Manager API? >>>>> Is it using a specific user/password combination? >>>>> >>>>> I know the User Sync has rangerusersync user and pass, and all that >>>>> information is stored in rangerusersync.jceks, but what about the HDFS >>>>> plugin or any other plugin? >>>>> I'm having issues with that, my plugin once enabled doesn't get displayed >>>>> in the UI and would like to check the credentials the plugin is using to >>>>> use the API. >>>>> >>>>> For the User Sync - Policy Manager communication works fine. >>>>> >>>>> >>>>> >>>>> Thanks.
