Can you please check if the following exists in the hdfs-site.xml configuration
file in your environment?
<property>
<name>dfs.namenode.inode.attributes.provider.class</name>
<value>org.apache.ranger.authorization.hadoop.RangerHdfsAuthorizer</value>
</property>
-Abhay
On 1/11/19, 9:40 AM, "Kevin Risden" <kris...@apache.org> wrote:
How about the configuration files? Are they in the HDFS conf directory?
Kevin Risden
On Fri, Jan 11, 2019 at 12:38 PM Odon Copon <odonco...@gmail.com> wrote:
>
> In the logs I can see the classpath as one of the first messages it
prints. The following jars appear there:
> -
/usr/share/hadoop/share/hadoop/hdfs/lib/ranger-plugin-classloader-1.2.0.jar
> -
/usr/share/hadoop/share/hadoop/hdfs/lib/ranger-hdfs-plugin-shim-1.2.0.jar
>
> On Fri, 11 Jan 2019 at 17:27,
Kevin Risden <kris...@apache.org> wrote:
>>
>> Do you have the ranger plugin on the hdfs classpath? Could be that
nothing is getting picked up because it's not installed in the right location?
>>
>> Kevin Risden
>>
>> On Fri, Jan 11, 2019, 12:19 Odon Copon <odonco...@gmail.com wrote:
>>>
>>> Ok, I'm running out of ideas to debug the issue.
>>> No logs like Bosco posted before, and I cannot find a reason for this
not to work, when the endpoint is accessible and is correctly set in
install.properties.
>>>
>>> On Fri, 11 Jan 2019 at 15:42, Odon Copon <odonco...@gmail.com> wrote:
>>>>
>>>> Thanks Bosco, that's really helpful.
>>>> Ran the following search "grep -irnw '/' -e 'AuditProviderFactory'
--exclude \*.java --exclude \*.class --exclude \*.jar 2>/dev/null" trying to
look for "AuditProviderFactory" keyword, but there's nothing with that word in
it (exluding java, class and jar files). So no log files like yours.
>>>> What could be causing this? I'm restarting the namenode executing the
following command: "sudo /etc/init.d/hadoop-namenode restart", is this enough
or is there any other internal component that needs to be restarted as well?
like the nodemanager per example.
>>>>
>>>> On Fri, 11 Jan 2019 at 15:34, Don Bosco Durai <bo...@apache.org> wrote:
>>>>>
>>>>> You might want to look into the logs. Here are some of mine from the
HDFS namenode log ..
>>>>>
>>>>>
>>>>>
>>>>> 2019-01-10 06:52:22,128 INFO provider.AuditProviderFactory
(AuditProviderFactory.java:init(150)) - AUDIT PROPERTY:
ranger.plugin.hdfs.policy.rest.url=http://orange1.mydomain.com:6080
>>>>>
>>>>>
>>>>>
>>>>> 2019-01-10 06:52:22,200 INFO service.RangerBasePlugin
(RangerBasePlugin.java:init(151)) - PolicyEngineOptions: { evaluatorType: auto,
cacheAuditResult: true, disableContextEnrichers: false,
disableCustomConditions: false, disableTrieLookupPrefilter: false }
>>>>>
>>>>> 2019-01-10 06:52:23,274 INFO util.PolicyRefresher
(PolicyRefresher.java:loadPolicyfromPolicyAdmin(277)) -
PolicyRefresher(serviceName=orange_hadoop): found updated version.
lastKnownVersion=-1; newVersion=3
>>>>>
>>>>> 2019-01-10 06:52:23,337 INFO util.RangerResourceTrie
(RangerResourceTrie.java:<init>(112)) - resourceName=path; optIgnoreCase=false;
optWildcard=true; wildcardChars=*?{}\; nodeCount=18; leafNodeCount=1;
singleChildNodeCount=17; maxDepth=18; evaluatorListCount=0;
wildcardEvaluatorListCount=2; evaluatorListRefCount=17;
wildcardEvaluatorListRefCount=15
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> Bosco
>>>>>
>>>>>
>>>>>
>>>>> From: Odon Copon <odonco...@gmail.com>
>>>>> Reply-To: <user@ranger.apache.org>
>>>>> Date: Friday, January 11, 2019 at 7:23 AM
>>>>> To: <user@ranger.apache.org>
>>>>> Subject: Re: Accessing Ranger Policy Manager API from HDFS plugin
>>>>>
>>>>>
>>>>>
>>>>> On the namenode I'm editing install.properties file and then "sudo
./enable-hdfs-plugin.sh".
>>>>>
>>>>> Then I'm restarting the namenode service.
>>>>>
>>>>>
>>>>>
>>>>> On Fri, 11 Jan 2019 at 15:19, Don Bosco Durai <bo...@apache.org>
wrote:
>>>>>
>>>>> It is every 30 seconds. If you are not seeing anything, then it might
be a configuration issue. How are you enabling the HDFS plugin?
>>>>>
>>>>>
>>>>>
>>>>> Bosco
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> From: Odon Copon <odonco...@gmail.com>
>>>>> Reply-To: <user@ranger.apache.org>
>>>>> Date: Friday, January 11, 2019 at 7:18 AM
>>>>> To: <user@ranger.apache.org>
>>>>> Subject: Re: Accessing Ranger Policy Manager API from HDFS plugin
>>>>>
>>>>>
>>>>>
>>>>> 1.- How often is the plugin pulling policies from the API?
>>>>>
>>>>> 2.- I don't see anything in the logs regarding the plugin, how can I
ensure is it running correctly?
>>>>>
>>>>> 3.- If I run a manual CURL requesting the policies I can see the UI
is showing that attempt to retrieve the policies, with a 200. But just the
manual CURL requests, no requests coming from the plugin.
>>>>>
>>>>>
>>>>>
>>>>> On Fri, 11 Jan 2019 at 14:58, Don Bosco Durai <bo...@apache.org>
wrote:
>>>>>
>>>>> Also make sure you have the correct hostname and port for Ranger
Admin.
>>>>>
>>>>>
>>>>>
>>>>> Bosco
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> From: Velmurugan Periasamy <vperias...@hortonworks.com>
>>>>> Reply-To: <user@ranger.apache.org>
>>>>> Date: Friday, January 11, 2019 at 6:32 AM
>>>>> To: "user@ranger.apache.org" <user@ranger.apache.org>
>>>>> Subject: Re: Accessing Ranger Policy Manager API from HDFS plugin
>>>>>
>>>>>
>>>>>
>>>>> You can check namenode log for any errors from HDFS plugin.
>>>>>
>>>>>
>>>>>
>>>>> ________________________________
>>>>>
>>>>> From: Odon Copon <odonco...@gmail.com>
>>>>> Sent: Friday, January 11, 2019 9:21 AM
>>>>> To: user@ranger.apache.org
>>>>> Subject: Re: Accessing Ranger Policy Manager API from HDFS plugin
>>>>>
>>>>>
>>>>>
>>>>> I fired manually a CURL request to
"/service/plugins/policies/download/<service_name>" and now the UI is
displaying some information in plugin tab.
>>>>>
>>>>> 1. Is Ranger Admin thinking the call was made from the plugin and is
trying to list it?
>>>>>
>>>>> 2. If plugin would have executed this request, the UI should have
displayed this information earlier, right?
>>>>>
>>>>> 3. Any specific log to check for more information?
>>>>>
>>>>>
>>>>>
>>>>> On Fri, 11 Jan 2019 at 14:07, Velmurugan Periasamy
<vperias...@hortonworks.com> wrote:
>>>>>
>>>>> You should see plugin sync'ing policies in plugin tab. If it is not
showing up, you need to check the logs for any error messages.
>>>>>
>>>>> ________________________________
>>>>>
>>>>> From: Odon Copon <odonco...@gmail.com>
>>>>> Sent: Friday, January 11, 2019 8:47 AM
>>>>> To: user@ranger.apache.org
>>>>> Subject: Re: Accessing Ranger Policy Manager API from HDFS plugin
>>>>>
>>>>>
>>>>>
>>>>> ok, seems "service/plugins/policies/download/" has public access, so
confirms what we have been discussing, no authorization is required to download
the policies. Good to know, thanks guys.
>>>>>
>>>>> The question know is how to know the plugin is calling this endpoint,
and working properly, because the UI doesn't display this plugin. Any tip on
this?
>>>>>
>>>>>
>>>>>
>>>>> On Fri, 11 Jan 2019 at 13:08, Odon Copon <odonco...@gmail.com> wrote:
>>>>>
>>>>> Yes, makes sense to have a 2-way SSL between the plugin and Ranger
Admin, but:
>>>>>
>>>>> - 1. Does it mean there's no authentication at all between them?
>>>>>
>>>>> - 2. If there's no authentication, shouldn't a simple CURL work? At
the moment if no user/pass is provided the API returns 401, or is there another
different endpoint? If so, which one is it?
>>>>>
>>>>> - 3. What is the best way to debug the plugin is communicating or
trying to communicate with Ranger admin?
>>>>>
>>>>>
>>>>>
>>>>> Thanks.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Fri, 11 Jan 2019 at 12:53, Velmurugan Periasamy
<vperias...@hortonworks.com> wrote:
>>>>>
>>>>> If there is no kerberos HDFS plugin uses the open Download policies
API, so it is recommended to use 2-way SSL between HDFS plugin and Ranger Admin.
>>>>>
>>>>>
>>>>> On Jan 11, 2019, at 5:26 AM, Odon Copon <odonco...@gmail.com> wrote:
>>>>>
>>>>> I cannot perform a CURL to the API from the namenode without
user/password, I get a 401 when doing that. So it might required credentials to
do that. If I use the admin/password credentials or rangerusersync credentials
the CURL works. So wondering if those credentials need to be setup somewhere.
>>>>>
>>>>>
>>>>>
>>>>> On Fri, 11 Jan 2019 at 10:15, Don Bosco Durai <bo...@apache.org>
wrote:
>>>>>
>>>>> > In terms of "no authentication", is the HDFS plugin using Policy
Manager API with no credentials at all?
>>>>>
>>>>> No credentials, because there is no user/password for HDFS service
user. It’s been a while, I think we used to have admin/password before, but it
was taken out eventually. The code might be still there…
>>>>>
>>>>>
>>>>>
>>>>> > What's the first action the plugin is performing to be detected by
the UI as active and 200 response?
>>>>>
>>>>> Abhay or Madhan might be able to give you more specifics. Since the
plugins are polling and it knows the previous version number, if there are no
changes, then it is not registered in the UI. The plugins primarily pull the
policies and tags from Ranger Admin. Rest everything is done by the plugin
within the component.
>>>>>
>>>>>
>>>>>
>>>>> Bosco
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> From: Odon Copon <odonco...@gmail.com>
>>>>> Reply-To: <user@ranger.apache.org>
>>>>> Date: Friday, January 11, 2019 at 2:03 AM
>>>>> To: <user@ranger.apache.org>
>>>>> Subject: Re: Accessing Ranger Policy Manager API from HDFS plugin
>>>>>
>>>>>
>>>>>
>>>>> Indeed, I know that at the moment without something like Kerberos,
users can impersonate others, but I'm currently building a POC with the basic
security to evaluate Ranger, and once is ready, start improving the security
and scalability. But thank you for pointing that out.
>>>>>
>>>>> In terms of "no authentication", is the HDFS plugin using Policy
Manager API with no credentials at all? or default ones?
>>>>>
>>>>> What's the first action the plugin is performing to be detected by
the UI as active and 200 response? Some king of ping/heartbeat? or just a rest
petition to download the policies?
>>>>>
>>>>> Is there anywhere where I can see in the logs what kind of actions
the plugin is doing? I don't find any log information coming from the plugin.
>>>>>
>>>>>
>>>>>
>>>>> Thanks!
>>>>>
>>>>>
>>>>>
>>>>> On Fri, 11 Jan 2019 at 09:53, Don Bosco Durai <bo...@apache.org>
wrote:
>>>>>
>>>>> If there is no Kerberos, then you have 2 options:
>>>>>
>>>>> No authentication (default)
>>>>> Two way SSL to authenticate the request from the plugin.
>>>>>
>>>>>
>>>>>
>>>>> Note, if it is non-Kerberos environment, then authorization cannot be
enforced, because users can impersonate anyone.
>>>>>
>>>>>
>>>>>
>>>>> Bosco
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> From: Odon Copon <odonco...@gmail.com>
>>>>> Reply-To: <user@ranger.apache.org>
>>>>> Date: Friday, January 11, 2019 at 1:22 AM
>>>>> To: <user@ranger.apache.org>
>>>>> Subject: Re: Accessing Ranger Policy Manager API from HDFS plugin
>>>>>
>>>>>
>>>>>
>>>>> Mmm, but what if the system is not using Kerberos?
>>>>>
>>>>>
>>>>>
>>>>> On Fri, 11 Jan 2019, 04:21 Velmurugan Periasamy
<vperias...@hortonworks.com wrote:
>>>>>
>>>>> Yes, that's what I referred to.
>>>>>
>>>>>
>>>>>
>>>>> ________________________________
>>>>>
>>>>> From: Odon Copon <odonco...@gmail.com>
>>>>> Sent: Thursday, January 10, 2019 5:07 PM
>>>>> To: user@ranger.apache.org
>>>>> Subject: Re: Accessing Ranger Policy Manager API from HDFS plugin
>>>>>
>>>>>
>>>>>
>>>>> Are we talking about principal in Kerberos or any other principal I'm
not understanding?
>>>>>
>>>>>
>>>>>
>>>>> On Thu, 10 Jan 2019 at 18:05, Odon Copon <odonco...@gmail.com> wrote:
>>>>>
>>>>> What do you mean by HDFS plugin uses service (Namenode) user's
principal ?
>>>>>
>>>>> Could you provide an example?
>>>>> Thanks.
>>>>>
>>>>>
>>>>>
>>>>> On Thu, 10 Jan 2019 at 17:08, Velmurugan Periasamy
<vperias...@hortonworks.com> wrote:
>>>>>
>>>>> HDFS plugin uses service (Namenode) user's principal.
>>>>>
>>>>>
>>>>>
>>>>> ________________________________
>>>>>
>>>>> From: Odon Copon <odonco...@gmail.com>
>>>>> Sent: Thursday, January 10, 2019 8:59 AM
>>>>> To: user@ranger.apache.org
>>>>> Subject: Accessing Ranger Policy Manager API from HDFS plugin
>>>>>
>>>>>
>>>>>
>>>>> Hi,
>>>>>
>>>>> How does the Ranger HDFS plugin communicates with the Policy Manager
API? Is it using a specific user/password combination?
>>>>>
>>>>> I know the User Sync has rangerusersync user and pass, and all that
information is stored in rangerusersync.jceks, but what about the HDFS plugin
or any other plugin?
>>>>> I'm having issues with that, my plugin once enabled doesn't get
displayed in the UI and would like to check the credentials the plugin is using
to use the API.
>>>>>
>>>>> For the User Sync - Policy Manager communication works fine.
>>>>>
>>>>>
>>>>>
>>>>> Thanks.
