Re: Question about group policies

[Don Bosco Durai]

Ramesh, thanks.

Alex, it seems you are confusing between the Plugin Interface and the Ranger 
API (isAccessAllowed()).  

For HAWQ, you will be implementing the Plugin Interface (akin 
RangerKafkaAuthorizer) and in that plugin you will (or could) use Hadoop Common 
to get the groups for the users. If you share you plugin code, we can review 
and help.

> - Might there be a difference between handling LDAP groups vs. groups 
> manually created ?
To avoid confusion, you should ignore the users and groups created manually in 
Ranger Admin. The users and groups in Ranger Admin are there only to help 
associating users/groups to policies. If you are doing sync with LDAP, then you 
shouldn’t be manually creating any users or groups. This could be mis-leading 
for some.

Happy to clarify more if needed.

Thanks

Bosco



On 3/30/17, 2:39 PM, "Ramesh Mani"  wrote:

Alex,

What I  was mentioning when you do plugin.isAccessAllowed(request), your
request should contain both user / group  and to get the group information
you can use Hadoop UserGroupInformation API.

That is what Don Bosco Durai was mention in this last email.

Additional comments I have put against your questions below.

Thanks,
Ramesh



On 3/29/17, 3:14 PM, "Alexander Denissov"  wrote:

>Don, Ramesh, Abhay -- thank you for your replies.
>
>I am still quite confused, though :( While Ramesh and Abhay state that a
>client needs to provide group membership explicitly when calling
>isAccessAllowed() plugin API, Don implies that it is not necessary and we
>can only call with a username.
>
>Also, one of our engineers tested with LDAP groups and says that LDAP
>groups work, while groups created via Ranger UI do not. By "work" I mean
>when a user is a member of the group and only a group policy is defined,
>then passing only the username results in policy evaluating correctly and
>granting access to the resource. I have not yet tested this LDAP scenario
>myself.
Ramesh Mani: Groups created via Ranger must be the groups which are in 
OS
or in LDAP.
 Users and Groups that are in Ranger are only for Policy 
creation
and Login into Ranger admin.
>
>So, I'll try asking again:
>- Does the client have to pass user groups to API call or passing just a
>username is sufficient ?
Ramesh Mani: 
For  plugin.isAccessAllowed(request), request should have user 
and
group. Prior to building the request you will need to make the hadoop api
call to create the UGI and create the user and group and use it in the
request.
Refer this : 
https://github.com/apache/ranger/blob/master/plugin-kafka/src/main/java/org
/apache/ranger/authorization/kafka/authorizer/RangerKafkaAuthorizer.java#L1
39
Here plugin provides only the username to create UGI.
Hadoop user group mapping should be done correctly to get this user /
group mapping to be resolved . Also can make sure by doing “hdfs groups
” resolves to get the groups for that user for the user you are
doing.

Refer this parameter for more details:

https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/co
re-default.xml
org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback


>- If Ranger plugin is able to get user-group membership from Ranger
>Admin, does it happen during policy sync or as a separate process ? If
>separate, how often does the sync happen ?

Ramesh Mani:
Ranger Plugin gets only the policies defined in the ranger admin
periodically ( every 30 sec by default)
User and Groups are determined by the above mentioned hadoop 
api in the
plugin when request is created.

>- Might passing an empty set for roles parameter to the API circumvent
>automatic lookup (if such even exists) ? Should we pass null instead ?
Ramesh Mani :
Not sure which api you are mentioning here?
>- Might there be a difference between handling LDAP groups vs. groups
>manually created ?
Ramesh Mani.
The core-site.xml has set of param for LDAP user group mapping. 
Or other
methods to use SSSD / Centrify / NSLCD / Winbind  to connect linux OS with
LDAP. 
First you can try with  Linux OS level user group mapping.
>--
>Thanks,
>Alex.
>
>On 2017-03-24 16:12 (-0700), Don Bosco Durai  wrote:
>> Alex
>> 
>> Both Abhay and Ramesh are correct. In the Hadoop eco-system we want to
>>ensure that the users and groups are consistent across all components.
>>And generally, AD/LDAP or Unix system user/groups are the source of
>>truth.
>> 
>> 
>> >>This also means user <--> group mapping in Ranger 

Re: Question about group policies

[Ramesh Mani]

Alex,

What I  was mentioning when you do plugin.isAccessAllowed(request), your
request should contain both user / group  and to get the group information
you can use Hadoop UserGroupInformation API.

That is what Don Bosco Durai was mention in this last email.

Additional comments I have put against your questions below.

Thanks,
Ramesh



On 3/29/17, 3:14 PM, "Alexander Denissov"  wrote:

>Don, Ramesh, Abhay -- thank you for your replies.
>
>I am still quite confused, though :( While Ramesh and Abhay state that a
>client needs to provide group membership explicitly when calling
>isAccessAllowed() plugin API, Don implies that it is not necessary and we
>can only call with a username.
>
>Also, one of our engineers tested with LDAP groups and says that LDAP
>groups work, while groups created via Ranger UI do not. By "work" I mean
>when a user is a member of the group and only a group policy is defined,
>then passing only the username results in policy evaluating correctly and
>granting access to the resource. I have not yet tested this LDAP scenario
>myself.
Ramesh Mani: Groups created via Ranger must be the groups which are in 
OS
or in LDAP.
 Users and Groups that are in Ranger are only for Policy 
creation
and Login into Ranger admin.
>
>So, I'll try asking again:
>- Does the client have to pass user groups to API call or passing just a
>username is sufficient ?
Ramesh Mani: 
For  plugin.isAccessAllowed(request), request should have user 
and
group. Prior to building the request you will need to make the hadoop api
call to create the UGI and create the user and group and use it in the
request.
Refer this : 
https://github.com/apache/ranger/blob/master/plugin-kafka/src/main/java/org
/apache/ranger/authorization/kafka/authorizer/RangerKafkaAuthorizer.java#L1
39
Here plugin provides only the username to create UGI.
Hadoop user group mapping should be done correctly to get this user /
group mapping to be resolved . Also can make sure by doing “hdfs groups
” resolves to get the groups for that user for the user you are
doing.

Refer this parameter for more details:

https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-common/co
re-default.xml
org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback


>- If Ranger plugin is able to get user-group membership from Ranger
>Admin, does it happen during policy sync or as a separate process ? If
>separate, how often does the sync happen ?

Ramesh Mani:
Ranger Plugin gets only the policies defined in the ranger admin
periodically ( every 30 sec by default)
User and Groups are determined by the above mentioned hadoop 
api in the
plugin when request is created.

>- Might passing an empty set for roles parameter to the API circumvent
>automatic lookup (if such even exists) ? Should we pass null instead ?
Ramesh Mani :
Not sure which api you are mentioning here? 
>- Might there be a difference between handling LDAP groups vs. groups
>manually created ?
Ramesh Mani.
The core-site.xml has set of param for LDAP user group mapping. 
Or other
methods to use SSSD / Centrify / NSLCD / Winbind  to connect linux OS with
LDAP. 
First you can try with  Linux OS level user group mapping.
>--
>Thanks,
>Alex.
>
>On 2017-03-24 16:12 (-0700), Don Bosco Durai  wrote:
>> Alex
>> 
>> Both Abhay and Ramesh are correct. In the Hadoop eco-system we want to
>>ensure that the users and groups are consistent across all components.
>>And generally, AD/LDAP or Unix system user/groups are the source of
>>truth.
>> 
>> 
>> >>This also means user <--> group mapping in Ranger is NOT the
>>source of
>> >>truth, but rather a mirror of some other authentication system
>>(OS, LDAP,
>> >>etc) and a service will need to fetch this information upon user
>> >>authentication and provide to Ranger ?
>> 
>> Based on some of the earlier discussion on the HAWQ/Ranger integration
>>design model, it was decided to run the Ranger Plugin in another
>>process. If it is still the case, you just need to send the user from
>>the HAWQ side and the Ranger Plugin would be able to get the groups from
>>the AD/LDAP/Unix using Hadoop Common APIs. Ranger does the same for
>>Kafka and Solr integrations, because both these systems call the Ranger
>>plugin API only with the username.
>> 
>> Let me know if you need additional information.
>> 
>> Thanks
>> 
>> Bosco
>> 
>> 
>> 
>> 
>> On 3/24/17, 12:50 PM, "Ramesh Mani"  wrote:
>> 
>> Adding to Abhay comment,
>> 
>> In most of the Ranger Plugin from the components side we use
>> org.apache.hadoop.security.UserGroupInformation API
>> 
>>https://hadoop.apache.org/docs/r1.0.4/api/org/apache/hadoop/security/User
>>Gr

Re: Question about group policies

[Don Bosco Durai]

Alex

Both Abhay and Ramesh are correct. In the Hadoop eco-system we want to ensure 
that the users and groups are consistent across all components. And generally, 
AD/LDAP or Unix system user/groups are the source of truth.


>>This also means user <--> group mapping in Ranger is NOT the source of
>>truth, but rather a mirror of some other authentication system (OS, LDAP,
>>etc) and a service will need to fetch this information upon user
>>authentication and provide to Ranger ?

Based on some of the earlier discussion on the HAWQ/Ranger integration design 
model, it was decided to run the Ranger Plugin in another process. If it is 
still the case, you just need to send the user from the HAWQ side and the 
Ranger Plugin would be able to get the groups from the AD/LDAP/Unix using 
Hadoop Common APIs. Ranger does the same for Kafka and Solr integrations, 
because both these systems call the Ranger plugin API only with the username.

Let me know if you need additional information.

Thanks

Bosco




On 3/24/17, 12:50 PM, "Ramesh Mani"  wrote:

Adding to Abhay comment,

In most of the Ranger Plugin from the components side we use
org.apache.hadoop.security.UserGroupInformation API
https://hadoop.apache.org/docs/r1.0.4/api/org/apache/hadoop/security/UserGr
oupInformation.html which will wrap around JAAS and provides the mechanism
to determine the User and Groups. Please check if this can be used.

Thanks,
Ramesh

On 3/24/17, 12:03 PM, "Abhay Kulkarni"  wrote:

>Hi Alex,
>
>This is exactly right. Users, groups and their associations in Ranger
>(specifically Ranger Admin) are props for being able to define policies.
>They are not the Œsource of truth¹. It is expected that the correct user
><‹-> group associations will be available in the component (service) from
>appropriate authentication system, and provided to Ranger Plugin as part
>of authorization request.
>
>Thanks!
>-Abhay
>
>On 3/24/17, 11:51 AM, "Alexander Denissov"  wrote:
>
>>Hi Ranger experts,
>>
>>We are developing a custom Ranger Plugin for Apache HAWQ(incubating) and
>>noticed that group policies are not behaving as we expected.
>>
>>In Ranger, we define a user U (actually synched from OS). We then
>>manually
>>define group G and enroll user U into it. We then define a policy and
>>grant
>>a privilege to the group G in this policy.
>>
>>On the client side, we do not know that user U belongs to group G, as
>>this
>>information is only defined in Ranger. When we request policy evaluation,
>>we send an empty set for the userGroups API parameter, assuming Ranger
>>will
>>use its internal mapping. But the access is denied by Ranger.
>>
>>So, it seems Ranger will not use the information from its internal user
>><--> group mapping when evaluating policies and would rely on client
>>providing the set of groups for the user explicitly ?
>>
>>This also means user <--> group mapping in Ranger is NOT the source of
>>truth, but rather a mirror of some other authentication system (OS, LDAP,
>>etc) and a service will need to fetch this information upon user
>>authentication and provide to Ranger ?
>>
>>I will appreciate clarification on these points.
>>--
>>Thanks,
>>Alex.
>
>

Re: Question about group policies

[Ramesh Mani]

Adding to Abhay comment,

In most of the Ranger Plugin from the components side we use
org.apache.hadoop.security.UserGroupInformation API
https://hadoop.apache.org/docs/r1.0.4/api/org/apache/hadoop/security/UserGr
oupInformation.html which will wrap around JAAS and provides the mechanism
to determine the User and Groups. Please check if this can be used.

Thanks,
Ramesh

On 3/24/17, 12:03 PM, "Abhay Kulkarni"  wrote:

>Hi Alex,
>
>This is exactly right. Users, groups and their associations in Ranger
>(specifically Ranger Admin) are props for being able to define policies.
>They are not the Œsource of truth¹. It is expected that the correct user
><‹-> group associations will be available in the component (service) from
>appropriate authentication system, and provided to Ranger Plugin as part
>of authorization request.
>
>Thanks!
>-Abhay
>
>On 3/24/17, 11:51 AM, "Alexander Denissov"  wrote:
>
>>Hi Ranger experts,
>>
>>We are developing a custom Ranger Plugin for Apache HAWQ(incubating) and
>>noticed that group policies are not behaving as we expected.
>>
>>In Ranger, we define a user U (actually synched from OS). We then
>>manually
>>define group G and enroll user U into it. We then define a policy and
>>grant
>>a privilege to the group G in this policy.
>>
>>On the client side, we do not know that user U belongs to group G, as
>>this
>>information is only defined in Ranger. When we request policy evaluation,
>>we send an empty set for the userGroups API parameter, assuming Ranger
>>will
>>use its internal mapping. But the access is denied by Ranger.
>>
>>So, it seems Ranger will not use the information from its internal user
>><--> group mapping when evaluating policies and would rely on client
>>providing the set of groups for the user explicitly ?
>>
>>This also means user <--> group mapping in Ranger is NOT the source of
>>truth, but rather a mirror of some other authentication system (OS, LDAP,
>>etc) and a service will need to fetch this information upon user
>>authentication and provide to Ranger ?
>>
>>I will appreciate clarification on these points.
>>--
>>Thanks,
>>Alex.
>
>

Re: Question about group policies

[Abhay Kulkarni]

Hi Alex,

This is exactly right. Users, groups and their associations in Ranger
(specifically Ranger Admin) are props for being able to define policies.
They are not the Œsource of truth¹. It is expected that the correct user
<‹-> group associations will be available in the component (service) from
appropriate authentication system, and provided to Ranger Plugin as part
of authorization request.

Thanks!
-Abhay

On 3/24/17, 11:51 AM, "Alexander Denissov"  wrote:

>Hi Ranger experts,
>
>We are developing a custom Ranger Plugin for Apache HAWQ(incubating) and
>noticed that group policies are not behaving as we expected.
>
>In Ranger, we define a user U (actually synched from OS). We then manually
>define group G and enroll user U into it. We then define a policy and
>grant
>a privilege to the group G in this policy.
>
>On the client side, we do not know that user U belongs to group G, as this
>information is only defined in Ranger. When we request policy evaluation,
>we send an empty set for the userGroups API parameter, assuming Ranger
>will
>use its internal mapping. But the access is denied by Ranger.
>
>So, it seems Ranger will not use the information from its internal user
><--> group mapping when evaluating policies and would rely on client
>providing the set of groups for the user explicitly ?
>
>This also means user <--> group mapping in Ranger is NOT the source of
>truth, but rather a mirror of some other authentication system (OS, LDAP,
>etc) and a service will need to fetch this information upon user
>authentication and provide to Ranger ?
>
>I will appreciate clarification on these points.
>--
>Thanks,
>Alex.

Question about group policies

[Alexander Denissov]

Hi Ranger experts,

We are developing a custom Ranger Plugin for Apache HAWQ(incubating) and
noticed that group policies are not behaving as we expected.

In Ranger, we define a user U (actually synched from OS). We then manually
define group G and enroll user U into it. We then define a policy and grant
a privilege to the group G in this policy.

On the client side, we do not know that user U belongs to group G, as this
information is only defined in Ranger. When we request policy evaluation,
we send an empty set for the userGroups API parameter, assuming Ranger will
use its internal mapping. But the access is denied by Ranger.

So, it seems Ranger will not use the information from its internal user
<--> group mapping when evaluating policies and would rely on client
providing the set of groups for the user explicitly ?

This also means user <--> group mapping in Ranger is NOT the source of
truth, but rather a mirror of some other authentication system (OS, LDAP,
etc) and a service will need to fetch this information upon user
authentication and provide to Ranger ?

I will appreciate clarification on these points.
--
Thanks,
Alex.