Re: Review Request 60949: Optimize Ranger code for authorization of HDFS 'getContentSummary' and 'delete' commands

2017-07-26 Thread Madhan Neethiraj 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60949/#review181466
---


Ship it!




Ship It!

- Madhan Neethiraj


On July 25, 2017, 10:22 p.m., Abhay Kulkarni wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60949/
> ---
> 
> (Updated July 25, 2017, 10:22 p.m.)
> 
> 
> Review request for ranger and Madhan Neethiraj.
> 
> 
> Bugs: RANGER-1695
> https://issues.apache.org/jira/browse/RANGER-1695
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> HDFS commands such as getContentSummary and delete require Ranger to 
> authorize access to entire directory hierarchy rooted at the directory passed 
> in as argument. Ranger could optimize this code by pruning the directory tree 
> whenever a sub-directory, and directory hierarchy rooted in it, is authorized 
> for requested access. This could potentially save many spurious authorization 
> API calls with Ranger policy engine.
> Also, it will be useful to gather and log performance metrics for the Ranger 
> authorization API call for every Ranger plugin.
> 
> 
> Diffs
> -
> 
>   
> agents-common/src/main/java/org/apache/ranger/authorization/hadoop/constants/RangerHadoopConstants.java
>  83f720a 
>   
> agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyEngineImpl.java
>  c72c8b5 
>   
> agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyEngineOptions.java
>  63162f6 
>   
> agents-common/src/main/java/org/apache/ranger/plugin/service/RangerBasePlugin.java
>  06b8f4d 
>   
> agents-common/src/main/java/org/apache/ranger/plugin/util/RangerPerfCollectorTracer.java
>  d899c6f 
>   
> hbase-agent/src/main/java/org/apache/ranger/authorization/hbase/RangerAuthorizationCoprocessor.java
>  cf2ffcf 
>   
> hdfs-agent/src/main/java/org/apache/ranger/authorization/hadoop/RangerHdfsAuthorizer.java
>  d28685a 
>   
> hdfs-agent/src/test/java/org/apache/ranger/services/hdfs/HDFSRangerTest.java 
> fe6a705 
>   hdfs-agent/src/test/resources/hdfs-policies.json 6055dd5 
>   hdfs-agent/src/test/resources/log4j.properties PRE-CREATION 
>   hdfs-agent/src/test/resources/ranger-hdfs-security.xml 9b1176e 
>   
> hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAuthorizer.java
>  56ef187 
>   
> knox-agent/src/main/java/org/apache/ranger/authorization/knox/RangerPDPKnoxFilter.java
>  1d58b21 
>   
> plugin-atlas/src/main/java/org/apache/ranger/authorization/atlas/authorizer/RangerAtlasAuthorizer.java
>  9712f95 
>   
> plugin-kafka/src/main/java/org/apache/ranger/authorization/kafka/authorizer/RangerKafkaAuthorizer.java
>  b6e052f 
>   
> plugin-kms/src/main/java/org/apache/ranger/authorization/kms/authorizer/RangerKmsAuthorizer.java
>  4cda8fa 
>   
> plugin-solr/src/main/java/org/apache/ranger/authorization/solr/authorizer/RangerSolrAuthorizer.java
>  0cc9de9 
>   
> plugin-yarn/src/main/java/org/apache/ranger/authorization/yarn/authorizer/RangerYarnAuthorizer.java
>  c589060 
>   ranger-tools/conf/log4j.properties 4ead802 
>   
> ranger-tools/src/main/java/org/apache/ranger/policyengine/RangerPolicyenginePerfTester.java
>  056c548 
>   
> storm-agent/src/main/java/org/apache/ranger/authorization/storm/authorizer/RangerStormAuthorizer.java
>  23c2b5f 
> 
> 
> Diff: https://reviews.apache.org/r/60949/diff/2/
> 
> 
> Testing
> ---
> 
> Tested with local VM
> 
> 
> Thanks,
> 
> Abhay Kulkarni
> 
>

Re: Review Request 60949: Optimize Ranger code for authorization of HDFS 'getContentSummary' and 'delete' commands

2017-07-25 Thread Abhay Kulkarni 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60949/
---

(Updated July 25, 2017, 10:22 p.m.)


Review request for ranger and Madhan Neethiraj.


Changes
---

Addressed review comments


Bugs: RANGER-1695
https://issues.apache.org/jira/browse/RANGER-1695


Repository: ranger


Description
---

HDFS commands such as getContentSummary and delete require Ranger to authorize 
access to entire directory hierarchy rooted at the directory passed in as 
argument. Ranger could optimize this code by pruning the directory tree 
whenever a sub-directory, and directory hierarchy rooted in it, is authorized 
for requested access. This could potentially save many spurious authorization 
API calls with Ranger policy engine.
Also, it will be useful to gather and log performance metrics for the Ranger 
authorization API call for every Ranger plugin.


Diffs (updated)
-

  
agents-common/src/main/java/org/apache/ranger/authorization/hadoop/constants/RangerHadoopConstants.java
 83f720a 
  
agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyEngineImpl.java
 c72c8b5 
  
agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyEngineOptions.java
 63162f6 
  
agents-common/src/main/java/org/apache/ranger/plugin/service/RangerBasePlugin.java
 06b8f4d 
  
agents-common/src/main/java/org/apache/ranger/plugin/util/RangerPerfCollectorTracer.java
 d899c6f 
  
hbase-agent/src/main/java/org/apache/ranger/authorization/hbase/RangerAuthorizationCoprocessor.java
 cf2ffcf 
  
hdfs-agent/src/main/java/org/apache/ranger/authorization/hadoop/RangerHdfsAuthorizer.java
 d28685a 
  hdfs-agent/src/test/java/org/apache/ranger/services/hdfs/HDFSRangerTest.java 
fe6a705 
  hdfs-agent/src/test/resources/hdfs-policies.json 6055dd5 
  hdfs-agent/src/test/resources/log4j.properties PRE-CREATION 
  hdfs-agent/src/test/resources/ranger-hdfs-security.xml 9b1176e 
  
hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAuthorizer.java
 56ef187 
  
knox-agent/src/main/java/org/apache/ranger/authorization/knox/RangerPDPKnoxFilter.java
 1d58b21 
  
plugin-atlas/src/main/java/org/apache/ranger/authorization/atlas/authorizer/RangerAtlasAuthorizer.java
 9712f95 
  
plugin-kafka/src/main/java/org/apache/ranger/authorization/kafka/authorizer/RangerKafkaAuthorizer.java
 b6e052f 
  
plugin-kms/src/main/java/org/apache/ranger/authorization/kms/authorizer/RangerKmsAuthorizer.java
 4cda8fa 
  
plugin-solr/src/main/java/org/apache/ranger/authorization/solr/authorizer/RangerSolrAuthorizer.java
 0cc9de9 
  
plugin-yarn/src/main/java/org/apache/ranger/authorization/yarn/authorizer/RangerYarnAuthorizer.java
 c589060 
  ranger-tools/conf/log4j.properties 4ead802 
  
ranger-tools/src/main/java/org/apache/ranger/policyengine/RangerPolicyenginePerfTester.java
 056c548 
  
storm-agent/src/main/java/org/apache/ranger/authorization/storm/authorizer/RangerStormAuthorizer.java
 23c2b5f 


Diff: https://reviews.apache.org/r/60949/diff/2/

Changes: https://reviews.apache.org/r/60949/diff/1-2/


Testing
---

Tested with local VM


Thanks,

Abhay Kulkarni

Re: Review Request 60949: Optimize Ranger code for authorization of HDFS 'getContentSummary' and 'delete' commands

2017-07-24 Thread Madhan Neethiraj 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60949/#review181314
---




hbase-agent/src/main/java/org/apache/ranger/authorization/hbase/RangerAuthorizationCoprocessor.java
Lines 340 (patched)


evaluateAccess() is used only for column family authorizations. Some 
requirePermission() calls end up calling AuthorizationSession.authorize(), 
without hitting evaluateAccess() - like postGetTableDescriptors(), 
authorizeAccess(),  Please review.

Consider adding perf-log to a higher level method like requirePermission() 
- which can capture the time taken by audit as well.

Even with this change, some authorizatin calls from the component would end 
up producing multiple perf-logs - for example: preCheckAndDelete(), 
preCheckAndPut(), preIncrementColumnValue(), postListProcedures(). It will be 
good to address this as well.



hdfs-agent/src/main/java/org/apache/ranger/authorization/hadoop/RangerHdfsAuthorizer.java
Lines 409 (patched)


Time taken for native authorization is aleady included in 
checkPermission(). Additional level of perf measurement may not be useful.



plugin-kafka/src/main/java/org/apache/ranger/authorization/kafka/authorizer/RangerKafkaAuthorizer.java
Lines 196 (patched)


Consider moving this to earlier point, @ line #134 - to ensure that the log 
captures the total time spent in the authorizer. Similarly for other authorizer 
calls as well.


- Madhan Neethiraj


On July 19, 2017, 12:57 a.m., Abhay Kulkarni wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60949/
> ---
> 
> (Updated July 19, 2017, 12:57 a.m.)
> 
> 
> Review request for ranger and Madhan Neethiraj.
> 
> 
> Bugs: RANGER-1695
> https://issues.apache.org/jira/browse/RANGER-1695
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> HDFS commands such as getContentSummary and delete require Ranger to 
> authorize access to entire directory hierarchy rooted at the directory passed 
> in as argument. Ranger could optimize this code by pruning the directory tree 
> whenever a sub-directory, and directory hierarchy rooted in it, is authorized 
> for requested access. This could potentially save many spurious authorization 
> API calls with Ranger policy engine.
> Also, it will be useful to gather and log performance metrics for the Ranger 
> authorization API call for every Ranger plugin.
> 
> 
> Diffs
> -
> 
>   
> agents-common/src/main/java/org/apache/ranger/authorization/hadoop/constants/RangerHadoopConstants.java
>  83f720a 
>   
> agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyEngineImpl.java
>  101013b 
>   
> agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyEngineOptions.java
>  63162f6 
>   
> agents-common/src/main/java/org/apache/ranger/plugin/service/RangerBasePlugin.java
>  06b8f4d 
>   
> agents-common/src/main/java/org/apache/ranger/plugin/util/RangerPerfCollectorTracer.java
>  d899c6f 
>   
> hbase-agent/src/main/java/org/apache/ranger/authorization/hbase/RangerAuthorizationCoprocessor.java
>  cf2ffcf 
>   
> hdfs-agent/src/main/java/org/apache/ranger/authorization/hadoop/RangerHdfsAuthorizer.java
>  d28685a 
>   
> hdfs-agent/src/test/java/org/apache/ranger/services/hdfs/HDFSRangerTest.java 
> fe6a705 
>   hdfs-agent/src/test/resources/hdfs-policies.json 6055dd5 
>   hdfs-agent/src/test/resources/log4j.properties PRE-CREATION 
>   hdfs-agent/src/test/resources/ranger-hdfs-security.xml 9b1176e 
>   
> hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAuthorizer.java
>  56ef187 
>   
> knox-agent/src/main/java/org/apache/ranger/authorization/knox/RangerPDPKnoxFilter.java
>  1d58b21 
>   
> plugin-atlas/src/main/java/org/apache/ranger/authorization/atlas/authorizer/RangerAtlasAuthorizer.java
>  9712f95 
>   
> plugin-kafka/src/main/java/org/apache/ranger/authorization/kafka/authorizer/RangerKafkaAuthorizer.java
>  b6e052f 
>   
> plugin-kms/src/main/java/org/apache/ranger/authorization/kms/authorizer/RangerKmsAuthorizer.java
>  4cda8fa 
>   
> plugin-solr/src/main/java/org/apache/ranger/authorization/solr/authorizer/RangerSolrAuthorizer.java
>  0cc9de9 
>   
> plugin-yarn/src/main/java/org/apache/ranger/authorization/yarn/authorizer/RangerYarnAuthorizer.java
>  c589060 
>   ranger-tools/conf/log4j.properties 4ead802 
>   
> ranger-tools/src/main/java/org/apache/ranger/policyengine/RangerPolicyenginePerfTester.java
>  056c548 
>   
>

Review Request 60949: Optimize Ranger code for authorization of HDFS 'getContentSummary' and 'delete' commands

2017-07-18 Thread Abhay Kulkarni 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/60949/
---

Review request for ranger and Madhan Neethiraj.


Summary (updated)
-

Optimize Ranger code for authorization of HDFS 'getContentSummary' and 'delete' 
commands


Bugs: RANGER-1695
https://issues.apache.org/jira/browse/RANGER-1695


Repository: ranger


Description (updated)
---

HDFS commands such as getContentSummary and delete require Ranger to authorize 
access to entire directory hierarchy rooted at the directory passed in as 
argument. Ranger could optimize this code by pruning the directory tree 
whenever a sub-directory, and directory hierarchy rooted in it, is authorized 
for requested access. This could potentially save many spurious authorization 
API calls with Ranger policy engine.
Also, it will be useful to gather and log performance metrics for the Ranger 
authorization API call for every Ranger plugin.


Diffs (updated)
-

  
agents-common/src/main/java/org/apache/ranger/authorization/hadoop/constants/RangerHadoopConstants.java
 83f720a 
  
agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyEngineImpl.java
 101013b 
  
agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyEngineOptions.java
 63162f6 
  
agents-common/src/main/java/org/apache/ranger/plugin/service/RangerBasePlugin.java
 06b8f4d 
  
agents-common/src/main/java/org/apache/ranger/plugin/util/RangerPerfCollectorTracer.java
 d899c6f 
  
hbase-agent/src/main/java/org/apache/ranger/authorization/hbase/RangerAuthorizationCoprocessor.java
 cf2ffcf 
  
hdfs-agent/src/main/java/org/apache/ranger/authorization/hadoop/RangerHdfsAuthorizer.java
 d28685a 
  hdfs-agent/src/test/java/org/apache/ranger/services/hdfs/HDFSRangerTest.java 
fe6a705 
  hdfs-agent/src/test/resources/hdfs-policies.json 6055dd5 
  hdfs-agent/src/test/resources/log4j.properties PRE-CREATION 
  hdfs-agent/src/test/resources/ranger-hdfs-security.xml 9b1176e 
  
hive-agent/src/main/java/org/apache/ranger/authorization/hive/authorizer/RangerHiveAuthorizer.java
 56ef187 
  
knox-agent/src/main/java/org/apache/ranger/authorization/knox/RangerPDPKnoxFilter.java
 1d58b21 
  
plugin-atlas/src/main/java/org/apache/ranger/authorization/atlas/authorizer/RangerAtlasAuthorizer.java
 9712f95 
  
plugin-kafka/src/main/java/org/apache/ranger/authorization/kafka/authorizer/RangerKafkaAuthorizer.java
 b6e052f 
  
plugin-kms/src/main/java/org/apache/ranger/authorization/kms/authorizer/RangerKmsAuthorizer.java
 4cda8fa 
  
plugin-solr/src/main/java/org/apache/ranger/authorization/solr/authorizer/RangerSolrAuthorizer.java
 0cc9de9 
  
plugin-yarn/src/main/java/org/apache/ranger/authorization/yarn/authorizer/RangerYarnAuthorizer.java
 c589060 
  ranger-tools/conf/log4j.properties 4ead802 
  
ranger-tools/src/main/java/org/apache/ranger/policyengine/RangerPolicyenginePerfTester.java
 056c548 
  
storm-agent/src/main/java/org/apache/ranger/authorization/storm/authorizer/RangerStormAuthorizer.java
 23c2b5f 


Diff: https://reviews.apache.org/r/60949/diff/1/


Testing (updated)
---

Tested with local VM


Thanks,

Abhay Kulkarni