date:20190704

Re: Review Request 71007: RANGER-2493: Ranger takes long time to override a service with many policies

2019-07-04 Thread Abhay Kulkarni



> On July 3, 2019, 5:49 p.m., Abhay Kulkarni wrote:
> > security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java
> > Lines 3010 (patched)
> > 
> >
> > Consider moving lines 3010-3013 to after line 3008.
> 
> Pradeep Agrawal wrote:
> @Abhay : Please read the comments added before the code.
> 
> public void deleteZonePolicies(Collection serviceNames, Long 
> zoneId) throws Exception {
>   if (CollectionUtils.isNotEmpty(serviceNames)) {
>   XXPolicyDao policyDao = daoMgr.getXXPolicy();
>   for (String serviceName : serviceNames) {
>   RangerService service = 
> getServiceByName(serviceName);
> //below statement will pull only policy IDs rather 
> XXPolicy object
>   List policyIds = 
> policyDao.findPolicyIdsByServiceNameAndZoneId(serviceName, zoneId);
>   List rangerPolicyList=new 
> ArrayList();
>   if (CollectionUtils.isNotEmpty(policyIds)) {
>   for (Long id : policyIds) {
> //from the policy id we need to get RangerPolicy 
> policy object which shall be added first in the rangerPolicyList
>   
> rangerPolicyList.add(getPolicy(id));
>   }
>   }
> // run through rangerPolicyList and delete ranger 
> policies.
>   for (RangerPolicy rangerPolicy : 
> rangerPolicyList) {
>   deletePolicy(rangerPolicy, service);
>   }
> //rangerPolicyList object is required for Trx logs and 
> Policy history.
>   
> createTrxLogsAndHistoryAfterDelete(rangerPolicyList, service);
>   }
>   }
>   }

I was suggesting this instead. Policies need to be deleted only if policyIds 
collection is non empty.

public void deleteZonePolicies(Collection serviceNames, Long zoneId) 
throws Exception {
if (CollectionUtils.isNotEmpty(serviceNames)) {
XXPolicyDao policyDao = daoMgr.getXXPolicy();
for (String serviceName : serviceNames) {
RangerService service = getServiceByName(serviceName);
//below statement will pull only policy IDs rather XXPolicy object
List policyIds = 
policyDao.findPolicyIdsByServiceNameAndZoneId(serviceName, zoneId);
if (CollectionUtils.isNotEmpty(policyIds)) {
List rangerPolicyList = new ArrayList<>();
for (Long id : policyIds) {
//from the policy id we need to get RangerPolicy policy 
object which shall be added first in the rangerPolicyList
rangerPolicyList.add(getPolicy(id));
}

// run through rangerPolicyList and delete ranger policies.
for (RangerPolicy rangerPolicy : rangerPolicyList) {
deletePolicy(rangerPolicy, service);
}
//rangerPolicyList object is required for Trx logs and Policy 
history.
createTrxLogsAndHistoryAfterDelete(rangerPolicyList, service);
}
}
}
}


- Abhay


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71007/#review216348
---


On July 4, 2019, 7:17 a.m., Pradeep Agrawal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71007/
> ---
> 
> (Updated July 4, 2019, 7:17 a.m.)
> 
> 
> Review request for ranger, Ankita Sinha, bhavik patel, Gautam Borad, Abhay 
> Kulkarni, Madhan Neethiraj, Mehul Parikh, Nikhil P, Nitin Galave, Ramesh 
> Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-2493
> https://issues.apache.org/jira/browse/RANGER-2493
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> ** Problem Statement ** Ranger takes an extremely long time to override a 
> service with many policies, will crash ranger if admin JVM heap size is the 1 
> GB default
> 
> 
> Diffs
> -
> 
>   security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java 
> bf50df3a8 
>   security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java 
> 51c0de56c 
>   security-admin/src/main/java/org/apache/ranger/db/XXPolicyDao.java 
> baf6b6ea4 
>   security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 
> 171d73bfa 
>   security-admin/src/main/resources/META-INF/jpa_named_queries.xml

Re: Review Request 71007: RANGER-2493: Ranger takes long time to override a service with many policies

2019-07-04 Thread Abhay Kulkarni


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71007/#review216395
---




security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java
Line 253 (original), 252 (patched)


It appears that this function is not used at all. Please consider removing 
it.


- Abhay Kulkarni


On July 4, 2019, 7:17 a.m., Pradeep Agrawal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/71007/
> ---
> 
> (Updated July 4, 2019, 7:17 a.m.)
> 
> 
> Review request for ranger, Ankita Sinha, bhavik patel, Gautam Borad, Abhay 
> Kulkarni, Madhan Neethiraj, Mehul Parikh, Nikhil P, Nitin Galave, Ramesh 
> Mani, Sailaja Polavarapu, and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-2493
> https://issues.apache.org/jira/browse/RANGER-2493
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> ** Problem Statement ** Ranger takes an extremely long time to override a 
> service with many policies, will crash ranger if admin JVM heap size is the 1 
> GB default
> 
> 
> Diffs
> -
> 
>   security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java 
> bf50df3a8 
>   security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java 
> 51c0de56c 
>   security-admin/src/main/java/org/apache/ranger/db/XXPolicyDao.java 
> baf6b6ea4 
>   security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 
> 171d73bfa 
>   security-admin/src/main/resources/META-INF/jpa_named_queries.xml eb0384b14 
>   security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java 
> dc845cf80 
>   security-admin/src/test/java/org/apache/ranger/rest/TestServiceREST.java 
> 9b9aa8377 
> 
> 
> Diff: https://reviews.apache.org/r/71007/diff/2/
> 
> 
> Testing
> ---
> 
> Tested 500 policy import json and unable to reproduce JVM issue.
> 
> 
> Thanks,
> 
> Pradeep Agrawal
> 
>

[jira] [Commented] (RANGER-2489) Missing dependencies in assembly for Presto plugin

2019-07-04 Thread Bolke de Bruin (JIRA)



[ 
https://issues.apache.org/jira/browse/RANGER-2489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16878539#comment-16878539
 ] 

Bolke de Bruin commented on RANGER-2489:


cc [~rmani] , [~abhayk]

> Missing dependencies in assembly for Presto plugin
> --
>
> Key: RANGER-2489
> URL: https://issues.apache.org/jira/browse/RANGER-2489
> Project: Ranger
>  Issue Type: Improvement
>  Components: plugins
>Reporter: Bolke de Bruin
>Assignee: Bolke de Bruin
>Priority: Major
> Attachments: 0001-RANGER-2489-Minor-updates-to-presto-plugin.patch
>
>
> If invoking with hostnames rather than ips extra dependencies are required by 
> the plugin for Presto.
> commons-codec commons-codec
>  com.kstruct gethostname4j
>  com.sun jna



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (RANGER-2493) Ranger takes long time to override a service with many policies

2019-07-04 Thread Pradeep Agrawal (JIRA)



 [ 
https://issues.apache.org/jira/browse/RANGER-2493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pradeep Agrawal updated RANGER-2493:

Attachment: 0001-RANGER-2493-Ranger-takes-long-time-to-override-a-ser.patch

> Ranger takes long time to override a service with many policies
> ---
>
> Key: RANGER-2493
> URL: https://issues.apache.org/jira/browse/RANGER-2493
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>Affects Versions: 2.0.0
>Reporter: Pradeep Agrawal
>Assignee: Pradeep Agrawal
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: 
> 0001-RANGER-2493-Ranger-takes-long-time-to-override-a-ser.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (RANGER-2493) Ranger takes long time to override a service with many policies

2019-07-04 Thread Pradeep Agrawal (JIRA)



 [ 
https://issues.apache.org/jira/browse/RANGER-2493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pradeep Agrawal updated RANGER-2493:

Attachment: (was: 
0001-RANGER-2493-Ranger-takes-long-time-to-override-a-ser.patch)

> Ranger takes long time to override a service with many policies
> ---
>
> Key: RANGER-2493
> URL: https://issues.apache.org/jira/browse/RANGER-2493
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>Affects Versions: 2.0.0
>Reporter: Pradeep Agrawal
>Assignee: Pradeep Agrawal
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: 
> 0001-RANGER-2493-Ranger-takes-long-time-to-override-a-ser.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Review Request 71007: RANGER-2493: Ranger takes long time to override a service with many policies

2019-07-04 Thread Pradeep Agrawal

> On July 3, 2019, 5:49 p.m., Abhay Kulkarni wrote:
> > security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java
> > Lines 2065 (patched)
> > 
> >
> > Is it likely that policy or service could be null? If so, consider 
> > putting all logic here under a check :
> > 
> > if (service != null && policy != null).

I am not sure what should be behaviour if policy is null. We can either throw 
the exception or we can skip other statements of the method. Since this api is 
mainly for improving bulk delete we can avoid throwing the exception. Now, I 
have added the check you suggested only for policy and not for the service. 

I am not adding a check for service as I want to enable this method for a use 
case when a user want to delete policy in bulk but policies may belong to 
different services ; in that case if he can't pass a specific service then he 
can pass service object value as null and the actual service object will be 
picked from policy object.

Please review the updated patch.

> On July 3, 2019, 5:49 p.m., Abhay Kulkarni wrote:
> > security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java
> > Lines 3010 (patched)
> > 
> >
> > Consider moving lines 3010-3013 to after line 3008.

@Abhay : Please read the comments added before the code.

public void deleteZonePolicies(Collection serviceNames, Long 
zoneId) throws Exception {
if (CollectionUtils.isNotEmpty(serviceNames)) {
XXPolicyDao policyDao = daoMgr.getXXPolicy();
for (String serviceName : serviceNames) {
RangerService service = 
getServiceByName(serviceName);
//below statement will pull only policy IDs rather XXPolicy 
object
List policyIds = 
policyDao.findPolicyIdsByServiceNameAndZoneId(serviceName, zoneId);
List rangerPolicyList=new 
ArrayList();
if (CollectionUtils.isNotEmpty(policyIds)) {
for (Long id : policyIds) {
//from the policy id we need to get RangerPolicy policy 
object which shall be added first in the rangerPolicyList

rangerPolicyList.add(getPolicy(id));
}
}
// run through rangerPolicyList and delete ranger policies.
for (RangerPolicy rangerPolicy : 
rangerPolicyList) {
deletePolicy(rangerPolicy, service);
}
//rangerPolicyList object is required for Trx logs and Policy 
history.

createTrxLogsAndHistoryAfterDelete(rangerPolicyList, service);
}
}
}

> On July 3, 2019, 5:49 p.m., Abhay Kulkarni wrote:
> > security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java
> > Line 111 (original)
> > 
> >
> > Why is this change needed?

https://thoughts-on-java.org/common-hibernate-mistakes-cripple-performance/  
(Mistake 7)
https://thoughts-on-java.org/hibernate-tips-remove-entities-persistence-context/
https://en.wikibooks.org/wiki/Java_Persistence/Persisting#Example_flush

I came across few articles(mentioned above) and thought that if we are not 
going to reuse those deleted object then we don't need to call flush() for them 
as flush() call will immediately try to send delete entity statements to the 
DB. Also if all the bulk policy delete is going to be in single transactions 
then probably JPA will keep all these objects in a cache and update entities 
cascading in the memory untill the transaction completes.

In the first articles its mentioned that the flush() call forces 
JPA/Eclipselink to perform a dirty check on all managed entities and to create 
and execute SQL statements for all pending insert, update or delete operations. 
That slows down the application because it prevents JPA/Eclipselink from using 
several internal optimizations.

> On July 3, 2019, 5:49 p.m., Abhay Kulkarni wrote:
> > security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java
> > Line 262 (original)
> > 
> >
> > Why is this change needed?

https://thoughts-on-java.org/common-hibernate-mistakes-cripple-performance/  
(Mistake 7)
https://thoughts-on-java.org/hibernate-tips-remove-entities-persistence-context/
https://en.wikibooks.org/wiki/Java_Persistence/Persisting#Example_flush

I came across few articles(mentioned above) and thought that if we are not 
going to reuse

Re: Review Request 71007: RANGER-2493: Ranger takes long time to override a service with many policies

2019-07-04 Thread Pradeep Agrawal


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71007/
---

(Updated July 4, 2019, 7:17 a.m.)


Review request for ranger, Ankita Sinha, bhavik patel, Gautam Borad, Abhay 
Kulkarni, Madhan Neethiraj, Mehul Parikh, Nikhil P, Nitin Galave, Ramesh Mani, 
Sailaja Polavarapu, and Velmurugan Periasamy.


Changes
---

Addressed review comments and added required note.


Bugs: RANGER-2493
https://issues.apache.org/jira/browse/RANGER-2493


Repository: ranger


Description
---

** Problem Statement ** Ranger takes an extremely long time to override a 
service with many policies, will crash ranger if admin JVM heap size is the 1 
GB default


Diffs (updated)
-

  security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java 
bf50df3a8 
  security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java 
51c0de56c 
  security-admin/src/main/java/org/apache/ranger/db/XXPolicyDao.java baf6b6ea4 
  security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java 
171d73bfa 
  security-admin/src/main/resources/META-INF/jpa_named_queries.xml eb0384b14 
  security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java 
dc845cf80 
  security-admin/src/test/java/org/apache/ranger/rest/TestServiceREST.java 
9b9aa8377 


Diff: https://reviews.apache.org/r/71007/diff/2/

Changes: https://reviews.apache.org/r/71007/diff/1-2/


Testing
---

Tested 500 policy import json and unable to reproduce JVM issue.


Thanks,

Pradeep Agrawal

Re: Review Request 71001: RANGER-1644 : Allow individuals to use the stronger encryption algorithms supported by JDK 1.8 .

2019-07-04 Thread Mateen Mansoori


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71001/
---

(Updated July 4, 2019, 6:47 a.m.)


Review request for ranger, bhavik patel, Gautam Borad, Abhay Kulkarni, Madhan 
Neethiraj, Mehul Parikh, Nikhil P, Pradeep Agrawal, and Velmurugan Periasamy.


Bugs: RANGER-1644
https://issues.apache.org/jira/browse/RANGER-1644


Repository: ranger


Description
---

Allow individuals to use stronger encryption algorithms supported by JDK 1.8.
Applied mainly original patch from Endre Kovacs. 
Just created the new patch by fixing the test case which was failing on latest 
master branch.


Diffs
-

  agents-common/src/main/java/org/apache/ranger/plugin/util/PasswordUtils.java 
6480b17 
  
agents-common/src/test/java/org/apache/ranger/plugin/util/PasswordUtilsTest.java
 4e135aa 
  security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java 
bf50df3 
  
security-admin/src/main/java/org/apache/ranger/service/RangerServiceService.java
 2ddb5f3 
  security-admin/src/main/resources/conf.dist/ranger-admin-default-site.xml 
8ebeeb5 
  security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java 
dc845cf 


Diff: https://reviews.apache.org/r/71001/diff/1/


Testing (updated)
---

Service Created And Updated : Service users password which was 
created(encrypted) with old algo i.e PBEWithMD5AndDES was updated successfully 
with new set of java 1.8 encryption algo's i.e 
https://docs.oracle.com/javase/8/docs/technotes/guides/security/SunProviders.html#SunJCEProvider
 .

-Service CRUD on fresh installation.
-Tested with different algorithm on jdk1.8
-Old Service CRUD after upgrade from 2.6 with jdk1.7 to 3.1maint with jdk1.8

PasswordUtilsTest: added new unit test and updated previous ones.
Added service update test: on service update new service password will be 
encrypted with the new algorithm.


Thanks,

Mateen Mansoori

Re: Review Request 71007: RANGER-2493: Ranger takes long time to override a service with many policies

Re: Review Request 71007: RANGER-2493: Ranger takes long time to override a service with many policies

[jira] [Commented] (RANGER-2489) Missing dependencies in assembly for Presto plugin

[jira] [Updated] (RANGER-2493) Ranger takes long time to override a service with many policies

[jira] [Updated] (RANGER-2493) Ranger takes long time to override a service with many policies

Re: Review Request 71007: RANGER-2493: Ranger takes long time to override a service with many policies

Re: Review Request 71007: RANGER-2493: Ranger takes long time to override a service with many policies

Re: Review Request 71001: RANGER-1644 : Allow individuals to use the stronger encryption algorithms supported by JDK 1.8 .

8 matches

Site Navigation

Mail list logo

Footer information