-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/71237/
-----------------------------------------------------------
(Updated Aug. 6, 2019, 3:29 a.m.)
Review request for ranger, Ankita Sinha, bhavik patel, Gautam Borad, Abhay
Kulkarni, Madhan Neethiraj, Mehul Parikh, Nikhil P, Nitin Galave, Ramesh Mani,
Sailaja Polavarapu, and Velmurugan Periasamy.
Changes
-------
Addressed review comments
Bugs: RANGER-2499
https://issues.apache.org/jira/browse/RANGER-2499
Repository: ranger
Description
-------
**This RR contains alternate solution of RANGER-2499. First solution RR is
https://reviews.apache.org/r/71236/**
**Problem Statement:** Ranger takes an extremely long time to import bulk
policies.
**Current performance(Without patch):**
Time taken to import 500 policies : 27+ Minutes
Time taken to import 1000 policies : 81+ Minutes
**Proposed Solution:**
in the bulk policy creation/deletion operation proposed patch shall call
EntityManager.flush() and EntityManager.clear() after processing a batch of 10
records.
** Note: **
Most of the changes in this patch comes from revert work of RANGER-2493 as I am
trying to generalize the solution for bulk create and update policies also.
Important changes are:
1) bulkModeOnlyFlushAndClear() method in the RangerBizUtil.java class.
2) call to bulkModeOnlyFlushAndClear() from ServiceREST and ServiceDBStore
wherever bulk create or update is made.
3) batch create/delete optimization and batch is configurable through
ranger-admin-default-site.xml file
4) few null checks in the existing code.
**References: **
https://en.wikibooks.org/wiki/Java_Persistence/Persisting
https://www.developerfusion.com/article/84945/flush-and-clear-or-mapping-antipatterns/
Diffs (updated)
-----
agents-common/src/main/java/org/apache/ranger/plugin/store/ServiceStore.java
d4879766f
security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java
e242d9092
security-admin/src/main/java/org/apache/ranger/biz/RangerBizUtil.java
d49ea9886
security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyRetriever.java
4815f5bdb
security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java
ef22354de
security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java
4f339087d
security-admin/src/main/java/org/apache/ranger/patch/PatchForUpdatingPolicyJson_J10019.java
0788fb5bc
security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java
348d072e6
security-admin/src/main/java/org/apache/ranger/service/RangerBaseModelService.java
de5d34307
security-admin/src/main/java/org/apache/ranger/service/RangerDataHistService.java
019816e49
security-admin/src/main/resources/conf.dist/ranger-admin-default-site.xml
eb7947a47
security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java
b6f13f47e
security-admin/src/test/java/org/apache/ranger/service/TestRangerDataHistService.java
655e17f40
Diff: https://reviews.apache.org/r/71237/diff/2/
Changes: https://reviews.apache.org/r/71237/diff/1-2/
Testing
-------
Time taken to import 500 policies(500 create) : 45 Seconds
Time taken to import 500 policies(500 delete+500 create) : 60 Seconds
Time taken to import 1000 policies(1000 create) : 76 Seconds
Time taken to import 1000 policies(1000 delete+1000 create) : 134 Seconds
Time taken to delete 500 policies : 30 Seconds
Thanks,
Pradeep Agrawal
