----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71236/#review217066 -----------------------------------------------------------
agents-common/src/main/java/org/apache/ranger/plugin/store/ServiceStore.java Line 68 (original), 68 (patched) <https://reviews.apache.org/r/71236/#comment304316> Please consider an alternative design where the value of boolean 'flush' - which controlls if JPA cache needs to be flushed - may be initiatized in the thread context. In this design, code changes are minimized. Also, it will make comparing performance with and without the fix easier, if such initialization also may be made dependent on value of some configuration parameter. security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java Lines 2748 (patched) <https://reviews.apache.org/r/71236/#comment304315> Is calling flush after 10 policy deletes/updates/creates somewhat arbitrary? Can this be controlled through some config parameter? - Abhay Kulkarni On Aug. 4, 2019, 8:14 a.m., Pradeep Agrawal wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71236/ > ----------------------------------------------------------- > > (Updated Aug. 4, 2019, 8:14 a.m.) > > > Review request for ranger, Ankita Sinha, bhavik patel, Gautam Borad, Abhay > Kulkarni, Madhan Neethiraj, Mehul Parikh, Nikhil P, Nitin Galave, Ramesh > Mani, Sailaja Polavarapu, and Velmurugan Periasamy. > > > Bugs: RANGER-2499 > https://issues.apache.org/jira/browse/RANGER-2499 > > > Repository: ranger > > > Description > ------- > > **Problem Statement:** Ranger takes an extremely long time to import bulk > policies. > **Current performance(Without patch):** > Time taken to import 500 policies : 27+ Minutes > Time taken to import 1000 policies : 81+ Minutes > > **Proposed Solution:** > in the bulk policy creation/deletion operation proposed patch shall call > EntityManager.flush() and EntityManager.clear() after processing a batch of > 10 records. > > **References: ** > https://en.wikibooks.org/wiki/Java_Persistence/Persisting > https://www.developerfusion.com/article/84945/flush-and-clear-or-mapping-antipatterns/ > > > Diffs > ----- > > > agents-common/src/main/java/org/apache/ranger/plugin/store/ServiceStore.java > d4879766f > security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java > e242d9092 > > security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyRetriever.java > 4815f5bdb > security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java > ef22354de > security-admin/src/main/java/org/apache/ranger/biz/XUserMgr.java 037c591e8 > security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java > 4f339087d > > security-admin/src/main/java/org/apache/ranger/patch/PatchForNifiResourceUpdateExclude_J10011.java > 1c48eebcb > > security-admin/src/main/java/org/apache/ranger/patch/PatchMigration_J10002.java > 0af7a1da2 > security-admin/src/main/java/org/apache/ranger/rest/AssetREST.java > 66a57289f > security-admin/src/main/java/org/apache/ranger/rest/PublicAPIs.java > 7df957973 > security-admin/src/main/java/org/apache/ranger/rest/PublicAPIsv2.java > 2a4c53b48 > security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java > 348d072e6 > > security-admin/src/main/java/org/apache/ranger/service/RangerBaseModelService.java > de5d34307 > > security-admin/src/main/java/org/apache/ranger/service/RangerPolicyLabelsService.java > 1a1b56e4c > security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java > b6f13f47e > security-admin/src/test/java/org/apache/ranger/rest/TestAssetREST.java > dce3b0b69 > security-admin/src/test/java/org/apache/ranger/rest/TestPublicAPIs.java > 2bf5ee6c9 > security-admin/src/test/java/org/apache/ranger/rest/TestPublicAPIsv2.java > f9ea26a31 > security-admin/src/test/java/org/apache/ranger/rest/TestServiceREST.java > a7e19bf03 > > > Diff: https://reviews.apache.org/r/71236/diff/1/ > > > Testing > ------- > > **Performance(With patch):** > > Time taken to import 500 policies : 84 Seconds > Time taken to import 1000 policies : 111 Seconds > > Time taken to delete 500 policies : 30 Seconds > > > Thanks, > > Pradeep Agrawal > >
