> On Aug. 4, 2019, 7:43 p.m., Abhay Kulkarni wrote: > > agents-common/src/main/java/org/apache/ranger/plugin/store/ServiceStore.java > > Line 68 (original), 68 (patched) > > <https://reviews.apache.org/r/71236/diff/1/?file=2159525#file2159525line68> > > > > Please consider an alternative design where the value of boolean > > 'flush' - which controlls if JPA cache needs to be flushed - may be > > initiatized in the thread context. In this design, code changes are > > minimized. Also, it will make comparing performance with and without the > > fix easier, if such initialization also may be made dependent on value of > > some configuration parameter.
I have changed the createpolicy signature in the ServiceREST class where the API can take addition parameter for flush. default value will be true which is same as current behavior. do you mean to use ThreadContext or ThreadLocal implementation here ? > On Aug. 4, 2019, 7:43 p.m., Abhay Kulkarni wrote: > > security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java > > Lines 2748 (patched) > > <https://reviews.apache.org/r/71236/diff/1/?file=2159536#file2159536line2749> > > > > Is calling flush after 10 policy deletes/updates/creates somewhat > > arbitrary? Can this be controlled through some config parameter? yes 10 is an arbitrary number and we can add this as a config. - Pradeep ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/71236/#review217066 ----------------------------------------------------------- On Aug. 4, 2019, 8:14 a.m., Pradeep Agrawal wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/71236/ > ----------------------------------------------------------- > > (Updated Aug. 4, 2019, 8:14 a.m.) > > > Review request for ranger, Ankita Sinha, bhavik patel, Gautam Borad, Abhay > Kulkarni, Madhan Neethiraj, Mehul Parikh, Nikhil P, Nitin Galave, Ramesh > Mani, Sailaja Polavarapu, and Velmurugan Periasamy. > > > Bugs: RANGER-2499 > https://issues.apache.org/jira/browse/RANGER-2499 > > > Repository: ranger > > > Description > ------- > > **Problem Statement:** Ranger takes an extremely long time to import bulk > policies. > **Current performance(Without patch):** > Time taken to import 500 policies : 27+ Minutes > Time taken to import 1000 policies : 81+ Minutes > > **Proposed Solution:** > in the bulk policy creation/deletion operation proposed patch shall call > EntityManager.flush() and EntityManager.clear() after processing a batch of > 10 records. > > **References: ** > https://en.wikibooks.org/wiki/Java_Persistence/Persisting > https://www.developerfusion.com/article/84945/flush-and-clear-or-mapping-antipatterns/ > > > Diffs > ----- > > > agents-common/src/main/java/org/apache/ranger/plugin/store/ServiceStore.java > d4879766f > security-admin/src/main/java/org/apache/ranger/biz/PolicyRefUpdater.java > e242d9092 > > security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyRetriever.java > 4815f5bdb > security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java > ef22354de > security-admin/src/main/java/org/apache/ranger/biz/XUserMgr.java 037c591e8 > security-admin/src/main/java/org/apache/ranger/common/db/BaseDao.java > 4f339087d > > security-admin/src/main/java/org/apache/ranger/patch/PatchForNifiResourceUpdateExclude_J10011.java > 1c48eebcb > > security-admin/src/main/java/org/apache/ranger/patch/PatchMigration_J10002.java > 0af7a1da2 > security-admin/src/main/java/org/apache/ranger/rest/AssetREST.java > 66a57289f > security-admin/src/main/java/org/apache/ranger/rest/PublicAPIs.java > 7df957973 > security-admin/src/main/java/org/apache/ranger/rest/PublicAPIsv2.java > 2a4c53b48 > security-admin/src/main/java/org/apache/ranger/rest/ServiceREST.java > 348d072e6 > > security-admin/src/main/java/org/apache/ranger/service/RangerBaseModelService.java > de5d34307 > > security-admin/src/main/java/org/apache/ranger/service/RangerPolicyLabelsService.java > 1a1b56e4c > security-admin/src/test/java/org/apache/ranger/biz/TestServiceDBStore.java > b6f13f47e > security-admin/src/test/java/org/apache/ranger/rest/TestAssetREST.java > dce3b0b69 > security-admin/src/test/java/org/apache/ranger/rest/TestPublicAPIs.java > 2bf5ee6c9 > security-admin/src/test/java/org/apache/ranger/rest/TestPublicAPIsv2.java > f9ea26a31 > security-admin/src/test/java/org/apache/ranger/rest/TestServiceREST.java > a7e19bf03 > > > Diff: https://reviews.apache.org/r/71236/diff/1/ > > > Testing > ------- > > **Performance(With patch):** > > Time taken to import 500 policies : 84 Seconds > Time taken to import 1000 policies : 111 Seconds > > Time taken to delete 500 policies : 30 Seconds > > > Thanks, > > Pradeep Agrawal > >
