> On Sept. 29, 2022, 12:31 a.m., Abhay Kulkarni wrote: > > agents-common/src/main/java/org/apache/ranger/plugin/util/RangerServiceTagsDeltaUtil.java > > Line 107 (original), 101 (patched) > > <https://reviews.apache.org/r/74144/diff/1-2/?file=2270242#file2270242line109> > > > > This conversion will be done every time applyDeltas() is called. This > > may be expensive operation. Is it possible to cache this map in ServiceTags > > but ignore it when serializing ServiceTags object?
Holding this map in ServiceTags will result in memory be not released until ServiceTag instance goes away. Currently this map is destroyed after applyDeltas() method returns. With a standalone program, found the following stats to create and populate maps: - 1m entries: memory=74mb, time=90ms - 2m entries: memory=141mb, time=130ms - 3m entries: memory=197mb, time=168ms I think it will help to release the memory after completion of applyDelats(). > On Sept. 29, 2022, 12:31 a.m., Abhay Kulkarni wrote: > > agents-common/src/main/java/org/apache/ranger/plugin/util/RangerServiceTagsDeltaUtil.java > > Line 142 (original), 130 (patched) > > <https://reviews.apache.org/r/74144/diff/1-2/?file=2270242#file2270242line151> > > > > If any resources are deleted or updated, this code iterates over all > > service-resources. Will this be expensive, as before? This is a single iteration on serviceResources, not a nested one; hence will not be as expesive as before. - Madhan ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74144/#review224719 ----------------------------------------------------------- On Sept. 29, 2022, 6:46 a.m., Madhan Neethiraj wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74144/ > ----------------------------------------------------------- > > (Updated Sept. 29, 2022, 6:46 a.m.) > > > Review request for ranger, Ankita Sinha, Kishor Gollapalliwar, Abhay > Kulkarni, Mehul Parikh, Pradeep Agrawal, Ramesh Mani, Sailaja Polavarapu, > Subhrat Chaudhary, and Velmurugan Periasamy. > > > Bugs: RANGER-3934 > https://issues.apache.org/jira/browse/RANGER-3934 > > > Repository: ranger > > > Description > ------- > > - updated several retrieval of XXService JPA object by replacing with > retrival of serviceId > - replaced several instances of Map<Long, Long> with Set<Long>, as these > instances had same value for key and value > - avoided expensive calls to RangerTagResourceMapService.getByTagId(tagId) > and RangerTagResourceMapService.getTagIdsForResourceId(serviceResourceId) > - updated several methods in XXTagResourceMapDao to avoid loading of > XXTagResourceMap JPA object; instead created XXTagResourceMap object from > individial fields retrieved from query > > > Diffs > ----- > > > agents-common/src/main/java/org/apache/ranger/plugin/util/RangerServiceTagsDeltaUtil.java > 8d9241c1c > security-admin/src/main/java/org/apache/ranger/biz/TagDBStore.java > d8154b7de > > security-admin/src/main/java/org/apache/ranger/db/XXRMSServiceResourceDao.java > afa754ba2 > security-admin/src/main/java/org/apache/ranger/db/XXServiceDao.java > 3cc3d9cef > security-admin/src/main/java/org/apache/ranger/db/XXTagResourceMapDao.java > 3f8b5b718 > > security-admin/src/main/java/org/apache/ranger/patch/PatchForAtlasServiceDefUpdate_J10013.java > b0f71e138 > security-admin/src/main/java/org/apache/ranger/rest/TagREST.java c7cf3bfb8 > security-admin/src/main/resources/META-INF/jpa_named_queries.xml e4a2354b0 > > > Diff: https://reviews.apache.org/r/74144/diff/3/ > > > Testing > ------- > > - with Ranger database having ~1m service-resources, ~2m tags and delta of > 20k resource & 40k tags: > -- before these optmizations, tag-cache update from delta didn't complete > event after a long time (1h35m) > -- with these optmizations, tag-cache update from the same delta completed > within 10 minutes > - there is scope for further optimization > RangerServiceTagsDeltaUtil.applyDelta() - as this took almost 97% time in > updating the cache (588 seconds); compare this to > TagDBStore.getServiceTagsDelta() which completed in 13 seeconds > > > Thanks, > > Madhan Neethiraj > >
