> On Oct. 18, 2022, 4:57 p.m., Abhay Kulkarni wrote: > > agents-common/src/main/java/org/apache/ranger/plugin/contextenricher/RangerAdminTagRetriever.java > > Lines 37 (patched) > > <https://reviews.apache.org/r/74175/diff/1/?file=2271011#file2271011line37> > > > > Is there a rough estimate of how much de-duping of tags save on the > > plugin (and admin) side?
tags dedup saved around 30% of memory with following test data (2gb vs 1.4gb): - 1m service-resources - 3m tags - 250 tag-defs Above savings will be seen on both plugin and admin side. > On Oct. 18, 2022, 4:57 p.m., Abhay Kulkarni wrote: > > agents-common/src/main/java/org/apache/ranger/plugin/util/RangerCommonConstants.java > > Line 53 (original), 53 (patched) > > <https://reviews.apache.org/r/74175/diff/1/?file=2271017#file2271017line53> > > > > How is this constant used? There is another constant in > > RangerAdminTagRetriever which seems to control de-duping as well. Though the constant name (RANGER_ADMIN_SUPPORTS_TAGS_DEDUP_DEFAULT) suggests this is used only in Ranger admin side, current implementation references this in both admin and plugin sides: - plugin: RangerServiceTagsDeltaUtil.applyDelta() - admin: TagDBStore.getServiceTags() Ideally, both sides should use different configurations. I suggest to track this update in a separate patch. > On Oct. 18, 2022, 4:57 p.m., Abhay Kulkarni wrote: > > agents-common/src/main/java/org/apache/ranger/plugin/util/RangerServiceTagsDeltaUtil.java > > Line 64 (original), 65 (patched) > > <https://reviews.apache.org/r/74175/diff/1/?file=2271018#file2271018line65> > > > > Please evaluate if this shallow copy can be done only in the Ranger > > admin process. In the plugin code, 1. making a copy may not be necessary, > > and 2. if 1 is correct, then copying will incur excessive overhead. (It is > > assumed that the time taken to process service-tags in the plugin is much > > less than the tag download interval). I looked at adding updateInplace flag to applyDelta(). But this method is used by RangerTagEnricher.setServiceTags(), which is used in both plugin and admin side. It might require considerable refactor to enable copy only at Ranger admin side. I suggest to track this via a separate patch. > On Oct. 18, 2022, 4:57 p.m., Abhay Kulkarni wrote: > > agents-common/src/main/java/org/apache/ranger/plugin/util/RangerServiceTagsDeltaUtil.java > > Line 253 (original), 269 (patched) > > <https://reviews.apache.org/r/74175/diff/1/?file=2271018#file2271018line269> > > > > It may be useful to keep GUIDs of tag-definitions, tags and > > service-resources around. If the exported (or downloaded) serviceTags are > > to be used to import them into another (parallel - for DR use-case) system, > > then having same GUIDs of same objects is desirable. Guid was set to null even before this patch. Also, enabling dedup would result in the downloaded tags be not usable for import in another env. Given this, a separate API to get importable tags will be necessary. - Madhan ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/74175/#review224809 ----------------------------------------------------------- On Oct. 17, 2022, 11:42 p.m., Madhan Neethiraj wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/74175/ > ----------------------------------------------------------- > > (Updated Oct. 17, 2022, 11:42 p.m.) > > > Review request for ranger, Ankita Sinha, Kishor Gollapalliwar, Abhay > Kulkarni, Mehul Parikh, Pradeep Agrawal, Ramesh Mani, Sailaja Polavarapu, > Subhrat Chaudhary, and Velmurugan Periasamy. > > > Bugs: RANGER-3951 > https://issues.apache.org/jira/browse/RANGER-3951 > > > Repository: ranger > > > Description > ------- > > - skipped initialization of collection members (Map, List) with new > collection instance; these members will be set to null by default > - skipped serializing empty collections, this helps to reduce size of JSON > downloaded to plugins > - fixed potential concurrent modification to ServiceTags instance at Ranger > admin > - updated tag-retriever to deDup tags, so that the plugins will have reduced > memory usage irrespective of the optimization at the server side > - updated server to dedup tags by default i.e. default value for > configuration ranger.admin.supports.tags.dedup is changed from false to true > > > Diffs > ----- > > > agents-common/src/main/java/org/apache/ranger/plugin/contextenricher/RangerAdminTagRetriever.java > 1df5587cb > > agents-common/src/main/java/org/apache/ranger/plugin/contextenricher/RangerTagForEval.java > 1c634b393 > > agents-common/src/main/java/org/apache/ranger/plugin/model/RangerServiceResource.java > 47d006048 > agents-common/src/main/java/org/apache/ranger/plugin/model/RangerTag.java > 9e620c32c > > agents-common/src/main/java/org/apache/ranger/plugin/model/RangerTagDef.java > f1a83cfaf > > agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyRepository.java > ff83c7434 > > agents-common/src/main/java/org/apache/ranger/plugin/util/RangerCommonConstants.java > 86f881f80 > > agents-common/src/main/java/org/apache/ranger/plugin/util/RangerServiceTagsDeltaUtil.java > 0365fc992 > agents-common/src/main/java/org/apache/ranger/plugin/util/ServiceTags.java > b4f250038 > > security-admin/src/main/java/org/apache/ranger/biz/RangerTagDBRetriever.java > c6b8ab937 > security-admin/src/main/java/org/apache/ranger/biz/TagDBStore.java > 4d3a56911 > > security-admin/src/main/java/org/apache/ranger/common/RangerServiceTagsCache.java > 576546f29 > > security-admin/src/main/java/org/apache/ranger/patch/PatchForUpdatingTagsJson_J10020.java > 301d5c30f > > security-admin/src/main/java/org/apache/ranger/rest/ServiceTagsProcessor.java > 1d6c48a4e > > > Diff: https://reviews.apache.org/r/74175/diff/1/ > > > Testing > ------- > > - verified that all tests pass successfully > - in an environment with 1m resources and ~3m tags, observed following > improvements: > -- 45% reduction in JSON file downloaded to plugins > -- 33% reduction in memory required for tags in plugins and Ranger admin > > > Thanks, > > Madhan Neethiraj > >
