----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73728/#review223773 -----------------------------------------------------------
agents-common/src/main/java/org/apache/ranger/plugin/util/RangerServiceTagsDeltaUtil.java Lines 68 (patched) <https://reviews.apache.org/r/73728/#comment312856> Given key and value are assigned the same object (#72), consider using Set<RangerTag>. agents-common/src/main/java/org/apache/ranger/plugin/util/ServiceTags.java Lines 227 (patched) <https://reviews.apache.org/r/73728/#comment312857> This dedup doesn't eliminate duplicate RangerTag entries from the map. Instead, it replaces duplicate RangerTag instances with the first instance. Removing these entries from 'tags', followed by updating the mappings held in 'resourceToTagIds' for removed entries, can improve the performance further - by eliminating serialization/deserialization of duplicate tags, and by avoiding creation of multiple evaluators in policy engine. Please review. Map<RangerTag, long> cachedTags = new HashMap<>(); Map<Long, Long> replacedIds = new HashMap<>(); Iterator<Map.Entry<Long, RangerTag>> iter = tags.entrySet().iterator(); while (iter.hasNext()) { Map.Entry<Long, RangerTag> entry = iter.next(); Long tagId = entry.getKey(); RangerTag tag = entry.getValue(); Long cachedTagId = cachedTags.get(tag); if (cachedTagId == null) { cachedTags.put(tag, tagId); } else { replacedIds.put(tagId, cachedTagId); iter.remove(); } } for (Map.Entry<Long, List<Long>> resourceEntry : resourceToTagIds.entrySet()) { ListIterator<Long> iter = resourceEntry.getValue().listIterator(); while (iter.hasNext()) { Long tagId = iter.next(); Long replacerTagId = replacedIds.get(tagId); if (replacerTagId != null) { iter.set(replacerTagId)'' } } } - Madhan Neethiraj On Nov. 27, 2021, 8:15 p.m., Abhay Kulkarni wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73728/ > ----------------------------------------------------------- > > (Updated Nov. 27, 2021, 8:15 p.m.) > > > Review request for ranger, Madhan Neethiraj and Velmurugan Periasamy. > > > Bugs: RANGER-3519 > https://issues.apache.org/jira/browse/RANGER-3519 > > > Repository: ranger > > > Description > ------- > > When the number of policies (and/or tagged resources) is large, the data > structures used by Ranger as indexes for policies (and/or tagged resources) > may need a very large heap memory because they are optimized for fast lookup. > It is desirable to be able to configure Ranger to have these structures > optimized for space in order to keep the heap requirements within acceptable > limit at the cost of somewhat slower lookup. > > In addition to changed committed by > (https://github.com/apache/ranger/commit/856571c4348e31725498c0922338339c76ebba02), > following boolean valued configuration parameter is added for the Ranger > admin server to optionally de-duplicate tags in ServiceTags object. > > ranger.admin.supports.tags.dedup (default: false) > > > Diffs > ----- > > > agents-common/src/main/java/org/apache/ranger/plugin/contextenricher/RangerFileBasedTagRetriever.java > b858879ba > > agents-common/src/main/java/org/apache/ranger/plugin/service/RangerDefaultRequestProcessor.java > facf05d43 > > agents-common/src/main/java/org/apache/ranger/plugin/util/RangerCommonConstants.java > 63bed50ce > > agents-common/src/main/java/org/apache/ranger/plugin/util/RangerServiceTagsDeltaUtil.java > 6b70b2259 > agents-common/src/main/java/org/apache/ranger/plugin/util/ServiceTags.java > 9e8e0cf6c > distro/src/main/assembly/ranger-tools.xml 5fdf3a5ef > ranger-tools/scripts/gen_service_tags.sh dd3ee8813 > > ranger-tools/src/main/java/org/apache/ranger/policyengine/RangerPolicyenginePerfTester.java > f667628dd > ranger-tools/src/test/resources/testdata/ranger-config.xml 218db0759 > ranger-tools/testdata/ranger-config.xml 08a83b9a5 > security-admin/src/main/java/org/apache/ranger/biz/TagDBStore.java > 1bdfce057 > > > Diff: https://reviews.apache.org/r/73728/diff/1/ > > > Testing > ------- > > - Executed all unit tests successfully > - Performance tested ranger-tools with a million tags > > > Thanks, > > Abhay Kulkarni > >
