[jira] [Commented] (RANGER-3373) Ranger Hbase plugin not compatible with Hbase 2.3.4
[ https://issues.apache.org/jira/browse/RANGER-3373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17407799#comment-17407799 ] Shivam Garg commented on RANGER-3373: - 2.1 branch repo already updated with 2.1.1-snapshot so I assume there will be one more release > Ranger Hbase plugin not compatible with Hbase 2.3.4 > --- > > Key: RANGER-3373 > URL: https://issues.apache.org/jira/browse/RANGER-3373 > Project: Ranger > Issue Type: Bug > Components: plugins >Affects Versions: 2.1.0, 3.0.0, 2.1.1 >Reporter: Shivam Garg >Priority: Major > Labels: patch > Fix For: 3.0.0, 2.1.1 > > Attachments: RANGER-3373-001-branch-2.1.patch, RANGER-3373-001.patch > > > Ranger is incompatible with Hbase 2.3.4 because AccessControlLists class has > been changed to PermissionStorage > https://issues.apache.org/jira/browse/HBASE-22084 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (RANGER-3379) Performance Analysis tool for Ranger Policy REST API
[ https://issues.apache.org/jira/browse/RANGER-3379?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Steven Ramirez updated RANGER-3379: --- Attachment: 0001-RANGER-3379-performance-tool-support-for-policy-APIs.patch > Performance Analysis tool for Ranger Policy REST API > > > Key: RANGER-3379 > URL: https://issues.apache.org/jira/browse/RANGER-3379 > Project: Ranger > Issue Type: Improvement > Components: Ranger >Reporter: Steven Ramirez >Assignee: Steven Ramirez >Priority: Major > Attachments: > 0001-RANGER-3379-performance-tool-support-for-policy-APIs.patch > > > The tool will focus on the creation, capture, and analysis of critical > performance metrics of Ranger REST API for policy management. The analysis of > these metrics will help predict trends of latencies suggesting performance > issues. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Review Request 73561: RANGER-3379: Performance Analysis tool for Ranger Policy REST API
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73561/ --- Review request for ranger. Bugs: RANGER-3379 https://issues.apache.org/jira/browse/RANGER-3379 Repository: ranger Description --- As part of ranger tools, I added a set of scripts that can test the perfomrance/stability of ranger policy API. This tool measures request latency and cpu utilization of ranger admin process. This tool provides options to run an exact amount of requests or run at a given rate over a set duration. Diffs - distro/src/main/assembly/ranger-tools.xml 3ae890c1e ranger-tools/scripts/load_generator.sh PRE-CREATION ranger-tools/scripts/log_parser.py PRE-CREATION ranger-tools/scripts/parse_logs.sh PRE-CREATION ranger-tools/scripts/policy_create_write_API.py PRE-CREATION ranger-tools/scripts/policy_delete_write_API.py PRE-CREATION ranger-tools/scripts/policy_get_write_API.py PRE-CREATION ranger-tools/scripts/policy_update_write_API.py PRE-CREATION Diff: https://reviews.apache.org/r/73561/diff/1/ Testing --- - verified the tool is part of the ranger tools packaging - tested for an hour at 24 requests per second - ran over an entire weekend at 12 requests per second - tested for a burst of 400 api requests Thanks, Steven Ramirez
[jira] [Commented] (RANGER-3378) StringBuilder.append spend too much time, should be replace to improve plugin performance
[ https://issues.apache.org/jira/browse/RANGER-3378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17407447#comment-17407447 ] caozhiqiang commented on RANGER-3378: - [~madhan] kafka has multi resources include topic, cluster, transactionalid and delegationtoken. So it should not do the similar improvements. But yarn and ES have single resource as queue and index, I will do this improvements to these plugins. > StringBuilder.append spend too much time, should be replace to improve plugin > performance > - > > Key: RANGER-3378 > URL: https://issues.apache.org/jira/browse/RANGER-3378 > Project: Ranger > Issue Type: Improvement > Components: plugins >Affects Versions: 2.0.1 >Reporter: caozhiqiang >Assignee: caozhiqiang >Priority: Major > Fix For: 3.0.0, 2.2.0 > > Attachments: RANGER-3378-branch-2.0.1.001.patch, > RANGER-3378-branch-2.0.1.003.patch, RANGER-3378-branch-2.0.1.004.patch, > getAsString.png, getAsString.png > > > When enable ranger hdfs plugin, the performance is degraded. Use java > analysis tools found the > org/apache/ranger/plugin/policyengine/RangerAccessResourceImpl.getAsString > function take too much time. We should replace it to improve performance. > !getAsString.png|width=939,height=371! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (RANGER-3390) improve plugin performance for yarn and es by replacing getAsString
[ https://issues.apache.org/jira/browse/RANGER-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caozhiqiang updated RANGER-3390: Description: The org/apache/ranger/plugin/policyengine/RangerAccessResourceImpl.getAsString function take too much time. We should replace it to improve performance for Yarn and ES. (was: The org/apache/ranger/plugin/policyengine/RangerAccessResourceImpl.getAsString function take too much time. We should replace it to improve performance.) > improve plugin performance for yarn and es by replacing getAsString > --- > > Key: RANGER-3390 > URL: https://issues.apache.org/jira/browse/RANGER-3390 > Project: Ranger > Issue Type: Improvement > Components: plugins >Affects Versions: 2.0.1 >Reporter: caozhiqiang >Assignee: caozhiqiang >Priority: Major > Attachments: RANGER-3390-branch-2.0.1.001.patch > > > The > org/apache/ranger/plugin/policyengine/RangerAccessResourceImpl.getAsString > function take too much time. We should replace it to improve performance for > Yarn and ES. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Review Request 73560: improve plugin performance for yarn and es by replacing getAsString
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73560/ --- Review request for ranger and Madhan Neethiraj. Bugs: RANGER-3390 https://issues.apache.org/jira/browse/RANGER-3390 Repository: ranger Description --- The org/apache/ranger/plugin/policyengine/RangerAccessResourceImpl.getAsString function take too much time. We should replace it to improve performance for yarn and ES. Diffs - plugin-elasticsearch/src/main/java/org/apache/ranger/authorization/elasticsearch/authorizer/RangerElasticsearchAuthorizer.java 31de63195 plugin-yarn/src/main/java/org/apache/ranger/authorization/yarn/authorizer/RangerYarnAuthorizer.java b49fb8ae8 Diff: https://reviews.apache.org/r/73560/diff/1/ Testing --- Thanks, cao zhiqiang
[jira] [Commented] (RANGER-3390) improve plugin performance for yarn and es by replacing getAsString
[ https://issues.apache.org/jira/browse/RANGER-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17407399#comment-17407399 ] caozhiqiang commented on RANGER-3390: - [~madhan], please help to review this patch related to RANGER-3378. Thank you. > improve plugin performance for yarn and es by replacing getAsString > --- > > Key: RANGER-3390 > URL: https://issues.apache.org/jira/browse/RANGER-3390 > Project: Ranger > Issue Type: Improvement > Components: plugins >Affects Versions: 2.0.1 >Reporter: caozhiqiang >Assignee: caozhiqiang >Priority: Major > Attachments: RANGER-3390-branch-2.0.1.001.patch > > > The > org/apache/ranger/plugin/policyengine/RangerAccessResourceImpl.getAsString > function take too much time. We should replace it to improve performance. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Review Request 73523: RANGER-3339 : Make Ranger Solr audit collection config-set configurable.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73523/#review223430 --- Ship it! Ship It! - Mehul Parikh On Aug. 17, 2021, 3:54 p.m., Mateen Mansoori wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/73523/ > --- > > (Updated Aug. 17, 2021, 3:54 p.m.) > > > Review request for ranger, Dhaval Shah, Abhay Kulkarni, Mehul Parikh, Pradeep > Agrawal, Vishal Suvagia, and Velmurugan Periasamy. > > > Bugs: RANGER-3339 > https://issues.apache.org/jira/browse/RANGER-3339 > > > Repository: ranger > > > Description > --- > > Currently ranger admin using a fixed config-set zip to create config, So this > changes are to make it configurable. > > > Diffs > - > > embeddedwebserver/scripts/ranger-admin-services.sh d7cabbcde > > embeddedwebserver/src/main/java/org/apache/ranger/server/tomcat/SolrCollectionBootstrapper.java > d04d1c897 > security-admin/scripts/install.properties ae7a00d3f > security-admin/scripts/setup.sh f6f6f564c > security-admin/src/main/resources/conf.dist/ranger-admin-site.xml 793c479bd > > > Diff: https://reviews.apache.org/r/73523/diff/1/ > > > Testing > --- > > Tested on local VM by providing custom config-set zip location and it was > able to create config-set and collection. > > > Thanks, > > Mateen Mansoori > >
[jira] [Updated] (RANGER-3390) improve plugin performance for yarn and es by replacing getAsString
[ https://issues.apache.org/jira/browse/RANGER-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caozhiqiang updated RANGER-3390: Attachment: RANGER-3390-branch-2.0.1.001.patch > improve plugin performance for yarn and es by replacing getAsString > --- > > Key: RANGER-3390 > URL: https://issues.apache.org/jira/browse/RANGER-3390 > Project: Ranger > Issue Type: Improvement > Components: plugins >Affects Versions: 2.0.1 >Reporter: caozhiqiang >Assignee: caozhiqiang >Priority: Major > Attachments: RANGER-3390-branch-2.0.1.001.patch > > > The > org/apache/ranger/plugin/policyengine/RangerAccessResourceImpl.getAsString > function take too much time. We should replace it to improve performance. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (RANGER-3390) improve plugin performance for yarn and es by replacing getAsString
[ https://issues.apache.org/jira/browse/RANGER-3390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] caozhiqiang updated RANGER-3390: External issue URL: (was: https://issues.apache.org/jira/browse/RANGER-3378) > improve plugin performance for yarn and es by replacing getAsString > --- > > Key: RANGER-3390 > URL: https://issues.apache.org/jira/browse/RANGER-3390 > Project: Ranger > Issue Type: Improvement > Components: plugins >Affects Versions: 2.0.1 >Reporter: caozhiqiang >Assignee: caozhiqiang >Priority: Major > > The > org/apache/ranger/plugin/policyengine/RangerAccessResourceImpl.getAsString > function take too much time. We should replace it to improve performance. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (RANGER-3390) improve plugin performance for yarn and es by replacing getAsString
caozhiqiang created RANGER-3390: --- Summary: improve plugin performance for yarn and es by replacing getAsString Key: RANGER-3390 URL: https://issues.apache.org/jira/browse/RANGER-3390 Project: Ranger Issue Type: Improvement Components: plugins Affects Versions: 2.0.1 Reporter: caozhiqiang Assignee: caozhiqiang The org/apache/ranger/plugin/policyengine/RangerAccessResourceImpl.getAsString function take too much time. We should replace it to improve performance. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (RANGER-3373) Ranger Hbase plugin not compatible with Hbase 2.3.4
[ https://issues.apache.org/jira/browse/RANGER-3373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17407300#comment-17407300 ] Pradeep Agrawal commented on RANGER-3373: - 2.1 is already a released branch so patch can't go in 2.1 branch as well. > Ranger Hbase plugin not compatible with Hbase 2.3.4 > --- > > Key: RANGER-3373 > URL: https://issues.apache.org/jira/browse/RANGER-3373 > Project: Ranger > Issue Type: Bug > Components: plugins >Affects Versions: 2.1.0, 3.0.0, 2.1.1 >Reporter: Shivam Garg >Priority: Major > Labels: patch > Fix For: 3.0.0, 2.1.1 > > Attachments: RANGER-3373-001-branch-2.1.patch, RANGER-3373-001.patch > > > Ranger is incompatible with Hbase 2.3.4 because AccessControlLists class has > been changed to PermissionStorage > https://issues.apache.org/jira/browse/HBASE-22084 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (RANGER-3373) Ranger Hbase plugin not compatible with Hbase 2.3.4
[ https://issues.apache.org/jira/browse/RANGER-3373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17407298#comment-17407298 ] Shivam Garg commented on RANGER-3373: - [~pradeep] We are already using the patch internally in our dev env and it works without any issue. Anyway, can you please review the branch-2.1 patch? > Ranger Hbase plugin not compatible with Hbase 2.3.4 > --- > > Key: RANGER-3373 > URL: https://issues.apache.org/jira/browse/RANGER-3373 > Project: Ranger > Issue Type: Bug > Components: plugins >Affects Versions: 2.1.0, 3.0.0, 2.1.1 >Reporter: Shivam Garg >Priority: Major > Labels: patch > Fix For: 3.0.0, 2.1.1 > > Attachments: RANGER-3373-001-branch-2.1.patch, RANGER-3373-001.patch > > > Ranger is incompatible with Hbase 2.3.4 because AccessControlLists class has > been changed to PermissionStorage > https://issues.apache.org/jira/browse/HBASE-22084 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (RANGER-3386) apache CollectionUtils package reduce ranger agent performance and should be replaced
[
https://issues.apache.org/jira/browse/RANGER-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17407276#comment-17407276
]
caozhiqiang edited comment on RANGER-3386 at 8/31/21, 11:57 AM:
OK. I will make this changer to my company's private branch.
I am not sure what's mean about adding a test case? Thank you.
was (Author: caozhiqiang):
OK. I will make this changer to my company's private branch. Thank you.
> apache CollectionUtils package reduce ranger agent performance and should be
> replaced
> -
>
> Key: RANGER-3386
> URL: https://issues.apache.org/jira/browse/RANGER-3386
> Project: Ranger
> Issue Type: Improvement
> Components: plugins
>Affects Versions: 2.0.1
>Reporter: caozhiqiang
>Assignee: caozhiqiang
>Priority: Major
> Attachments: CollectionUtils.png, RANGER-3386-branch-2.0.1.001.patch,
> RANGER-3386-branch-2.0.1.002.patch
>
>
> org.apache.commons.collections.CollectionUtils' performance is too poor and
> reduce the performance of ranger plugins, particularly with hdfs. There are
> too many places used CollectionUtils.isNotEmpty and CollectionUtils.isEmpty
> in agent component, so we should replace them.
> We can see many CollectionUtils call is take too much time in namenode
> benchmark's result.
> In this patch, I replace almost CollectionUtils functions in agents-common.
> After adding this patch, in creating file benchmark of hdfs, the performance
> can improve from creating *7000* files to *7600* files per second.
> By the way, I write a simple test code below. collection.isEmpty is almost
> take 0 milliseconds, but CollectionUtils.isNotEmpty take 5 milliseconds.
>
> {code:java}
> List list = new ArrayList();
> for(int i = 0; i<1000; i++)
> list.add(i);
> long startTime = System.currentTimeMillis();
> if(list != null && !list.isEmpty()){
> }
> long endTime = System.currentTimeMillis();
> long totalTime = endTime - startTime;
> System.out.println(totalTime);
> long startTime2 = System.currentTimeMillis();
> CollectionUtils.isNotEmpty(list);
> long endTime2 = System.currentTimeMillis();
> long totalTime2 = endTime2 - startTime2;
> System.out.println(totalTime2);
> {code}
>
> !CollectionUtils.png|width=680,height=324!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (RANGER-3386) apache CollectionUtils package reduce ranger agent performance and should be replaced
[
https://issues.apache.org/jira/browse/RANGER-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17407276#comment-17407276
]
caozhiqiang edited comment on RANGER-3386 at 8/31/21, 11:56 AM:
OK. I will make this changer to my company's private branch. Thank you.
was (Author: caozhiqiang):
OK. I will make this changer to my company's private branch. Thank you.
> apache CollectionUtils package reduce ranger agent performance and should be
> replaced
> -
>
> Key: RANGER-3386
> URL: https://issues.apache.org/jira/browse/RANGER-3386
> Project: Ranger
> Issue Type: Improvement
> Components: plugins
>Affects Versions: 2.0.1
>Reporter: caozhiqiang
>Assignee: caozhiqiang
>Priority: Major
> Attachments: CollectionUtils.png, RANGER-3386-branch-2.0.1.001.patch,
> RANGER-3386-branch-2.0.1.002.patch
>
>
> org.apache.commons.collections.CollectionUtils' performance is too poor and
> reduce the performance of ranger plugins, particularly with hdfs. There are
> too many places used CollectionUtils.isNotEmpty and CollectionUtils.isEmpty
> in agent component, so we should replace them.
> We can see many CollectionUtils call is take too much time in namenode
> benchmark's result.
> In this patch, I replace almost CollectionUtils functions in agents-common.
> After adding this patch, in creating file benchmark of hdfs, the performance
> can improve from creating *7000* files to *7600* files per second.
> By the way, I write a simple test code below. collection.isEmpty is almost
> take 0 milliseconds, but CollectionUtils.isNotEmpty take 5 milliseconds.
>
> {code:java}
> List list = new ArrayList();
> for(int i = 0; i<1000; i++)
> list.add(i);
> long startTime = System.currentTimeMillis();
> if(list != null && !list.isEmpty()){
> }
> long endTime = System.currentTimeMillis();
> long totalTime = endTime - startTime;
> System.out.println(totalTime);
> long startTime2 = System.currentTimeMillis();
> CollectionUtils.isNotEmpty(list);
> long endTime2 = System.currentTimeMillis();
> long totalTime2 = endTime2 - startTime2;
> System.out.println(totalTime2);
> {code}
>
> !CollectionUtils.png|width=680,height=324!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (RANGER-3386) apache CollectionUtils package reduce ranger agent performance and should be replaced
[
https://issues.apache.org/jira/browse/RANGER-3386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17407276#comment-17407276
]
caozhiqiang commented on RANGER-3386:
-
OK. I will make this changer to my company's private branch. Thank you.
> apache CollectionUtils package reduce ranger agent performance and should be
> replaced
> -
>
> Key: RANGER-3386
> URL: https://issues.apache.org/jira/browse/RANGER-3386
> Project: Ranger
> Issue Type: Improvement
> Components: plugins
>Affects Versions: 2.0.1
>Reporter: caozhiqiang
>Assignee: caozhiqiang
>Priority: Major
> Attachments: CollectionUtils.png, RANGER-3386-branch-2.0.1.001.patch,
> RANGER-3386-branch-2.0.1.002.patch
>
>
> org.apache.commons.collections.CollectionUtils' performance is too poor and
> reduce the performance of ranger plugins, particularly with hdfs. There are
> too many places used CollectionUtils.isNotEmpty and CollectionUtils.isEmpty
> in agent component, so we should replace them.
> We can see many CollectionUtils call is take too much time in namenode
> benchmark's result.
> In this patch, I replace almost CollectionUtils functions in agents-common.
> After adding this patch, in creating file benchmark of hdfs, the performance
> can improve from creating *7000* files to *7600* files per second.
> By the way, I write a simple test code below. collection.isEmpty is almost
> take 0 milliseconds, but CollectionUtils.isNotEmpty take 5 milliseconds.
>
> {code:java}
> List list = new ArrayList();
> for(int i = 0; i<1000; i++)
> list.add(i);
> long startTime = System.currentTimeMillis();
> if(list != null && !list.isEmpty()){
> }
> long endTime = System.currentTimeMillis();
> long totalTime = endTime - startTime;
> System.out.println(totalTime);
> long startTime2 = System.currentTimeMillis();
> CollectionUtils.isNotEmpty(list);
> long endTime2 = System.currentTimeMillis();
> long totalTime2 = endTime2 - startTime2;
> System.out.println(totalTime2);
> {code}
>
> !CollectionUtils.png|width=680,height=324!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (RANGER-3369) User Profile tab change password the upd_by_id does not update in database
[ https://issues.apache.org/jira/browse/RANGER-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17407187#comment-17407187 ] Haihui Xu commented on RANGER-3369: --- [~aakashpradeep] Please help to reviwe, thanks. > User Profile tab change password the upd_by_id does not update in database > -- > > Key: RANGER-3369 > URL: https://issues.apache.org/jira/browse/RANGER-3369 > Project: Ranger > Issue Type: Improvement > Components: admin >Affects Versions: 2.1.0, 2.2.0 >Reporter: Haihui Xu >Assignee: Haihui Xu >Priority: Major > Fix For: 3.0.0 > > Attachments: RANGER-3369.patch > > > When on User Profile front page change user password, the upd_by_id of table > x_portal_user does not update in database. It should be updated. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Issue Comment Deleted] (RANGER-3369) User Profile tab change password the upd_by_id does not update in database
[ https://issues.apache.org/jira/browse/RANGER-3369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haihui Xu updated RANGER-3369: -- Comment: was deleted (was: [~mad...@apache.org] Please review it, thanks.) > User Profile tab change password the upd_by_id does not update in database > -- > > Key: RANGER-3369 > URL: https://issues.apache.org/jira/browse/RANGER-3369 > Project: Ranger > Issue Type: Improvement > Components: admin >Affects Versions: 2.1.0, 2.2.0 >Reporter: Haihui Xu >Assignee: Haihui Xu >Priority: Major > Fix For: 3.0.0 > > Attachments: RANGER-3369.patch > > > When on User Profile front page change user password, the upd_by_id of table > x_portal_user does not update in database. It should be updated. -- This message was sent by Atlassian Jira (v8.3.4#803005)
