[jira] [Commented] (RANGER-2789) GET API service/xusers/users turns very slow when there are more than 1000 users

2020-05-31 Thread Pradeep Agrawal (Jira) 


[ 
https://issues.apache.org/jira/browse/RANGER-2789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120767#comment-17120767
 ] 

Pradeep Agrawal commented on RANGER-2789:
-

[~RickyMa]  : i still see old patch there

> GET API service/xusers/users turns very slow when there are more than 1000 
> users
> 
>
> Key: RANGER-2789
> URL: https://issues.apache.org/jira/browse/RANGER-2789
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>Affects Versions: 2.0.0
> Environment: hdp 2.2
>Reporter: RickyMa
>Priority: Minor
>  Labels: optimization, performance, ranger, rangeradmin, slow
> Attachments: RANGER-2789.patch, image-2020-04-10-19-37-31-570.png
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> We have more than 1000 users in production environment.
> When calling API 'service/xusers/users?pageSize=1000=0' using HTTP 
> GET method, Ranger Admin takes 10+ minutes to response.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (RANGER-2839) Assorted improvements for debugging and handling of thread terminations, clean-up of unused data, etc.

2020-05-31 Thread Pradeep Agrawal (Jira) 


 [ 
https://issues.apache.org/jira/browse/RANGER-2839?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Pradeep Agrawal updated RANGER-2839:

Fix Version/s: (was: master)
   2.1.0

> Assorted improvements for debugging and handling of thread terminations, 
> clean-up of unused data, etc.
> --
>
> Key: RANGER-2839
> URL: https://issues.apache.org/jira/browse/RANGER-2839
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>Affects Versions: master
>Reporter: Abhay Kulkarni
>Assignee: Abhay Kulkarni
>Priority: Major
> Fix For: 2.1.0
>
>
> These are various improvements/fixes needed for issues discovered during 
> testing with incremental policy/delta enabled.
> Highlights:
> 1. Threads are named for better debugging information.
> 2. super.join() call is made in a loop to ensure that thread is really 
> terminated even in the face of spurious/out-of-sync interruptions.
> 3. preCleanup() call ensures that clean-up happens at least and exactly once, 
> when incremental policy/tag feature is enabled.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72346: RANGER-2789: GET API service/xusers/users turns very slow when there are more than 1000 users

2020-05-31 Thread Pradeep Agrawal 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72346/#review220925
---




security-admin/src/main/resources/META-INF/jpa_named_queries.xml
Lines 227 (patched)


This will be  removed if you are considering to implement feedback given by 
madhan


- Pradeep Agrawal


On May 31, 2020, 4:11 p.m., Haoxiang Ma wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72346/
> ---
> 
> (Updated May 31, 2020, 4:11 p.m.)
> 
> 
> Review request for ranger.
> 
> 
> Bugs: RANGER-2789
> https://issues.apache.org/jira/browse/RANGER-2789
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> GET API service/xusers/users turns very slow when there are more than 1000 
> users
> 
> 
> Diffs
> -
> 
>   security-admin/src/main/java/org/apache/ranger/db/XXGroupUserDao.java 
> 1e41e708a 
>   security-admin/src/main/java/org/apache/ranger/entity/XXUser.java 0464e7b6e 
>   security-admin/src/main/java/org/apache/ranger/service/XUserService.java 
> 6ff8823da 
>   
> security-admin/src/main/java/org/apache/ranger/service/XUserServiceBase.java 
> 9cdc14ebf 
>   security-admin/src/main/java/org/apache/ranger/view/VXUser.java 96f6468f7 
>   security-admin/src/main/resources/META-INF/jpa_named_queries.xml 22e71e549 
> 
> 
> Diff: https://reviews.apache.org/r/72346/diff/1/
> 
> 
> Testing
> ---
> 
> 1.Compilation OK
> 2.Already used in production environment
> 
> 
> File Attachments
> 
> 
> RANGER-2789.patch
>   
> https://reviews.apache.org/media/uploaded/files/2020/05/29/419dc717-cec5-4dfe-b3cc-28e99f94884d__RANGER-2789.patch
> 
> 
> Thanks,
> 
> Haoxiang Ma
> 
>

Re: Review Request 72346: RANGER-2789: GET API service/xusers/users turns very slow when there are more than 1000 users

2020-05-31 Thread Pradeep Agrawal 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72346/#review220924
---




security-admin/src/main/java/org/apache/ranger/db/XXGroupUserDao.java
Lines 69 (patched)


The issue is not resolved, reopening it again



security-admin/src/main/java/org/apache/ranger/entity/XXUser.java
Lines 290 (patched)


Not fixed , reopening now.



security-admin/src/main/java/org/apache/ranger/view/VXUser.java
Lines 315 (patched)


Not fixed, reopening now



security-admin/src/main/java/org/apache/ranger/view/VXUser.java
Lines 323 (patched)


Not fixed, reopening now


- Pradeep Agrawal


On May 31, 2020, 4:11 p.m., Haoxiang Ma wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72346/
> ---
> 
> (Updated May 31, 2020, 4:11 p.m.)
> 
> 
> Review request for ranger.
> 
> 
> Bugs: RANGER-2789
> https://issues.apache.org/jira/browse/RANGER-2789
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> GET API service/xusers/users turns very slow when there are more than 1000 
> users
> 
> 
> Diffs
> -
> 
>   security-admin/src/main/java/org/apache/ranger/db/XXGroupUserDao.java 
> 1e41e708a 
>   security-admin/src/main/java/org/apache/ranger/entity/XXUser.java 0464e7b6e 
>   security-admin/src/main/java/org/apache/ranger/service/XUserService.java 
> 6ff8823da 
>   
> security-admin/src/main/java/org/apache/ranger/service/XUserServiceBase.java 
> 9cdc14ebf 
>   security-admin/src/main/java/org/apache/ranger/view/VXUser.java 96f6468f7 
>   security-admin/src/main/resources/META-INF/jpa_named_queries.xml 22e71e549 
> 
> 
> Diff: https://reviews.apache.org/r/72346/diff/1/
> 
> 
> Testing
> ---
> 
> 1.Compilation OK
> 2.Already used in production environment
> 
> 
> File Attachments
> 
> 
> RANGER-2789.patch
>   
> https://reviews.apache.org/media/uploaded/files/2020/05/29/419dc717-cec5-4dfe-b3cc-28e99f94884d__RANGER-2789.patch
> 
> 
> Thanks,
> 
> Haoxiang Ma
> 
>

[jira] [Commented] (RANGER-2676) Before obtaining AuthzAuditEvent Object should check xasecure.audit.destination.xx value

2020-05-31 Thread Haihui Xu (Jira) 


[ 
https://issues.apache.org/jira/browse/RANGER-2676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120745#comment-17120745
 ] 

Haihui Xu commented on RANGER-2676:
---

This may be performance issue, [~gautam] please review it

> Before obtaining AuthzAuditEvent Object should check 
> xasecure.audit.destination.xx value
> 
>
> Key: RANGER-2676
> URL: https://issues.apache.org/jira/browse/RANGER-2676
> Project: Ranger
>  Issue Type: Improvement
>  Components: audit, plugins
>Affects Versions: 1.0.0, 1.1.0, 2.0.0, 1.2.0
>Reporter: Haihui Xu
>Assignee: Haihui Xu
>Priority: Critical
>  Labels: patch
> Attachments: RANGER-2676.patch
>
>
> When after evalute policy and then process the auditlog，need to obtain 
> AuditHandler Object and AuthzAuditEvent object
> (1) AuditHandler object obtained when init plugin according to the value of 
> xasecure.audit.destination.db/solr/hdfs/...
> (2) So, obtaining AuthzAuditEvent should also according to the value of 
> xasecure.audit.destination.db/solr/hdfs/...,or when AuditHandler is null or 
> dummyHandler, constructing AuthzAuditEvent object is useless which is very 
> frequently called and thus result in the  performance decline of service(hdfs 
> namenode, kakfa broker,hiveserver2...)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (RANGER-2369) Cannot multi-thread process hive grant sql when installed ranger-hive-plugin

2020-05-31 Thread Haihui Xu (Jira) 


[ 
https://issues.apache.org/jira/browse/RANGER-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120744#comment-17120744
 ] 

Haihui Xu commented on RANGER-2369:
---

This may be mysql config error.

> Cannot multi-thread process hive grant sql when installed ranger-hive-plugin
> 
>
> Key: RANGER-2369
> URL: https://issues.apache.org/jira/browse/RANGER-2369
> Project: Ranger
>  Issue Type: Bug
>  Components: admin, Ranger
>Affects Versions: 0.5.3, 1.1.0
>Reporter: Haihui Xu
>Assignee: Gautam Borad
>Priority: Major
> Fix For: 1.2.1
>
>
> 1、enable ranger-hive-plugin successfully；
> 2、Multi-thread process “ grant select on table  tableName to user userName“；
> 3、the rangeradmin （0.5.3）log errors are：
> 2019-03-12 16:20:28,157 [http-bio-6080-exec-3] ERROR 
> org.apache.ranger.rest.ServiceREST (ServiceREST.java:951) - 
> grantAccess(NMG1_hive, GrantRevokeRequest={grantor=
> {hive}
> resource=\{column=*; table=bbzhenhao1; database=default; } users=\{hxatest } 
> groups={} accessTypes=\{select } delegateAdmin=\{false} enableAudit=\{true} 
> replaceExistingPermissions=\{false} isRecursive=\{false} 
> clientIPAddress=\{10.129.3.1} clientType=\{HIVESERVER2} requestData={} 
> sessionId=\{59259b8c-223c-45f5-95cd-a30671de4d79} }) failed
>  javax.persistence.PersistenceException: Exception [EclipseLink-4002] 
> (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): 
> org.eclipse.persistence.exceptions.DatabaseException
>  Internal Exception: 
> com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock 
> found when trying to get lock; try restarting transaction
>  Error Code: 1213
>  Call: UPDATE x_service SET policy_update_time = ?, policy_version = ?, 
> UPDATE_TIME = ?, version = ? WHERE ((id = ?) AND (version = ?))
>  bind => [6 parameters bound]
>  Query: UpdateObjectQuery(XXService [id=2])
>  at 
> org.eclipse.persistence.internal.jpa.EntityManagerImpl.flush(EntityManagerImpl.java:868)
>  at sun.reflect.GeneratedMethodAccessor53.invoke(Unknown Source)
>  at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:606)
>  at 
> org.springframework.orm.jpa.SharedEntityManagerCreator$SharedEntityManagerInvocationHandler.invoke(SharedEntityManagerCreator.java:240)
>  at com.sun.proxy.$Proxy20.flush(Unknown Source)
>  at org.apache.ranger.common.db.BaseDao.update(BaseDao.java:99)
>  at 
> org.apache.ranger.biz.ServiceDBStore.updatePolicyVersion(ServiceDBStore.java:1896)
>  at 
> org.apache.ranger.biz.ServiceDBStore.handlePolicyUpdate(ServiceDBStore.java:1864)
>  at 
> org.apache.ranger.biz.ServiceDBStore.createPolicy(ServiceDBStore.java:1384)
>  at org.apache.ranger.rest.ServiceREST.grantAccess(ServiceREST.java:946)
>  at 
> org.apache.ranger.rest.ServiceREST$$FastClassByCGLIB$$92dab672.invoke()
>  at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:191)
>  at 
> org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:689)
>  at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>  at 
> org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110)
>  at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>  at 
> org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:622)
>  at 
> org.apache.ranger.rest.ServiceREST$$EnhancerByCGLIB$$d842e7d5.grantAccess()
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>  at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (RANGER-2369) Cannot multi-thread process hive grant sql when installed ranger-hive-plugin

2020-05-31 Thread Haihui Xu (Jira) 


 [ 
https://issues.apache.org/jira/browse/RANGER-2369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Haihui Xu resolved RANGER-2369.
---
Resolution: Won't Do

> Cannot multi-thread process hive grant sql when installed ranger-hive-plugin
> 
>
> Key: RANGER-2369
> URL: https://issues.apache.org/jira/browse/RANGER-2369
> Project: Ranger
>  Issue Type: Bug
>  Components: admin, Ranger
>Affects Versions: 0.5.3, 1.1.0
>Reporter: Haihui Xu
>Assignee: Gautam Borad
>Priority: Major
> Fix For: 1.2.1
>
>
> 1、enable ranger-hive-plugin successfully；
> 2、Multi-thread process “ grant select on table  tableName to user userName“；
> 3、the rangeradmin （0.5.3）log errors are：
> 2019-03-12 16:20:28,157 [http-bio-6080-exec-3] ERROR 
> org.apache.ranger.rest.ServiceREST (ServiceREST.java:951) - 
> grantAccess(NMG1_hive, GrantRevokeRequest={grantor=
> {hive}
> resource=\{column=*; table=bbzhenhao1; database=default; } users=\{hxatest } 
> groups={} accessTypes=\{select } delegateAdmin=\{false} enableAudit=\{true} 
> replaceExistingPermissions=\{false} isRecursive=\{false} 
> clientIPAddress=\{10.129.3.1} clientType=\{HIVESERVER2} requestData={} 
> sessionId=\{59259b8c-223c-45f5-95cd-a30671de4d79} }) failed
>  javax.persistence.PersistenceException: Exception [EclipseLink-4002] 
> (Eclipse Persistence Services - 2.5.2.v20140319-9ad6abd): 
> org.eclipse.persistence.exceptions.DatabaseException
>  Internal Exception: 
> com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackException: Deadlock 
> found when trying to get lock; try restarting transaction
>  Error Code: 1213
>  Call: UPDATE x_service SET policy_update_time = ?, policy_version = ?, 
> UPDATE_TIME = ?, version = ? WHERE ((id = ?) AND (version = ?))
>  bind => [6 parameters bound]
>  Query: UpdateObjectQuery(XXService [id=2])
>  at 
> org.eclipse.persistence.internal.jpa.EntityManagerImpl.flush(EntityManagerImpl.java:868)
>  at sun.reflect.GeneratedMethodAccessor53.invoke(Unknown Source)
>  at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:606)
>  at 
> org.springframework.orm.jpa.SharedEntityManagerCreator$SharedEntityManagerInvocationHandler.invoke(SharedEntityManagerCreator.java:240)
>  at com.sun.proxy.$Proxy20.flush(Unknown Source)
>  at org.apache.ranger.common.db.BaseDao.update(BaseDao.java:99)
>  at 
> org.apache.ranger.biz.ServiceDBStore.updatePolicyVersion(ServiceDBStore.java:1896)
>  at 
> org.apache.ranger.biz.ServiceDBStore.handlePolicyUpdate(ServiceDBStore.java:1864)
>  at 
> org.apache.ranger.biz.ServiceDBStore.createPolicy(ServiceDBStore.java:1384)
>  at org.apache.ranger.rest.ServiceREST.grantAccess(ServiceREST.java:946)
>  at 
> org.apache.ranger.rest.ServiceREST$$FastClassByCGLIB$$92dab672.invoke()
>  at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:191)
>  at 
> org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:689)
>  at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>  at 
> org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:110)
>  at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>  at 
> org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:622)
>  at 
> org.apache.ranger.rest.ServiceREST$$EnhancerByCGLIB$$d842e7d5.grantAccess()
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>  at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>  at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72539: RANGER-2839: Assorted improvements for debugging and handling of thread terminations, clean-up of unused data, etc.

2020-05-31 Thread Madhan Neethiraj 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72539/#review220923
---


Ship it!




Ship It!

- Madhan Neethiraj


On May 31, 2020, 9:36 p.m., Abhay Kulkarni wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72539/
> ---
> 
> (Updated May 31, 2020, 9:36 p.m.)
> 
> 
> Review request for ranger, Madhan Neethiraj and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-2839
> https://issues.apache.org/jira/browse/RANGER-2839
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> These are various improvements/fixes needed for issues discovered during 
> testing of plugins with incremental policy/delta enabled.
> 
> Highlights:
> 1. Threads are named for better debugging information.
> 2. super.join() call is made in a loop to ensure that thread is really 
> terminated even in the face of spurious/out-of-sync interruptions.
> 3. preCleanup() call ensures that clean-up happens at least and exactly once, 
> when incremental policy/tag feature is enabled.
> 
> 
> Diffs
> -
> 
>   
> agents-common/src/main/java/org/apache/ranger/plugin/contextenricher/RangerTagEnricher.java
>  5d6a4036a 
>   
> agents-common/src/main/java/org/apache/ranger/plugin/contextenricher/RangerUserStoreRefresher.java
>  867ac90ad 
>   
> agents-common/src/main/java/org/apache/ranger/plugin/model/validation/RangerServiceDefHelper.java
>  943b3618f 
>   
> agents-common/src/main/java/org/apache/ranger/plugin/policyengine/PolicyEngine.java
>  297d02fbb 
>   
> agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyEngineImpl.java
>  b59440933 
>   
> agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyRepository.java
>  197c30f0d 
>   
> agents-common/src/main/java/org/apache/ranger/plugin/service/RangerBasePlugin.java
>  236a4ab17 
>   
> agents-common/src/main/java/org/apache/ranger/plugin/util/PolicyRefresher.java
>  63a98f2d4 
>   
> security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyAdminCache.java
>  df3c10d3f 
>   
> security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyAdminImpl.java 
> 1c63e94c0 
> 
> 
> Diff: https://reviews.apache.org/r/72539/diff/3/
> 
> 
> Testing
> ---
> 
> Tested along with HDFS and Hive plugins. Works as expected and provides good 
> set of debugging help for tracking creation and termination of threads 
> managed by Ranger plugin.
> 
> 
> Thanks,
> 
> Abhay Kulkarni
> 
>

Re: Review Request 72539: RANGER-2839: Assorted improvements for debugging and handling of thread terminations, clean-up of unused data, etc.

2020-05-31 Thread Abhay Kulkarni 


> On May 28, 2020, 5:35 p.m., Madhan Neethiraj wrote:
> > agents-common/src/main/java/org/apache/ranger/plugin/contextenricher/RangerTagEnricher.java
> > Line 562 (original), 567 (patched)
> > 
> >
> > serviceDefHelper.getServiceDef() => 
> > serviceDefHelper.getServiceDef().getName()
> >   - note that serviceDefHelper.getServiceDef() can return null, so this 
> > condition must be handled

As RangerServiceDefHelper object's initialization ensures that its reference to 
RangerServiceDef object is never null, RangerServiceDefHelper.getServiceDef() 
API is changed to never return null.


- Abhay


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72539/#review220906
---


On May 28, 2020, 5:12 p.m., Abhay Kulkarni wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72539/
> ---
> 
> (Updated May 28, 2020, 5:12 p.m.)
> 
> 
> Review request for ranger, Madhan Neethiraj and Velmurugan Periasamy.
> 
> 
> Bugs: RANGER-2839
> https://issues.apache.org/jira/browse/RANGER-2839
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> These are various improvements/fixes needed for issues discovered during 
> testing of plugins with incremental policy/delta enabled.
> 
> Highlights:
> 1. Threads are named for better debugging information.
> 2. super.join() call is made in a loop to ensure that thread is really 
> terminated even in the face of spurious/out-of-sync interruptions.
> 3. preCleanup() call ensures that clean-up happens at least and exactly once, 
> when incremental policy/tag feature is enabled.
> 
> 
> Diffs
> -
> 
>   
> agents-common/src/main/java/org/apache/ranger/plugin/contextenricher/RangerTagEnricher.java
>  5d6a4036a 
>   
> agents-common/src/main/java/org/apache/ranger/plugin/model/validation/RangerServiceDefHelper.java
>  943b3618f 
>   
> agents-common/src/main/java/org/apache/ranger/plugin/policyengine/PolicyEngine.java
>  297d02fbb 
>   
> agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyEngineImpl.java
>  b59440933 
>   
> agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyRepository.java
>  197c30f0d 
>   
> agents-common/src/main/java/org/apache/ranger/plugin/service/RangerBasePlugin.java
>  236a4ab17 
>   
> agents-common/src/main/java/org/apache/ranger/plugin/util/PolicyRefresher.java
>  63a98f2d4 
>   
> security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyAdminCache.java
>  df3c10d3f 
>   
> security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyAdminImpl.java 
> 1c63e94c0 
> 
> 
> Diff: https://reviews.apache.org/r/72539/diff/2/
> 
> 
> Testing
> ---
> 
> Tested along with HDFS and Hive plugins. Works as expected and provides good 
> set of debugging help for tracking creation and termination of threads 
> managed by Ranger plugin.
> 
> 
> Thanks,
> 
> Abhay Kulkarni
> 
>

Re: Review Request 72539: RANGER-2839: Assorted improvements for debugging and handling of thread terminations, clean-up of unused data, etc.

2020-05-31 Thread Abhay Kulkarni 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72539/
---

(Updated May 31, 2020, 9:36 p.m.)


Review request for ranger, Madhan Neethiraj and Velmurugan Periasamy.


Changes
---

Updated to fix issues raised by review comments.


Bugs: RANGER-2839
https://issues.apache.org/jira/browse/RANGER-2839


Repository: ranger


Description
---

These are various improvements/fixes needed for issues discovered during 
testing of plugins with incremental policy/delta enabled.

Highlights:
1. Threads are named for better debugging information.
2. super.join() call is made in a loop to ensure that thread is really 
terminated even in the face of spurious/out-of-sync interruptions.
3. preCleanup() call ensures that clean-up happens at least and exactly once, 
when incremental policy/tag feature is enabled.


Diffs (updated)
-

  
agents-common/src/main/java/org/apache/ranger/plugin/contextenricher/RangerTagEnricher.java
 5d6a4036a 
  
agents-common/src/main/java/org/apache/ranger/plugin/contextenricher/RangerUserStoreRefresher.java
 867ac90ad 
  
agents-common/src/main/java/org/apache/ranger/plugin/model/validation/RangerServiceDefHelper.java
 943b3618f 
  
agents-common/src/main/java/org/apache/ranger/plugin/policyengine/PolicyEngine.java
 297d02fbb 
  
agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyEngineImpl.java
 b59440933 
  
agents-common/src/main/java/org/apache/ranger/plugin/policyengine/RangerPolicyRepository.java
 197c30f0d 
  
agents-common/src/main/java/org/apache/ranger/plugin/service/RangerBasePlugin.java
 236a4ab17 
  
agents-common/src/main/java/org/apache/ranger/plugin/util/PolicyRefresher.java 
63a98f2d4 
  
security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyAdminCache.java 
df3c10d3f 
  security-admin/src/main/java/org/apache/ranger/biz/RangerPolicyAdminImpl.java 
1c63e94c0 


Diff: https://reviews.apache.org/r/72539/diff/3/

Changes: https://reviews.apache.org/r/72539/diff/2-3/


Testing
---

Tested along with HDFS and Hive plugins. Works as expected and provides good 
set of debugging help for tracking creation and termination of threads managed 
by Ranger plugin.


Thanks,

Abhay Kulkarni

[jira] [Commented] (RANGER-2789) GET API service/xusers/users turns very slow when there are more than 1000 users

2020-05-31 Thread RickyMa (Jira) 


[ 
https://issues.apache.org/jira/browse/RANGER-2789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120586#comment-17120586
 ] 

RickyMa commented on RANGER-2789:
-

[~pradeep] : My bad. I uploaded the newest patch last time, but I forgot to 
publish it... When you said 'published the changes', all I was thinking about 
'updating the pull request' Now I've really published the review request. 
But I'm new to this, so I'm not sure if I'm doing this right. Could you please 
check it again to see if you can see the patch right now?

> GET API service/xusers/users turns very slow when there are more than 1000 
> users
> 
>
> Key: RANGER-2789
> URL: https://issues.apache.org/jira/browse/RANGER-2789
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>Affects Versions: 2.0.0
> Environment: hdp 2.2
>Reporter: RickyMa
>Priority: Minor
>  Labels: optimization, performance, ranger, rangeradmin, slow
> Attachments: RANGER-2789.patch, image-2020-04-10-19-37-31-570.png
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> We have more than 1000 users in production environment.
> When calling API 'service/xusers/users?pageSize=1000=0' using HTTP 
> GET method, Ranger Admin takes 10+ minutes to response.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72346: RANGER-2789: GET API service/xusers/users turns very slow when there are more than 1000 users

2020-05-31 Thread Haoxiang Ma 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72346/
---

(Updated 五月 31, 2020, 4:11 p.m.)


Review request for ranger.


Bugs: RANGER-2789
https://issues.apache.org/jira/browse/RANGER-2789


Repository: ranger


Description
---

GET API service/xusers/users turns very slow when there are more than 1000 users


Diffs
-

  security-admin/src/main/java/org/apache/ranger/db/XXGroupUserDao.java 
1e41e708a 
  security-admin/src/main/java/org/apache/ranger/entity/XXUser.java 0464e7b6e 
  security-admin/src/main/java/org/apache/ranger/service/XUserService.java 
6ff8823da 
  security-admin/src/main/java/org/apache/ranger/service/XUserServiceBase.java 
9cdc14ebf 
  security-admin/src/main/java/org/apache/ranger/view/VXUser.java 96f6468f7 
  security-admin/src/main/resources/META-INF/jpa_named_queries.xml 22e71e549 


Diff: https://reviews.apache.org/r/72346/diff/1/


Testing
---

1.Compilation OK
2.Already used in production environment


File Attachments (updated)


RANGER-2789.patch
  
https://reviews.apache.org/media/uploaded/files/2020/05/29/419dc717-cec5-4dfe-b3cc-28e99f94884d__RANGER-2789.patch


Thanks,

Haoxiang Ma

[jira] [Commented] (RANGER-2789) GET API service/xusers/users turns very slow when there are more than 1000 users

2020-05-31 Thread Pradeep Agrawal (Jira) 


[ 
https://issues.apache.org/jira/browse/RANGER-2789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120475#comment-17120475
 ] 

Pradeep Agrawal commented on RANGER-2789:
-

[~RickyMa] : You need to upload your new patch in the Review board here: 
[https://reviews.apache.org/r/72346/]

> GET API service/xusers/users turns very slow when there are more than 1000 
> users
> 
>
> Key: RANGER-2789
> URL: https://issues.apache.org/jira/browse/RANGER-2789
> Project: Ranger
>  Issue Type: Bug
>  Components: Ranger
>Affects Versions: 2.0.0
> Environment: hdp 2.2
>Reporter: RickyMa
>Priority: Minor
>  Labels: optimization, performance, ranger, rangeradmin, slow
> Attachments: RANGER-2789.patch, image-2020-04-10-19-37-31-570.png
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> We have more than 1000 users in production environment.
> When calling API 'service/xusers/users?pageSize=1000=0' using HTTP 
> GET method, Ranger Admin takes 10+ minutes to response.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Review Request 72551: RANGER-2834: Upgrade Ranger to support Elasticsearch 7.6.0

2020-05-31 Thread bhavik patel 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/72551/#review220921
---


Ship it!




Ship It!

- bhavik patel


On May 27, 2020, 2:46 p.m., Pradeep Agrawal wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/72551/
> ---
> 
> (Updated May 27, 2020, 2:46 p.m.)
> 
> 
> Review request for ranger, Andrew Charneski, Ankita Sinha, Bolke de Bruin, 
> Don Bosco Durai, bhavik patel, Colm O hEigeartaigh, Gautam Borad, Jayendra 
> Parab, Abhay Kulkarni, Madhan Neethiraj, Mehul Parikh, Nikhil P, Nitin 
> Galave, pengjianhua, Ramesh Mani, Selvamohan Neethiraj, Sailaja Polavarapu, 
> Velmurugan Periasamy, Qiang Zhang, and Barna Zsombor Klara.
> 
> 
> Bugs: RANGER-2834
> https://issues.apache.org/jira/browse/RANGER-2834
> 
> 
> Repository: ranger
> 
> 
> Description
> ---
> 
> **Problem Statement:** Ranger currently supports elastic search version 6.2.2 
> which is around 2 year old.
> 
> **Proposed solution:** Here I am suggesting to upgrade to 7.6.0 version.
> 
> 
> Diffs
> -
> 
>   agents-audit/pom.xml 12d621f02 
>   
> agents-audit/src/main/java/org/apache/ranger/audit/destination/ElasticSearchAuditDestination.java
>  dc00fecfe 
>   
> plugin-elasticsearch/src/main/java/org/apache/ranger/authorization/elasticsearch/authorizer/RangerElasticsearchAuthorizer.java
>  bd0585c8a 
>   
> plugin-elasticsearch/src/main/java/org/apache/ranger/services/elasticsearch/RangerServiceElasticsearch.java
>  ed1c98e5c 
>   
> plugin-elasticsearch/src/main/java/org/apache/ranger/services/elasticsearch/client/ElasticsearchClient.java
>  d5c170d47 
>   pom.xml 5605a1701 
>   
> ranger-elasticsearch-plugin-shim/src/main/java/org/apache/ranger/authorization/elasticsearch/authorizer/RangerElasticsearchAuthorizer.java
>  8f33cb99b 
>   
> ranger-elasticsearch-plugin-shim/src/main/java/org/apache/ranger/authorization/elasticsearch/plugin/RangerElasticsearchPlugin.java
>  faeac5234 
>   
> ranger-elasticsearch-plugin-shim/src/main/java/org/apache/ranger/authorization/elasticsearch/plugin/action/filter/RangerSecurityActionFilter.java
>  60baaea62 
>   
> ranger-elasticsearch-plugin-shim/src/main/java/org/apache/ranger/authorization/elasticsearch/plugin/rest/filter/RangerSecurityRestFilter.java
>  117356a06 
>   
> ranger-elasticsearch-plugin-shim/src/main/java/org/apache/ranger/authorization/elasticsearch/plugin/utils/RequestUtils.java
>  8a04b06d5 
>   
> security-admin/src/main/java/org/apache/ranger/elasticsearch/ElasticSearchAccessAuditsService.java
>  c88726fbb 
>   
> security-admin/src/main/java/org/apache/ranger/elasticsearch/ElasticSearchUtil.java
>  32a45838f 
> 
> 
> Diff: https://reviews.apache.org/r/72551/diff/1/
> 
> 
> Testing
> ---
> 
> Tested ranger audits on elastic search 7.6.0 version
> 
> 
> Prerequisite: For testing of this patch RANGER-2837(RR: 72550) changes are 
> also required.
> 
> 
> Thanks,
> 
> Pradeep Agrawal
> 
>

[jira] [Commented] (RANGER-2837) Add missing dependencies in assembly

2020-05-31 Thread Bhavik Patel (Jira) 


[ 
https://issues.apache.org/jira/browse/RANGER-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120473#comment-17120473
 ] 

Bhavik Patel commented on RANGER-2837:
--

commit link of master branch: 
https://github.com/apache/ranger/commit/d4d19bb056f1c2d32eda250f8a990bcf3c2d57df

> Add missing dependencies in assembly
> 
>
> Key: RANGER-2837
> URL: https://issues.apache.org/jira/browse/RANGER-2837
> Project: Ranger
>  Issue Type: Sub-task
>  Components: Ranger
>Reporter: Bhavik Patel
>Assignee: Bhavik Patel
>Priority: Major
> Attachments: RANGER-2837-Add-missing-dependencies-in-assembly.patch
>
>
> When we enabled audit source as Elasticsearch, then plugins are not able to 
> write the logs to ES service as in packaging required dependencies were not 
> added.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (RANGER-2836) Make Elasticsearch port and protocol properties configurable

2020-05-31 Thread Bhavik Patel (Jira) 


[ 
https://issues.apache.org/jira/browse/RANGER-2836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17120472#comment-17120472
 ] 

Bhavik Patel commented on RANGER-2836:
--

Commit link of the master branch: 
https://github.com/apache/ranger/commit/d722786c8914e762ab1334d245855e0e440af5fe

> Make Elasticsearch port and protocol properties configurable
> 
>
> Key: RANGER-2836
> URL: https://issues.apache.org/jira/browse/RANGER-2836
> Project: Ranger
>  Issue Type: Bug
>  Components: audit, Ranger
>Reporter: Bhavik Patel
>Assignee: Bhavik Patel
>Priority: Major
> Attachments: 
> RANGER-2836-Make-Elasticsearch-port-and-protocol-pro.patch
>
>
> Make Elasticsearch port and protocol properties configurable



--
This message was sent by Atlassian Jira
(v8.3.4#803005)