[
https://issues.apache.org/jira/browse/RANGER-5453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
StarBoy1005 updated RANGER-5453:
--------------------------------
Description:
Once the Ranger service contains a large number of policies and a user is
associated with many of them, deleting that user can cause threads to be
blocked for a long time and may prevent the user from being deleted
successfully.
For example I created 30.000 policies in my ranger, now I delete the user from
web ui, then got a 500 resp after about 60s.
*1.png*
*blocked thread:*
"http-nio-6080-exec-8" #54 daemon prio=5 os_prio=0 cpu=45805.69ms
elapsed=87423.32s tid=0x00007f87287f9800 nid=0xa4475 waiting for monitor entry
[0x00007f86acad3000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.ranger.biz.XUserMgr.deleteXUser(XUserMgr.java:2222)
- waiting to lock <0x00000000c1f35cf8> (a
org.apache.ranger.biz.XUserMgr)
at
org.apache.ranger.biz.XUserMgr$$FastClassBySpringCGLIB$$57c6d473.invoke(<generated>)
at
org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at
org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:689)
at
org.apache.ranger.biz.XUserMgr$$EnhancerBySpringCGLIB$$30f8511d.deleteXUser(<generated>)
at
org.apache.ranger.rest.XUserREST.deleteSingleUserByUserId(XUserREST.java:1279)
at
org.apache.ranger.rest.XUserREST$$FastClassBySpringCGLIB$$b2a65360.invoke(<generated>)
at
org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at
org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:783)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at
org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753)
at
org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:61)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at
org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753)
at
org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123)
at
org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388)
at
org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at
org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753)
at
org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:698)
at
org.apache.ranger.rest.XUserREST$$EnhancerBySpringCGLIB$$ce896aa1.deleteSingleUserByUserId(<generated>)
*runnable thread:*
"http-nio-6080-exec-1" #47 daemon prio=5 os_prio=0 cpu=6512837.33ms
elapsed=87423.32s tid=0x00007f8728823800 nid=0xa446e runnable
[0x00007f86ad1da000]
java.lang.Thread.State: RUNNABLE
at java.lang.System.identityHashCode(Native Method)
at java.util.IdentityHashMap.hash(IdentityHashMap.java:297)
at java.util.IdentityHashMap.put(IdentityHashMap.java:428)
at
org.eclipse.persistence.internal.sessions.UnitOfWorkChangeSet.addObjectChangeSetForIdentity(UnitOfWorkChangeSet.java:275)
at
org.eclipse.persistence.internal.descriptors.ObjectBuilder.createObjectChangeSet(ObjectBuilder.java:2875)
at
org.eclipse.persistence.descriptors.changetracking.DeferredChangeDetectionPolicy.createObjectChangeSetThroughComparison(DeferredChangeDetectionPolicy.java:155)
at
org.eclipse.persistence.descriptors.changetracking.DeferredChangeDetectionPolicy.createObjectChangeSet(DeferredChangeDetectionPolicy.java:146)
at
org.eclipse.persistence.descriptors.changetracking.DeferredChangeDetectionPolicy.calculateChanges(DeferredChangeDetectionPolicy.java:91)
at
org.eclipse.persistence.descriptors.changetracking.DeferredChangeDetectionPolicy.calculateChangesForExistingObject(DeferredChangeDetectionPolicy.java:56)
at
org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.calculateChanges(UnitOfWorkImpl.java:664)
at
org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.writeChanges(RepeatableWriteUnitOfWork.java:438)
at
org.eclipse.persistence.internal.jpa.EntityManagerImpl.flush(EntityManagerImpl.java:863)
at
org.eclipse.persistence.internal.jpa.QueryImpl.performPreQueryFlush(QueryImpl.java:963)
at
org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:207)
at
org.eclipse.persistence.internal.jpa.QueryImpl.getResultList(QueryImpl.java:469)
at
org.apache.ranger.db.XXPolicyRefGroupDao.findUpdatedGroupNamesByPolicy(XXPolicyRefGroupDao.java:73)
at
org.apache.ranger.biz.RangerPolicyRetriever$RetrieverContext.<init>(RangerPolicyRetriever.java:506)
at
org.apache.ranger.biz.RangerPolicyRetriever.getPolicy(RangerPolicyRetriever.java:259)
at
org.apache.ranger.biz.RangerPolicyRetriever.getPolicy(RangerPolicyRetriever.java:233)
at
org.apache.ranger.service.RangerPolicyService.populateViewBean(RangerPolicyService.java:143)
at
org.apache.ranger.service.RangerPolicyService.populateViewBean(RangerPolicyService.java:55)
at
org.apache.ranger.service.RangerBaseModelService.postUpdate(RangerBaseModelService.java:256)
at
org.apache.ranger.service.RangerBaseModelService.update(RangerBaseModelService.java:251)
at
org.apache.ranger.biz.ServiceDBStore.updatePolicy(ServiceDBStore.java:2212)
at org.apache.ranger.biz.XUserMgr.deleteXUser(XUserMgr.java:2346)
- locked <0x00000000c1f35cf8> (a org.apache.ranger.biz.XUserMgr)
at
org.apache.ranger.biz.XUserMgr$$FastClassBySpringCGLIB$$57c6d473.invoke(<generated>)
at
org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
at
org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:689)
at
org.apache.ranger.biz.XUserMgr$$EnhancerBySpringCGLIB$$30f8511d.deleteXUser(<generated>)
at
org.apache.ranger.rest.XUserREST.deleteSingleUserByUserId(XUserREST.java:1279)
at
org.apache.ranger.rest.XUserREST$$FastClassBySpringCGLIB$$b2a65360.invoke(<generated>)
at
org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
I suspect that the large number of policies is causing excessive database
queries in this loop, which not only makes the operation very time‑consuming
but also leads to OOM issues in the Ranger admin.
2.png
Pls share some solution in this situation, the number of policies can't reduce
now.
was:
Once the Ranger service contains a large number of policies and a user is
associated with many of them, deleting that user can cause threads to be
blocked for a long time and may prevent the user from being deleted
successfully.
For example I created 30.000 policies in my ranger, now I delete the user from
web ui, then got a 500 resp after about 60s.
jstack:
> Once the Ranger service contains a large number of policies and a user is
> associated with many of them, deleting that user can cause threads to be
> blocked for a long time and may prevent the user from being deleted
> successfully.
> ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>
> Key: RANGER-5453
> URL: https://issues.apache.org/jira/browse/RANGER-5453
> Project: Ranger
> Issue Type: Improvement
> Components: admin
> Affects Versions: 2.3.0
> Reporter: StarBoy1005
> Priority: Major
> Attachments: 1.png
>
>
> Once the Ranger service contains a large number of policies and a user is
> associated with many of them, deleting that user can cause threads to be
> blocked for a long time and may prevent the user from being deleted
> successfully.
> For example I created 30.000 policies in my ranger, now I delete the user
> from web ui, then got a 500 resp after about 60s.
> *1.png*
> *blocked thread:*
> "http-nio-6080-exec-8" #54 daemon prio=5 os_prio=0 cpu=45805.69ms
> elapsed=87423.32s tid=0x00007f87287f9800 nid=0xa4475 waiting for monitor
> entry [0x00007f86acad3000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at org.apache.ranger.biz.XUserMgr.deleteXUser(XUserMgr.java:2222)
> - waiting to lock <0x00000000c1f35cf8> (a
> org.apache.ranger.biz.XUserMgr)
> at
> org.apache.ranger.biz.XUserMgr$$FastClassBySpringCGLIB$$57c6d473.invoke(<generated>)
> at
> org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
> at
> org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:689)
> at
> org.apache.ranger.biz.XUserMgr$$EnhancerBySpringCGLIB$$30f8511d.deleteXUser(<generated>)
> at
> org.apache.ranger.rest.XUserREST.deleteSingleUserByUserId(XUserREST.java:1279)
> at
> org.apache.ranger.rest.XUserREST$$FastClassBySpringCGLIB$$b2a65360.invoke(<generated>)
> at
> org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
> at
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:783)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
> at
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753)
> at
> org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:61)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
> at
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753)
> at
> org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123)
> at
> org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388)
> at
> org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
> at
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753)
> at
> org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:698)
> at
> org.apache.ranger.rest.XUserREST$$EnhancerBySpringCGLIB$$ce896aa1.deleteSingleUserByUserId(<generated>)
> *runnable thread:*
> "http-nio-6080-exec-1" #47 daemon prio=5 os_prio=0 cpu=6512837.33ms
> elapsed=87423.32s tid=0x00007f8728823800 nid=0xa446e runnable
> [0x00007f86ad1da000]
> java.lang.Thread.State: RUNNABLE
> at java.lang.System.identityHashCode(Native Method)
> at java.util.IdentityHashMap.hash(IdentityHashMap.java:297)
> at java.util.IdentityHashMap.put(IdentityHashMap.java:428)
> at
> org.eclipse.persistence.internal.sessions.UnitOfWorkChangeSet.addObjectChangeSetForIdentity(UnitOfWorkChangeSet.java:275)
> at
> org.eclipse.persistence.internal.descriptors.ObjectBuilder.createObjectChangeSet(ObjectBuilder.java:2875)
> at
> org.eclipse.persistence.descriptors.changetracking.DeferredChangeDetectionPolicy.createObjectChangeSetThroughComparison(DeferredChangeDetectionPolicy.java:155)
> at
> org.eclipse.persistence.descriptors.changetracking.DeferredChangeDetectionPolicy.createObjectChangeSet(DeferredChangeDetectionPolicy.java:146)
> at
> org.eclipse.persistence.descriptors.changetracking.DeferredChangeDetectionPolicy.calculateChanges(DeferredChangeDetectionPolicy.java:91)
> at
> org.eclipse.persistence.descriptors.changetracking.DeferredChangeDetectionPolicy.calculateChangesForExistingObject(DeferredChangeDetectionPolicy.java:56)
> at
> org.eclipse.persistence.internal.sessions.UnitOfWorkImpl.calculateChanges(UnitOfWorkImpl.java:664)
> at
> org.eclipse.persistence.internal.sessions.RepeatableWriteUnitOfWork.writeChanges(RepeatableWriteUnitOfWork.java:438)
> at
> org.eclipse.persistence.internal.jpa.EntityManagerImpl.flush(EntityManagerImpl.java:863)
> at
> org.eclipse.persistence.internal.jpa.QueryImpl.performPreQueryFlush(QueryImpl.java:963)
> at
> org.eclipse.persistence.internal.jpa.QueryImpl.executeReadQuery(QueryImpl.java:207)
> at
> org.eclipse.persistence.internal.jpa.QueryImpl.getResultList(QueryImpl.java:469)
> at
> org.apache.ranger.db.XXPolicyRefGroupDao.findUpdatedGroupNamesByPolicy(XXPolicyRefGroupDao.java:73)
> at
> org.apache.ranger.biz.RangerPolicyRetriever$RetrieverContext.<init>(RangerPolicyRetriever.java:506)
> at
> org.apache.ranger.biz.RangerPolicyRetriever.getPolicy(RangerPolicyRetriever.java:259)
> at
> org.apache.ranger.biz.RangerPolicyRetriever.getPolicy(RangerPolicyRetriever.java:233)
> at
> org.apache.ranger.service.RangerPolicyService.populateViewBean(RangerPolicyService.java:143)
> at
> org.apache.ranger.service.RangerPolicyService.populateViewBean(RangerPolicyService.java:55)
> at
> org.apache.ranger.service.RangerBaseModelService.postUpdate(RangerBaseModelService.java:256)
> at
> org.apache.ranger.service.RangerBaseModelService.update(RangerBaseModelService.java:251)
> at
> org.apache.ranger.biz.ServiceDBStore.updatePolicy(ServiceDBStore.java:2212)
> at org.apache.ranger.biz.XUserMgr.deleteXUser(XUserMgr.java:2346)
> - locked <0x00000000c1f35cf8> (a org.apache.ranger.biz.XUserMgr)
> at
> org.apache.ranger.biz.XUserMgr$$FastClassBySpringCGLIB$$57c6d473.invoke(<generated>)
> at
> org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
> at
> org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:689)
> at
> org.apache.ranger.biz.XUserMgr$$EnhancerBySpringCGLIB$$30f8511d.deleteXUser(<generated>)
> at
> org.apache.ranger.rest.XUserREST.deleteSingleUserByUserId(XUserREST.java:1279)
> at
> org.apache.ranger.rest.XUserREST$$FastClassBySpringCGLIB$$b2a65360.invoke(<generated>)
> at
> org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)
>
> I suspect that the large number of policies is causing excessive database
> queries in this loop, which not only makes the operation very time‑consuming
> but also leads to OOM issues in the Ranger admin.
> 2.png
>
> Pls share some solution in this situation, the number of policies can't
> reduce now.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
