[
https://issues.apache.org/jira/browse/SENTRY-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Kolbasov updated SENTRY-1771:
---------------------------------------
Description:
When running performance test, we found there are OOM, and the memory dump
shows that there are 6 requests from HDFS to get full update for permission
when OOM happens.
Server sends back full update if the sequence number from client is 1 at
DBUpdateForwarder.getAllUpdatesFrom()
Example of call stack at the Sentry server is
{code}
166,999K (17.4%) (org.datanucleus.ExecutionContextThreadedImpl)
<-- Java Local (org.datanucleus.ExecutionContextThreadedImpl)
[@c01196f0,@c07dbb10]
99,019K (10.3%) (j.u.HashSet, j.u.HashMap)
<-- org.datanucleus.ExecutionContextThreadedImpl.reachabilityPersistedIds
<-- Java Local (org.datanucleus.ExecutionContextThreadedImpl)
[@c01196f0,@c07dbb10]
85,196K (8.9%) (org.datanucleus.identity.OIDImpl)
<-- {j.u.HashSet} <--
org.datanucleus.ExecutionContextThreadedImpl.reachabilityPersistedIds <-- Java
Local (org.datanucleus.ExecutionContextThreadedImpl) [@c01196f0,@c07dbb10]
64,606K (6.7%)
<-- org.datanucleus.identity.OIDImpl.toString <-- {j.u.HashSet} <--
org.datanucleus.ExecutionContextThreadedImpl.reachabilityPersistedIds <-- Java
Local (org.datanucleus.ExecutionContextThreadedImpl) [@c01196f0,@c07dbb10]
GC root stack trace:
org.datanucleus.store.rdbms.scostore.JoinSetStore.iterator(JoinSetStore.java:914)
org.datanucleus.store.types.backed.Set.loadFromStore(Set.java:323)
org.datanucleus.store.types.backed.Set.initialise(Set.java:272)
org.datanucleus.store.types.SCOUtils.createSCOWrapper(SCOUtils.java:256)
org.datanucleus.store.types.SCOUtils.newSCOInstance(SCOUtils.java:142)
org.datanucleus.store.rdbms.mapping.java.AbstractContainerMapping.replaceFieldWithWrapper(AbstractContainerMapping.java:399)
org.datanucleus.store.rdbms.mapping.java.AbstractContainerMapping.postFetch(AbstractContainerMapping.java:417)
org.datanucleus.store.rdbms.request.FetchRequest.execute(FetchRequest.java:420)
org.datanucleus.store.rdbms.RDBMSPersistenceHandler.fetchObject(RDBMSPersistenceHandler.java:324)
org.datanucleus.state.AbstractStateManager.loadFieldsFromDatastore(AbstractStateManager.java:1122)
org.datanucleus.state.JDOStateManager.loadUnloadedFieldsInFetchPlan(JDOStateManager.java:3000)
org.datanucleus.state.AbstractStateManager.loadFieldsInFetchPlan(AbstractStateManager.java:1064)
org.datanucleus.ExecutionContextImpl.performDetachAllOnTxnEndPreparation(ExecutionContextImpl.java:4574)
org.datanucleus.ExecutionContextImpl.preCommit(ExecutionContextImpl.java:4259)
org.datanucleus.ExecutionContextImpl.transactionPreCommit(ExecutionContextImpl.java:654)
org.datanucleus.TransactionImpl.internalPreCommit(TransactionImpl.java:379)
org.datanucleus.TransactionImpl.commit(TransactionImpl.java:268)
org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:98)
org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransaction(TransactionManager.java:113)
org.apache.sentry.provider.db.service.persistent.SentryStore.retrieveFullPermssionsImage(SentryStore.java:2225)
org.apache.sentry.hdfs.PermImageRetriever.retrieveFullImage(PermImageRetriever.java:56)
org.apache.sentry.hdfs.PermImageRetriever.retrieveFullImage(PermImageRetriever.java:40)
org.apache.sentry.hdfs.DBUpdateForwarder.getAllUpdatesFrom(DBUpdateForwarder.java:85)
org.apache.sentry.hdfs.SentryPlugin.getAllPermsUpdatesFrom(SentryPlugin.java:178)
org.apache.sentry.hdfs.SentryHDFSServiceProcessor.get_all_authz_updates_from(SentryHDFSServiceProcessor.java:48)
org.apache.sentry.hdfs.service.thrift.SentryHDFSService$Processor$get_all_authz_updates_from.getResult(SentryHDFSService.java:403)
org.apache.sentry.hdfs.service.thrift.SentryHDFSService$Processor$get_all_authz_updates_from.getResult(SentryHDFSService.java:388)
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
org.apache.sentry.hdfs.SentryHDFSServiceProcessorFactory$ProcessorWrapper.process(SentryHDFSServiceProcessorFactory.java:47)
org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
{code}
was:
When running performance test, we found there are OOM, and the memory dump
shows that there are 6 requests from HDFS to get full update for permission
when OOM happens.
Server sends back full update if the sequence number from client is 1 at
DBUpdateForwarder.getAllUpdatesFrom()
Example of call stack at the Sentry server is
166,999K (17.4%) (org.datanucleus.ExecutionContextThreadedImpl)
<-- Java Local (org.datanucleus.ExecutionContextThreadedImpl)
[@c01196f0,@c07dbb10]
99,019K (10.3%) (j.u.HashSet, j.u.HashMap)
<-- org.datanucleus.ExecutionContextThreadedImpl.reachabilityPersistedIds
<-- Java Local (org.datanucleus.ExecutionContextThreadedImpl)
[@c01196f0,@c07dbb10]
85,196K (8.9%) (org.datanucleus.identity.OIDImpl)
<-- {j.u.HashSet} <--
org.datanucleus.ExecutionContextThreadedImpl.reachabilityPersistedIds <-- Java
Local (org.datanucleus.ExecutionContextThreadedImpl) [@c01196f0,@c07dbb10]
64,606K (6.7%)
<-- org.datanucleus.identity.OIDImpl.toString <-- {j.u.HashSet} <--
org.datanucleus.ExecutionContextThreadedImpl.reachabilityPersistedIds <-- Java
Local (org.datanucleus.ExecutionContextThreadedImpl) [@c01196f0,@c07dbb10]
GC root stack trace:
org.datanucleus.store.rdbms.scostore.JoinSetStore.iterator(JoinSetStore.java:914)
org.datanucleus.store.types.backed.Set.loadFromStore(Set.java:323)
org.datanucleus.store.types.backed.Set.initialise(Set.java:272)
org.datanucleus.store.types.SCOUtils.createSCOWrapper(SCOUtils.java:256)
org.datanucleus.store.types.SCOUtils.newSCOInstance(SCOUtils.java:142)
org.datanucleus.store.rdbms.mapping.java.AbstractContainerMapping.replaceFieldWithWrapper(AbstractContainerMapping.java:399)
org.datanucleus.store.rdbms.mapping.java.AbstractContainerMapping.postFetch(AbstractContainerMapping.java:417)
org.datanucleus.store.rdbms.request.FetchRequest.execute(FetchRequest.java:420)
org.datanucleus.store.rdbms.RDBMSPersistenceHandler.fetchObject(RDBMSPersistenceHandler.java:324)
org.datanucleus.state.AbstractStateManager.loadFieldsFromDatastore(AbstractStateManager.java:1122)
org.datanucleus.state.JDOStateManager.loadUnloadedFieldsInFetchPlan(JDOStateManager.java:3000)
org.datanucleus.state.AbstractStateManager.loadFieldsInFetchPlan(AbstractStateManager.java:1064)
org.datanucleus.ExecutionContextImpl.performDetachAllOnTxnEndPreparation(ExecutionContextImpl.java:4574)
org.datanucleus.ExecutionContextImpl.preCommit(ExecutionContextImpl.java:4259)
org.datanucleus.ExecutionContextImpl.transactionPreCommit(ExecutionContextImpl.java:654)
org.datanucleus.TransactionImpl.internalPreCommit(TransactionImpl.java:379)
org.datanucleus.TransactionImpl.commit(TransactionImpl.java:268)
org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:98)
org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransaction(TransactionManager.java:113)
org.apache.sentry.provider.db.service.persistent.SentryStore.retrieveFullPermssionsImage(SentryStore.java:2225)
org.apache.sentry.hdfs.PermImageRetriever.retrieveFullImage(PermImageRetriever.java:56)
org.apache.sentry.hdfs.PermImageRetriever.retrieveFullImage(PermImageRetriever.java:40)
org.apache.sentry.hdfs.DBUpdateForwarder.getAllUpdatesFrom(DBUpdateForwarder.java:85)
org.apache.sentry.hdfs.SentryPlugin.getAllPermsUpdatesFrom(SentryPlugin.java:178)
org.apache.sentry.hdfs.SentryHDFSServiceProcessor.get_all_authz_updates_from(SentryHDFSServiceProcessor.java:48)
org.apache.sentry.hdfs.service.thrift.SentryHDFSService$Processor$get_all_authz_updates_from.getResult(SentryHDFSService.java:403)
org.apache.sentry.hdfs.service.thrift.SentryHDFSService$Processor$get_all_authz_updates_from.getResult(SentryHDFSService.java:388)
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
org.apache.sentry.hdfs.SentryHDFSServiceProcessorFactory$ProcessorWrapper.process(SentryHDFSServiceProcessorFactory.java:47)
org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
> HDFS client requests full permission update multiple times
> ----------------------------------------------------------
>
> Key: SENTRY-1771
> URL: https://issues.apache.org/jira/browse/SENTRY-1771
> Project: Sentry
> Issue Type: Sub-task
> Components: Sentry
> Affects Versions: sentry-ha-redesign
> Reporter: Na Li
> Assignee: Na Li
> Labels: hfs
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1771.001-sentry-ha-redesign.patch,
> SENTRY-1771.002-sentry-ha-redesign.patch
>
>
> When running performance test, we found there are OOM, and the memory dump
> shows that there are 6 requests from HDFS to get full update for permission
> when OOM happens.
> Server sends back full update if the sequence number from client is 1 at
> DBUpdateForwarder.getAllUpdatesFrom()
> Example of call stack at the Sentry server is
> {code}
> 166,999K (17.4%) (org.datanucleus.ExecutionContextThreadedImpl)
> <-- Java Local (org.datanucleus.ExecutionContextThreadedImpl)
> [@c01196f0,@c07dbb10]
> 99,019K (10.3%) (j.u.HashSet, j.u.HashMap)
> <--
> org.datanucleus.ExecutionContextThreadedImpl.reachabilityPersistedIds <--
> Java Local (org.datanucleus.ExecutionContextThreadedImpl)
> [@c01196f0,@c07dbb10]
> 85,196K (8.9%) (org.datanucleus.identity.OIDImpl)
> <-- {j.u.HashSet} <--
> org.datanucleus.ExecutionContextThreadedImpl.reachabilityPersistedIds <--
> Java Local (org.datanucleus.ExecutionContextThreadedImpl)
> [@c01196f0,@c07dbb10]
> 64,606K (6.7%)
> <-- org.datanucleus.identity.OIDImpl.toString <-- {j.u.HashSet} <--
> org.datanucleus.ExecutionContextThreadedImpl.reachabilityPersistedIds <--
> Java Local (org.datanucleus.ExecutionContextThreadedImpl)
> [@c01196f0,@c07dbb10]
> GC root stack trace:
>
> org.datanucleus.store.rdbms.scostore.JoinSetStore.iterator(JoinSetStore.java:914)
> org.datanucleus.store.types.backed.Set.loadFromStore(Set.java:323)
> org.datanucleus.store.types.backed.Set.initialise(Set.java:272)
> org.datanucleus.store.types.SCOUtils.createSCOWrapper(SCOUtils.java:256)
> org.datanucleus.store.types.SCOUtils.newSCOInstance(SCOUtils.java:142)
>
> org.datanucleus.store.rdbms.mapping.java.AbstractContainerMapping.replaceFieldWithWrapper(AbstractContainerMapping.java:399)
>
> org.datanucleus.store.rdbms.mapping.java.AbstractContainerMapping.postFetch(AbstractContainerMapping.java:417)
>
> org.datanucleus.store.rdbms.request.FetchRequest.execute(FetchRequest.java:420)
>
> org.datanucleus.store.rdbms.RDBMSPersistenceHandler.fetchObject(RDBMSPersistenceHandler.java:324)
>
> org.datanucleus.state.AbstractStateManager.loadFieldsFromDatastore(AbstractStateManager.java:1122)
>
> org.datanucleus.state.JDOStateManager.loadUnloadedFieldsInFetchPlan(JDOStateManager.java:3000)
>
> org.datanucleus.state.AbstractStateManager.loadFieldsInFetchPlan(AbstractStateManager.java:1064)
>
> org.datanucleus.ExecutionContextImpl.performDetachAllOnTxnEndPreparation(ExecutionContextImpl.java:4574)
>
> org.datanucleus.ExecutionContextImpl.preCommit(ExecutionContextImpl.java:4259)
>
> org.datanucleus.ExecutionContextImpl.transactionPreCommit(ExecutionContextImpl.java:654)
>
> org.datanucleus.TransactionImpl.internalPreCommit(TransactionImpl.java:379)
> org.datanucleus.TransactionImpl.commit(TransactionImpl.java:268)
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:98)
>
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransaction(TransactionManager.java:113)
>
> org.apache.sentry.provider.db.service.persistent.SentryStore.retrieveFullPermssionsImage(SentryStore.java:2225)
>
> org.apache.sentry.hdfs.PermImageRetriever.retrieveFullImage(PermImageRetriever.java:56)
>
> org.apache.sentry.hdfs.PermImageRetriever.retrieveFullImage(PermImageRetriever.java:40)
>
> org.apache.sentry.hdfs.DBUpdateForwarder.getAllUpdatesFrom(DBUpdateForwarder.java:85)
>
> org.apache.sentry.hdfs.SentryPlugin.getAllPermsUpdatesFrom(SentryPlugin.java:178)
>
> org.apache.sentry.hdfs.SentryHDFSServiceProcessor.get_all_authz_updates_from(SentryHDFSServiceProcessor.java:48)
>
> org.apache.sentry.hdfs.service.thrift.SentryHDFSService$Processor$get_all_authz_updates_from.getResult(SentryHDFSService.java:403)
>
> org.apache.sentry.hdfs.service.thrift.SentryHDFSService$Processor$get_all_authz_updates_from.getResult(SentryHDFSService.java:388)
> org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>
> org.apache.sentry.hdfs.SentryHDFSServiceProcessorFactory$ProcessorWrapper.process(SentryHDFSServiceProcessorFactory.java:47)
>
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
>
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
