[
https://issues.apache.org/jira/browse/SENTRY-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023915#comment-16023915
]
Alexander Kolbasov commented on SENTRY-1771:
--------------------------------------------
Are you really sure that we have concurrent sends of snapshots? The conclusion
comes from the jxray output of the heap dump analizer which may show the same
thread responsible for multiple memory consumption issues.
> HDFS client requests full permission update multiple times
> ----------------------------------------------------------
>
> Key: SENTRY-1771
> URL: https://issues.apache.org/jira/browse/SENTRY-1771
> Project: Sentry
> Issue Type: Sub-task
> Components: Sentry
> Affects Versions: sentry-ha-redesign
> Reporter: Na Li
> Assignee: Na Li
> Labels: hfs
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1771.001-sentry-ha-redesign.patch,
> SENTRY-1771.002-sentry-ha-redesign.patch
>
>
> When running performance test, we found there are OOM, and the memory dump
> shows that there are 6 requests from HDFS to get full update for permission
> when OOM happens.
> Server sends back full update if the sequence number from client is 1 at
> DBUpdateForwarder.getAllUpdatesFrom()
> Example of call stack at the Sentry server is
> {code}
> 166,999K (17.4%) (org.datanucleus.ExecutionContextThreadedImpl)
> <-- Java Local (org.datanucleus.ExecutionContextThreadedImpl)
> [@c01196f0,@c07dbb10]
> 99,019K (10.3%) (j.u.HashSet, j.u.HashMap)
> <--
> org.datanucleus.ExecutionContextThreadedImpl.reachabilityPersistedIds <--
> Java Local (org.datanucleus.ExecutionContextThreadedImpl)
> [@c01196f0,@c07dbb10]
> 85,196K (8.9%) (org.datanucleus.identity.OIDImpl)
> <-- {j.u.HashSet} <--
> org.datanucleus.ExecutionContextThreadedImpl.reachabilityPersistedIds <--
> Java Local (org.datanucleus.ExecutionContextThreadedImpl)
> [@c01196f0,@c07dbb10]
> 64,606K (6.7%)
> <-- org.datanucleus.identity.OIDImpl.toString <-- {j.u.HashSet} <--
> org.datanucleus.ExecutionContextThreadedImpl.reachabilityPersistedIds <--
> Java Local (org.datanucleus.ExecutionContextThreadedImpl)
> [@c01196f0,@c07dbb10]
> GC root stack trace:
>
> org.datanucleus.store.rdbms.scostore.JoinSetStore.iterator(JoinSetStore.java:914)
> org.datanucleus.store.types.backed.Set.loadFromStore(Set.java:323)
> org.datanucleus.store.types.backed.Set.initialise(Set.java:272)
> org.datanucleus.store.types.SCOUtils.createSCOWrapper(SCOUtils.java:256)
> org.datanucleus.store.types.SCOUtils.newSCOInstance(SCOUtils.java:142)
>
> org.datanucleus.store.rdbms.mapping.java.AbstractContainerMapping.replaceFieldWithWrapper(AbstractContainerMapping.java:399)
>
> org.datanucleus.store.rdbms.mapping.java.AbstractContainerMapping.postFetch(AbstractContainerMapping.java:417)
>
> org.datanucleus.store.rdbms.request.FetchRequest.execute(FetchRequest.java:420)
>
> org.datanucleus.store.rdbms.RDBMSPersistenceHandler.fetchObject(RDBMSPersistenceHandler.java:324)
>
> org.datanucleus.state.AbstractStateManager.loadFieldsFromDatastore(AbstractStateManager.java:1122)
>
> org.datanucleus.state.JDOStateManager.loadUnloadedFieldsInFetchPlan(JDOStateManager.java:3000)
>
> org.datanucleus.state.AbstractStateManager.loadFieldsInFetchPlan(AbstractStateManager.java:1064)
>
> org.datanucleus.ExecutionContextImpl.performDetachAllOnTxnEndPreparation(ExecutionContextImpl.java:4574)
>
> org.datanucleus.ExecutionContextImpl.preCommit(ExecutionContextImpl.java:4259)
>
> org.datanucleus.ExecutionContextImpl.transactionPreCommit(ExecutionContextImpl.java:654)
>
> org.datanucleus.TransactionImpl.internalPreCommit(TransactionImpl.java:379)
> org.datanucleus.TransactionImpl.commit(TransactionImpl.java:268)
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:98)
>
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransaction(TransactionManager.java:113)
>
> org.apache.sentry.provider.db.service.persistent.SentryStore.retrieveFullPermssionsImage(SentryStore.java:2225)
>
> org.apache.sentry.hdfs.PermImageRetriever.retrieveFullImage(PermImageRetriever.java:56)
>
> org.apache.sentry.hdfs.PermImageRetriever.retrieveFullImage(PermImageRetriever.java:40)
>
> org.apache.sentry.hdfs.DBUpdateForwarder.getAllUpdatesFrom(DBUpdateForwarder.java:85)
>
> org.apache.sentry.hdfs.SentryPlugin.getAllPermsUpdatesFrom(SentryPlugin.java:178)
>
> org.apache.sentry.hdfs.SentryHDFSServiceProcessor.get_all_authz_updates_from(SentryHDFSServiceProcessor.java:48)
>
> org.apache.sentry.hdfs.service.thrift.SentryHDFSService$Processor$get_all_authz_updates_from.getResult(SentryHDFSService.java:403)
>
> org.apache.sentry.hdfs.service.thrift.SentryHDFSService$Processor$get_all_authz_updates_from.getResult(SentryHDFSService.java:388)
> org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>
> org.apache.sentry.hdfs.SentryHDFSServiceProcessorFactory$ProcessorWrapper.process(SentryHDFSServiceProcessorFactory.java:47)
>
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
>
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
