Na Li created SENTRY-1771:
-----------------------------
Summary: HDFS client requests full permission update multiple times
Key: SENTRY-1771
URL: https://issues.apache.org/jira/browse/SENTRY-1771
Project: Sentry
Issue Type: Task
Components: Sentry
Affects Versions: sentry-ha-redesign
Reporter: Na Li
Assignee: Na Li
Priority: Critical
When running performance test, we found there are OOM, and the memory dump
shows that there are 6 requests from HDFS to get full update for permission
when OOM happens.
Server sends back full update if the sequence number from client is 1 at
DBUpdateForwarder.getAllUpdatesFrom()
Example of call stack at the Sentry server is
166,999K (17.4%) (org.datanucleus.ExecutionContextThreadedImpl)
<-- Java Local (org.datanucleus.ExecutionContextThreadedImpl)
[@c01196f0,@c07dbb10]
99,019K (10.3%) (j.u.HashSet, j.u.HashMap)
<-- org.datanucleus.ExecutionContextThreadedImpl.reachabilityPersistedIds
<-- Java Local (org.datanucleus.ExecutionContextThreadedImpl)
[@c01196f0,@c07dbb10]
85,196K (8.9%) (org.datanucleus.identity.OIDImpl)
<-- {j.u.HashSet} <--
org.datanucleus.ExecutionContextThreadedImpl.reachabilityPersistedIds <-- Java
Local (org.datanucleus.ExecutionContextThreadedImpl) [@c01196f0,@c07dbb10]
64,606K (6.7%)
<-- org.datanucleus.identity.OIDImpl.toString <-- {j.u.HashSet} <--
org.datanucleus.ExecutionContextThreadedImpl.reachabilityPersistedIds <-- Java
Local (org.datanucleus.ExecutionContextThreadedImpl) [@c01196f0,@c07dbb10]
GC root stack trace:
org.datanucleus.store.rdbms.scostore.JoinSetStore.iterator(JoinSetStore.java:914)
org.datanucleus.store.types.backed.Set.loadFromStore(Set.java:323)
org.datanucleus.store.types.backed.Set.initialise(Set.java:272)
org.datanucleus.store.types.SCOUtils.createSCOWrapper(SCOUtils.java:256)
org.datanucleus.store.types.SCOUtils.newSCOInstance(SCOUtils.java:142)
org.datanucleus.store.rdbms.mapping.java.AbstractContainerMapping.replaceFieldWithWrapper(AbstractContainerMapping.java:399)
org.datanucleus.store.rdbms.mapping.java.AbstractContainerMapping.postFetch(AbstractContainerMapping.java:417)
org.datanucleus.store.rdbms.request.FetchRequest.execute(FetchRequest.java:420)
org.datanucleus.store.rdbms.RDBMSPersistenceHandler.fetchObject(RDBMSPersistenceHandler.java:324)
org.datanucleus.state.AbstractStateManager.loadFieldsFromDatastore(AbstractStateManager.java:1122)
org.datanucleus.state.JDOStateManager.loadUnloadedFieldsInFetchPlan(JDOStateManager.java:3000)
org.datanucleus.state.AbstractStateManager.loadFieldsInFetchPlan(AbstractStateManager.java:1064)
org.datanucleus.ExecutionContextImpl.performDetachAllOnTxnEndPreparation(ExecutionContextImpl.java:4574)
org.datanucleus.ExecutionContextImpl.preCommit(ExecutionContextImpl.java:4259)
org.datanucleus.ExecutionContextImpl.transactionPreCommit(ExecutionContextImpl.java:654)
org.datanucleus.TransactionImpl.internalPreCommit(TransactionImpl.java:379)
org.datanucleus.TransactionImpl.commit(TransactionImpl.java:268)
org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:98)
org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransaction(TransactionManager.java:113)
org.apache.sentry.provider.db.service.persistent.SentryStore.retrieveFullPermssionsImage(SentryStore.java:2225)
org.apache.sentry.hdfs.PermImageRetriever.retrieveFullImage(PermImageRetriever.java:56)
org.apache.sentry.hdfs.PermImageRetriever.retrieveFullImage(PermImageRetriever.java:40)
org.apache.sentry.hdfs.DBUpdateForwarder.getAllUpdatesFrom(DBUpdateForwarder.java:85)
org.apache.sentry.hdfs.SentryPlugin.getAllPermsUpdatesFrom(SentryPlugin.java:178)
org.apache.sentry.hdfs.SentryHDFSServiceProcessor.get_all_authz_updates_from(SentryHDFSServiceProcessor.java:48)
org.apache.sentry.hdfs.service.thrift.SentryHDFSService$Processor$get_all_authz_updates_from.getResult(SentryHDFSService.java:403)
org.apache.sentry.hdfs.service.thrift.SentryHDFSService$Processor$get_all_authz_updates_from.getResult(SentryHDFSService.java:388)
org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
org.apache.sentry.hdfs.SentryHDFSServiceProcessorFactory$ProcessorWrapper.process(SentryHDFSServiceProcessorFactory.java:47)
org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
java.lang.Thread.run(Thread.java:745)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
