[
https://issues.apache.org/jira/browse/SENTRY-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023984#comment-16023984
]
Na Li commented on SENTRY-1771:
-------------------------------
in v2, there are 3 types of changes:
1. Sync SentryAuthorizationInfo.update(). This is to avoid multiple HDFS
requests at the same time
2. Add logging in DBUpdateForwarder.getAllUpdatesFrom(). INFO level messages
are logged when getting full snapshot. This makes it easy to catch if we are
getting full snapshot more than expected
3. Fix the bug that causes sentry server returns empty update to client when
HDFS just starts and there is no changes in perm/path change table.
> HDFS client requests full permission update multiple times
> ----------------------------------------------------------
>
> Key: SENTRY-1771
> URL: https://issues.apache.org/jira/browse/SENTRY-1771
> Project: Sentry
> Issue Type: Sub-task
> Components: Sentry
> Affects Versions: sentry-ha-redesign
> Reporter: Na Li
> Assignee: Na Li
> Labels: hfs
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1771.001-sentry-ha-redesign.patch,
> SENTRY-1771.002-sentry-ha-redesign.patch
>
>
> When running performance test, we found there are OOM, and the memory dump
> shows that there are 6 requests from HDFS to get full update for permission
> when OOM happens.
> Server sends back full update if the sequence number from client is 1 at
> DBUpdateForwarder.getAllUpdatesFrom()
> Example of call stack at the Sentry server is
> {code}
> 166,999K (17.4%) (org.datanucleus.ExecutionContextThreadedImpl)
> <-- Java Local (org.datanucleus.ExecutionContextThreadedImpl)
> [@c01196f0,@c07dbb10]
> 99,019K (10.3%) (j.u.HashSet, j.u.HashMap)
> <--
> org.datanucleus.ExecutionContextThreadedImpl.reachabilityPersistedIds <--
> Java Local (org.datanucleus.ExecutionContextThreadedImpl)
> [@c01196f0,@c07dbb10]
> 85,196K (8.9%) (org.datanucleus.identity.OIDImpl)
> <-- {j.u.HashSet} <--
> org.datanucleus.ExecutionContextThreadedImpl.reachabilityPersistedIds <--
> Java Local (org.datanucleus.ExecutionContextThreadedImpl)
> [@c01196f0,@c07dbb10]
> 64,606K (6.7%)
> <-- org.datanucleus.identity.OIDImpl.toString <-- {j.u.HashSet} <--
> org.datanucleus.ExecutionContextThreadedImpl.reachabilityPersistedIds <--
> Java Local (org.datanucleus.ExecutionContextThreadedImpl)
> [@c01196f0,@c07dbb10]
> GC root stack trace:
>
> org.datanucleus.store.rdbms.scostore.JoinSetStore.iterator(JoinSetStore.java:914)
> org.datanucleus.store.types.backed.Set.loadFromStore(Set.java:323)
> org.datanucleus.store.types.backed.Set.initialise(Set.java:272)
> org.datanucleus.store.types.SCOUtils.createSCOWrapper(SCOUtils.java:256)
> org.datanucleus.store.types.SCOUtils.newSCOInstance(SCOUtils.java:142)
>
> org.datanucleus.store.rdbms.mapping.java.AbstractContainerMapping.replaceFieldWithWrapper(AbstractContainerMapping.java:399)
>
> org.datanucleus.store.rdbms.mapping.java.AbstractContainerMapping.postFetch(AbstractContainerMapping.java:417)
>
> org.datanucleus.store.rdbms.request.FetchRequest.execute(FetchRequest.java:420)
>
> org.datanucleus.store.rdbms.RDBMSPersistenceHandler.fetchObject(RDBMSPersistenceHandler.java:324)
>
> org.datanucleus.state.AbstractStateManager.loadFieldsFromDatastore(AbstractStateManager.java:1122)
>
> org.datanucleus.state.JDOStateManager.loadUnloadedFieldsInFetchPlan(JDOStateManager.java:3000)
>
> org.datanucleus.state.AbstractStateManager.loadFieldsInFetchPlan(AbstractStateManager.java:1064)
>
> org.datanucleus.ExecutionContextImpl.performDetachAllOnTxnEndPreparation(ExecutionContextImpl.java:4574)
>
> org.datanucleus.ExecutionContextImpl.preCommit(ExecutionContextImpl.java:4259)
>
> org.datanucleus.ExecutionContextImpl.transactionPreCommit(ExecutionContextImpl.java:654)
>
> org.datanucleus.TransactionImpl.internalPreCommit(TransactionImpl.java:379)
> org.datanucleus.TransactionImpl.commit(TransactionImpl.java:268)
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:98)
>
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransaction(TransactionManager.java:113)
>
> org.apache.sentry.provider.db.service.persistent.SentryStore.retrieveFullPermssionsImage(SentryStore.java:2225)
>
> org.apache.sentry.hdfs.PermImageRetriever.retrieveFullImage(PermImageRetriever.java:56)
>
> org.apache.sentry.hdfs.PermImageRetriever.retrieveFullImage(PermImageRetriever.java:40)
>
> org.apache.sentry.hdfs.DBUpdateForwarder.getAllUpdatesFrom(DBUpdateForwarder.java:85)
>
> org.apache.sentry.hdfs.SentryPlugin.getAllPermsUpdatesFrom(SentryPlugin.java:178)
>
> org.apache.sentry.hdfs.SentryHDFSServiceProcessor.get_all_authz_updates_from(SentryHDFSServiceProcessor.java:48)
>
> org.apache.sentry.hdfs.service.thrift.SentryHDFSService$Processor$get_all_authz_updates_from.getResult(SentryHDFSService.java:403)
>
> org.apache.sentry.hdfs.service.thrift.SentryHDFSService$Processor$get_all_authz_updates_from.getResult(SentryHDFSService.java:388)
> org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>
> org.apache.sentry.hdfs.SentryHDFSServiceProcessorFactory$ProcessorWrapper.process(SentryHDFSServiceProcessorFactory.java:47)
>
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
>
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
