[
https://issues.apache.org/jira/browse/SENTRY-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024048#comment-16024048
]
Na Li commented on SENTRY-1771:
-------------------------------
"So the old code returned an emty list when there are no updates. Why do you
think that it is a problem now?"
I think it is a problem because when there are permission info in MSentryGroup
and MSentryPrivilege, but no entry in MSentryPermChange (It could happen when
use upgrade Sentry), HDFS should get permission update to generate ACL in its
first request. However, the existing behavior returns no update until
MSentryPermChange has entries.
> HDFS client requests full permission update multiple times
> ----------------------------------------------------------
>
> Key: SENTRY-1771
> URL: https://issues.apache.org/jira/browse/SENTRY-1771
> Project: Sentry
> Issue Type: Sub-task
> Components: Sentry
> Affects Versions: sentry-ha-redesign
> Reporter: Na Li
> Assignee: Na Li
> Labels: hfs
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1771.001-sentry-ha-redesign.patch,
> SENTRY-1771.002-sentry-ha-redesign.patch
>
>
> When running performance test, we found there are OOM, and the memory dump
> shows that there are 6 requests from HDFS to get full update for permission
> when OOM happens.
> Server sends back full update if the sequence number from client is 1 at
> DBUpdateForwarder.getAllUpdatesFrom()
> Example of call stack at the Sentry server is
> {code}
> 166,999K (17.4%) (org.datanucleus.ExecutionContextThreadedImpl)
> <-- Java Local (org.datanucleus.ExecutionContextThreadedImpl)
> [@c01196f0,@c07dbb10]
> 99,019K (10.3%) (j.u.HashSet, j.u.HashMap)
> <--
> org.datanucleus.ExecutionContextThreadedImpl.reachabilityPersistedIds <--
> Java Local (org.datanucleus.ExecutionContextThreadedImpl)
> [@c01196f0,@c07dbb10]
> 85,196K (8.9%) (org.datanucleus.identity.OIDImpl)
> <-- {j.u.HashSet} <--
> org.datanucleus.ExecutionContextThreadedImpl.reachabilityPersistedIds <--
> Java Local (org.datanucleus.ExecutionContextThreadedImpl)
> [@c01196f0,@c07dbb10]
> 64,606K (6.7%)
> <-- org.datanucleus.identity.OIDImpl.toString <-- {j.u.HashSet} <--
> org.datanucleus.ExecutionContextThreadedImpl.reachabilityPersistedIds <--
> Java Local (org.datanucleus.ExecutionContextThreadedImpl)
> [@c01196f0,@c07dbb10]
> GC root stack trace:
>
> org.datanucleus.store.rdbms.scostore.JoinSetStore.iterator(JoinSetStore.java:914)
> org.datanucleus.store.types.backed.Set.loadFromStore(Set.java:323)
> org.datanucleus.store.types.backed.Set.initialise(Set.java:272)
> org.datanucleus.store.types.SCOUtils.createSCOWrapper(SCOUtils.java:256)
> org.datanucleus.store.types.SCOUtils.newSCOInstance(SCOUtils.java:142)
>
> org.datanucleus.store.rdbms.mapping.java.AbstractContainerMapping.replaceFieldWithWrapper(AbstractContainerMapping.java:399)
>
> org.datanucleus.store.rdbms.mapping.java.AbstractContainerMapping.postFetch(AbstractContainerMapping.java:417)
>
> org.datanucleus.store.rdbms.request.FetchRequest.execute(FetchRequest.java:420)
>
> org.datanucleus.store.rdbms.RDBMSPersistenceHandler.fetchObject(RDBMSPersistenceHandler.java:324)
>
> org.datanucleus.state.AbstractStateManager.loadFieldsFromDatastore(AbstractStateManager.java:1122)
>
> org.datanucleus.state.JDOStateManager.loadUnloadedFieldsInFetchPlan(JDOStateManager.java:3000)
>
> org.datanucleus.state.AbstractStateManager.loadFieldsInFetchPlan(AbstractStateManager.java:1064)
>
> org.datanucleus.ExecutionContextImpl.performDetachAllOnTxnEndPreparation(ExecutionContextImpl.java:4574)
>
> org.datanucleus.ExecutionContextImpl.preCommit(ExecutionContextImpl.java:4259)
>
> org.datanucleus.ExecutionContextImpl.transactionPreCommit(ExecutionContextImpl.java:654)
>
> org.datanucleus.TransactionImpl.internalPreCommit(TransactionImpl.java:379)
> org.datanucleus.TransactionImpl.commit(TransactionImpl.java:268)
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:98)
>
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransaction(TransactionManager.java:113)
>
> org.apache.sentry.provider.db.service.persistent.SentryStore.retrieveFullPermssionsImage(SentryStore.java:2225)
>
> org.apache.sentry.hdfs.PermImageRetriever.retrieveFullImage(PermImageRetriever.java:56)
>
> org.apache.sentry.hdfs.PermImageRetriever.retrieveFullImage(PermImageRetriever.java:40)
>
> org.apache.sentry.hdfs.DBUpdateForwarder.getAllUpdatesFrom(DBUpdateForwarder.java:85)
>
> org.apache.sentry.hdfs.SentryPlugin.getAllPermsUpdatesFrom(SentryPlugin.java:178)
>
> org.apache.sentry.hdfs.SentryHDFSServiceProcessor.get_all_authz_updates_from(SentryHDFSServiceProcessor.java:48)
>
> org.apache.sentry.hdfs.service.thrift.SentryHDFSService$Processor$get_all_authz_updates_from.getResult(SentryHDFSService.java:403)
>
> org.apache.sentry.hdfs.service.thrift.SentryHDFSService$Processor$get_all_authz_updates_from.getResult(SentryHDFSService.java:388)
> org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>
> org.apache.sentry.hdfs.SentryHDFSServiceProcessorFactory$ProcessorWrapper.process(SentryHDFSServiceProcessorFactory.java:47)
>
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
>
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
