[
https://issues.apache.org/jira/browse/SENTRY-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16017498#comment-16017498
]
kalyan kumar kalvagadda commented on SENTRY-1771:
-------------------------------------------------
*Suggestion on potential solution*
[~lina.li]
I was thinking about this issue and see a simple fix that can solve this issue.
I'm not sure if you have one. Just see if my input would help you.
SentryAuthorizationInfo periodically runs to get the updates from sentry
servers.
It calls update() method which underneath sends a request to sentry server and
updates. I think the issue is that SentryAuthorizationInfo is scheduled to run
before it
received update response for the previous request. This issue can be solved by
making SentryUpdater::getUpdates synchronized and making
SentryAuthorizationInfo authzInfo, volatile.
That way we can serialize the update requests from sentry plug-in and also make
sure that each request is made based on the update received by previous
request. This
would address below issues
1. There will not be multiple outsanding requests.
2. There will be only one full snap shot request
> HDFS client requests full permission update multiple times
> ----------------------------------------------------------
>
> Key: SENTRY-1771
> URL: https://issues.apache.org/jira/browse/SENTRY-1771
> Project: Sentry
> Issue Type: Sub-task
> Components: Sentry
> Affects Versions: sentry-ha-redesign
> Reporter: Na Li
> Assignee: Na Li
> Priority: Critical
> Labels: security
>
> When running performance test, we found there are OOM, and the memory dump
> shows that there are 6 requests from HDFS to get full update for permission
> when OOM happens.
> Server sends back full update if the sequence number from client is 1 at
> DBUpdateForwarder.getAllUpdatesFrom()
> Example of call stack at the Sentry server is
> 166,999K (17.4%) (org.datanucleus.ExecutionContextThreadedImpl)
> <-- Java Local (org.datanucleus.ExecutionContextThreadedImpl)
> [@c01196f0,@c07dbb10]
> 99,019K (10.3%) (j.u.HashSet, j.u.HashMap)
> <--
> org.datanucleus.ExecutionContextThreadedImpl.reachabilityPersistedIds <--
> Java Local (org.datanucleus.ExecutionContextThreadedImpl)
> [@c01196f0,@c07dbb10]
> 85,196K (8.9%) (org.datanucleus.identity.OIDImpl)
> <-- {j.u.HashSet} <--
> org.datanucleus.ExecutionContextThreadedImpl.reachabilityPersistedIds <--
> Java Local (org.datanucleus.ExecutionContextThreadedImpl)
> [@c01196f0,@c07dbb10]
> 64,606K (6.7%)
> <-- org.datanucleus.identity.OIDImpl.toString <-- {j.u.HashSet} <--
> org.datanucleus.ExecutionContextThreadedImpl.reachabilityPersistedIds <--
> Java Local (org.datanucleus.ExecutionContextThreadedImpl)
> [@c01196f0,@c07dbb10]
> GC root stack trace:
>
> org.datanucleus.store.rdbms.scostore.JoinSetStore.iterator(JoinSetStore.java:914)
> org.datanucleus.store.types.backed.Set.loadFromStore(Set.java:323)
> org.datanucleus.store.types.backed.Set.initialise(Set.java:272)
> org.datanucleus.store.types.SCOUtils.createSCOWrapper(SCOUtils.java:256)
> org.datanucleus.store.types.SCOUtils.newSCOInstance(SCOUtils.java:142)
>
> org.datanucleus.store.rdbms.mapping.java.AbstractContainerMapping.replaceFieldWithWrapper(AbstractContainerMapping.java:399)
>
> org.datanucleus.store.rdbms.mapping.java.AbstractContainerMapping.postFetch(AbstractContainerMapping.java:417)
>
> org.datanucleus.store.rdbms.request.FetchRequest.execute(FetchRequest.java:420)
>
> org.datanucleus.store.rdbms.RDBMSPersistenceHandler.fetchObject(RDBMSPersistenceHandler.java:324)
>
> org.datanucleus.state.AbstractStateManager.loadFieldsFromDatastore(AbstractStateManager.java:1122)
>
> org.datanucleus.state.JDOStateManager.loadUnloadedFieldsInFetchPlan(JDOStateManager.java:3000)
>
> org.datanucleus.state.AbstractStateManager.loadFieldsInFetchPlan(AbstractStateManager.java:1064)
>
> org.datanucleus.ExecutionContextImpl.performDetachAllOnTxnEndPreparation(ExecutionContextImpl.java:4574)
>
> org.datanucleus.ExecutionContextImpl.preCommit(ExecutionContextImpl.java:4259)
>
> org.datanucleus.ExecutionContextImpl.transactionPreCommit(ExecutionContextImpl.java:654)
>
> org.datanucleus.TransactionImpl.internalPreCommit(TransactionImpl.java:379)
> org.datanucleus.TransactionImpl.commit(TransactionImpl.java:268)
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:98)
>
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransaction(TransactionManager.java:113)
>
> org.apache.sentry.provider.db.service.persistent.SentryStore.retrieveFullPermssionsImage(SentryStore.java:2225)
>
> org.apache.sentry.hdfs.PermImageRetriever.retrieveFullImage(PermImageRetriever.java:56)
>
> org.apache.sentry.hdfs.PermImageRetriever.retrieveFullImage(PermImageRetriever.java:40)
>
> org.apache.sentry.hdfs.DBUpdateForwarder.getAllUpdatesFrom(DBUpdateForwarder.java:85)
>
> org.apache.sentry.hdfs.SentryPlugin.getAllPermsUpdatesFrom(SentryPlugin.java:178)
>
> org.apache.sentry.hdfs.SentryHDFSServiceProcessor.get_all_authz_updates_from(SentryHDFSServiceProcessor.java:48)
>
> org.apache.sentry.hdfs.service.thrift.SentryHDFSService$Processor$get_all_authz_updates_from.getResult(SentryHDFSService.java:403)
>
> org.apache.sentry.hdfs.service.thrift.SentryHDFSService$Processor$get_all_authz_updates_from.getResult(SentryHDFSService.java:388)
> org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>
> org.apache.sentry.hdfs.SentryHDFSServiceProcessorFactory$ProcessorWrapper.process(SentryHDFSServiceProcessorFactory.java:47)
>
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
>
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> java.lang.Thread.run(Thread.java:745)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
