[
https://issues.apache.org/jira/browse/SENTRY-1771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16026859#comment-16026859
]
Na Li commented on SENTRY-1771:
-------------------------------
Using Vamsee's cluster with patch v2 running for overnight, we can see the full
snapshot is returned at least once in a second. The reason is "Certain perm
delta is missing in SENTRY_PERM_CHANEG table! Current size of elements = 9 and
expected size = 10, from changeID: 248035. The table may get corrupted."
2017-05-26 06:18:18,142 INFO org.apache.sentry.hdfs.DBUpdateForwarder: ####
GetAllUpdatesFrom is getting full image with isDeltaAvailable:true,
isDeltaEmpty:true, reqSeqNum:245507, curSeqNum:245515
2017-05-26 06:18:19,558 INFO org.apache.sentry.hdfs.DBUpdateForwarder: ####
GetAllUpdatesFrom return full image with isDeltaAvailable:true,
isDeltaEmpty:true, reqSeqNum:245507, curSeqNum:245515
2017-05-26 06:19:26,818 INFO org.apache.sentry.hdfs.DBUpdateForwarder: ####
GetAllUpdatesFrom is getting full image with isDeltaAvailable:true,
isDeltaEmpty:true, reqSeqNum:246571, curSeqNum:246580
2017-05-26 06:19:28,545 INFO org.apache.sentry.hdfs.DBUpdateForwarder: ####
GetAllUpdatesFrom return full image with isDeltaAvailable:true,
isDeltaEmpty:true, reqSeqNum:246571, curSeqNum:246580
2017-05-26 06:20:01,795 INFO org.apache.sentry.hdfs.DBUpdateForwarder: ####
GetAllUpdatesFrom is getting full image with isDeltaAvailable:true,
isDeltaEmpty:true, reqSeqNum:247113, curSeqNum:247120
2017-05-26 06:20:03,444 INFO org.apache.sentry.hdfs.DBUpdateForwarder: ####
GetAllUpdatesFrom return full image with isDeltaAvailable:true,
isDeltaEmpty:true, reqSeqNum:247113, curSeqNum:247120
2017-05-26 06:20:51,953 INFO org.apache.sentry.hdfs.DBUpdateForwarder: ####
GetAllUpdatesFrom is getting full image with isDeltaAvailable:true,
isDeltaEmpty:true, reqSeqNum:247883, curSeqNum:247891
2017-05-26 06:20:53,288 INFO org.apache.sentry.hdfs.DBUpdateForwarder: ####
GetAllUpdatesFrom return full image with isDeltaAvailable:true,
isDeltaEmpty:true, reqSeqNum:247883, curSeqNum:247891
2017-05-26 06:21:02,136 ERROR
org.apache.sentry.provider.db.service.persistent.SentryStore: Certain perm
delta is missing in SENTRY_PERM_CHANEG table! Current size of elements = 9 and
expected size = 10, from changeID: 248035. The table may get corrupted.
2017-05-26 06:21:02,134 INFO DataNucleus.Query: Reading in results for query
"SELECT FROM org.apache.sentry.provider.db.service.model.MSentryPermChange
WHERE this.changeID == id PARAMETERS long id" since the connection used is
closing
2017-05-26 06:21:02,136 ERROR
org.apache.sentry.provider.db.service.persistent.SentryStore: Certain perm
delta is missing in SENTRY_PERM_CHANEG table! Current size of elements = 9 and
expected size = 10, from changeID: 248035. The table may get corrupted.
2017-05-26 06:21:02,136 INFO org.apache.sentry.hdfs.DBUpdateForwarder: ####
GetAllUpdatesFrom is getting full image with isDeltaAvailable:true,
isDeltaEmpty:true, reqSeqNum:248035, curSeqNum:248044
2017-05-26 06:21:03,678 INFO org.apache.sentry.hdfs.DBUpdateForwarder: ####
GetAllUpdatesFrom return full image with isDeltaAvailable:true,
isDeltaEmpty:true, reqSeqNum:248035, curSeqNum:248044
2017-05-26 06:21:07,947 INFO DataNucleus.Query: Reading in results for query
"SELECT FROM org.apache.sentry.provider.db.service.model.MSentryPermChange
WHERE this.changeID == id PARAMETERS long id" since the connection used is
closing
2017-05-26 06:21:07,950 ERROR
org.apache.sentry.provider.db.service.persistent.SentryStore: Certain perm
delta is missing in SENTRY_PERM_CHANEG table! Current size of elements = 7 and
expected size = 8, from changeID: 248122. The table may get corrupted.
2017-05-26 06:21:07,952 INFO org.apache.sentry.hdfs.DBUpdateForwarder: ####
GetAllUpdatesFrom is getting full image with isDeltaAvailable:true,
isDeltaEmpty:true, reqSeqNum:248122, curSeqNum:248129
2017-05-26 06:21:09,426 INFO org.apache.sentry.hdfs.DBUpdateForwarder: ####
GetAllUpdatesFrom return full image with isDeltaAvailable:true,
isDeltaEmpty:true, reqSeqNum:248122, curSeqNum:248129
> HDFS client requests full permission update multiple times
> ----------------------------------------------------------
>
> Key: SENTRY-1771
> URL: https://issues.apache.org/jira/browse/SENTRY-1771
> Project: Sentry
> Issue Type: Sub-task
> Components: Sentry
> Affects Versions: sentry-ha-redesign
> Reporter: Na Li
> Assignee: Na Li
> Labels: hfs
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1771.001-sentry-ha-redesign.patch,
> SENTRY-1771.002-sentry-ha-redesign.patch
>
>
> When running performance test, we found there are OOM, and the memory dump
> shows that there are 6 requests from HDFS to get full update for permission
> when OOM happens.
> Server sends back full update if the sequence number from client is 1 at
> DBUpdateForwarder.getAllUpdatesFrom()
> Example of call stack at the Sentry server is
> {code}
> 166,999K (17.4%) (org.datanucleus.ExecutionContextThreadedImpl)
> <-- Java Local (org.datanucleus.ExecutionContextThreadedImpl)
> [@c01196f0,@c07dbb10]
> 99,019K (10.3%) (j.u.HashSet, j.u.HashMap)
> <--
> org.datanucleus.ExecutionContextThreadedImpl.reachabilityPersistedIds <--
> Java Local (org.datanucleus.ExecutionContextThreadedImpl)
> [@c01196f0,@c07dbb10]
> 85,196K (8.9%) (org.datanucleus.identity.OIDImpl)
> <-- {j.u.HashSet} <--
> org.datanucleus.ExecutionContextThreadedImpl.reachabilityPersistedIds <--
> Java Local (org.datanucleus.ExecutionContextThreadedImpl)
> [@c01196f0,@c07dbb10]
> 64,606K (6.7%)
> <-- org.datanucleus.identity.OIDImpl.toString <-- {j.u.HashSet} <--
> org.datanucleus.ExecutionContextThreadedImpl.reachabilityPersistedIds <--
> Java Local (org.datanucleus.ExecutionContextThreadedImpl)
> [@c01196f0,@c07dbb10]
> GC root stack trace:
>
> org.datanucleus.store.rdbms.scostore.JoinSetStore.iterator(JoinSetStore.java:914)
> org.datanucleus.store.types.backed.Set.loadFromStore(Set.java:323)
> org.datanucleus.store.types.backed.Set.initialise(Set.java:272)
> org.datanucleus.store.types.SCOUtils.createSCOWrapper(SCOUtils.java:256)
> org.datanucleus.store.types.SCOUtils.newSCOInstance(SCOUtils.java:142)
>
> org.datanucleus.store.rdbms.mapping.java.AbstractContainerMapping.replaceFieldWithWrapper(AbstractContainerMapping.java:399)
>
> org.datanucleus.store.rdbms.mapping.java.AbstractContainerMapping.postFetch(AbstractContainerMapping.java:417)
>
> org.datanucleus.store.rdbms.request.FetchRequest.execute(FetchRequest.java:420)
>
> org.datanucleus.store.rdbms.RDBMSPersistenceHandler.fetchObject(RDBMSPersistenceHandler.java:324)
>
> org.datanucleus.state.AbstractStateManager.loadFieldsFromDatastore(AbstractStateManager.java:1122)
>
> org.datanucleus.state.JDOStateManager.loadUnloadedFieldsInFetchPlan(JDOStateManager.java:3000)
>
> org.datanucleus.state.AbstractStateManager.loadFieldsInFetchPlan(AbstractStateManager.java:1064)
>
> org.datanucleus.ExecutionContextImpl.performDetachAllOnTxnEndPreparation(ExecutionContextImpl.java:4574)
>
> org.datanucleus.ExecutionContextImpl.preCommit(ExecutionContextImpl.java:4259)
>
> org.datanucleus.ExecutionContextImpl.transactionPreCommit(ExecutionContextImpl.java:654)
>
> org.datanucleus.TransactionImpl.internalPreCommit(TransactionImpl.java:379)
> org.datanucleus.TransactionImpl.commit(TransactionImpl.java:268)
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:98)
>
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransaction(TransactionManager.java:113)
>
> org.apache.sentry.provider.db.service.persistent.SentryStore.retrieveFullPermssionsImage(SentryStore.java:2225)
>
> org.apache.sentry.hdfs.PermImageRetriever.retrieveFullImage(PermImageRetriever.java:56)
>
> org.apache.sentry.hdfs.PermImageRetriever.retrieveFullImage(PermImageRetriever.java:40)
>
> org.apache.sentry.hdfs.DBUpdateForwarder.getAllUpdatesFrom(DBUpdateForwarder.java:85)
>
> org.apache.sentry.hdfs.SentryPlugin.getAllPermsUpdatesFrom(SentryPlugin.java:178)
>
> org.apache.sentry.hdfs.SentryHDFSServiceProcessor.get_all_authz_updates_from(SentryHDFSServiceProcessor.java:48)
>
> org.apache.sentry.hdfs.service.thrift.SentryHDFSService$Processor$get_all_authz_updates_from.getResult(SentryHDFSService.java:403)
>
> org.apache.sentry.hdfs.service.thrift.SentryHDFSService$Processor$get_all_authz_updates_from.getResult(SentryHDFSService.java:388)
> org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>
> org.apache.sentry.hdfs.SentryHDFSServiceProcessorFactory$ProcessorWrapper.process(SentryHDFSServiceProcessorFactory.java:47)
>
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
>
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
