[
https://issues.apache.org/jira/browse/SENTRY-1821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16066773#comment-16066773
]
Na Li commented on SENTRY-1821:
-------------------------------
Even with fuzzy logic at client retry, when there are many clients, the
collision still happens. It gets worse when more clients are updating sentry.
The solution is not scalable.
It would be the best if we can minimize transaction failure when two or more
transactions are making updates. Permission updates are the main issue, getting
worse with more clients. Path update is from one thread in HMSFollower,
independent on the number of clients, so it is not a big issue.
Can we add synchronization in DeltaTransactionBlock.persistUpdate? In this way,
when more than one threads are updating the permission, only one thread will
update changeID, when it finishes, the next thread on the same Sentry server
will make update. In this way, no transaction will fail if there is only one
Sentry server. When there are more Sentry servers, the collision will happen
among the Sentry servers, not depending on how many clients are updating
Sentry, so it is more scalable.
> Transactions could fail to commit to the database under load
> ------------------------------------------------------------
>
> Key: SENTRY-1821
> URL: https://issues.apache.org/jira/browse/SENTRY-1821
> Project: Sentry
> Issue Type: Sub-task
> Components: Sentry
> Affects Versions: sentry-ha-redesign
> Reporter: Vamsee Yarlagadda
> Assignee: Alexander Kolbasov
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1821.01-sentry-ha-redesign.patch
>
>
> I notice this problem when trying to issue some GRANT commands where the
> transactions fail after exhausting the number of retries to commit to the
> database.
> Current load conditions:
> * 20 parallel clients each issuing a GRANT command on MariaDB backed Sentry
> server.
> Even with this relatively small load, we came across this issue which could
> get become a major problem under high load. Looking at the retry logic, we do
> have static wait before retrying but most probably all the clients are waking
> up at the same time which keeps hitting the same problem and thus exhausts
> all the retries.
> Instead of a static wait, we can think of implementing an exponential backoff
> as a way to avoid this.
> {code}
> Caused by: java.lang.RuntimeException: Unknown error for request:
> TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2,
> requestorUserName:hive, roleName:gr_role_2017_06_27_16_48_10_0_0_8,
> privileges:[TSentryPrivilege(privilegeScope:DATABASE, serverName:server1,
> dbName:gr_db_2017_06_27_16_48_10, tableName:, URI:, action:*,
> createTime:1498607304833, grantOption:FALSE, columnName:)]), message: The
> transaction has reached max retry number, Exception thrown when executing
> query. Server Stacktrace: java.lang.Exception: The transaction has reached
> max retry number, Exception thrown when executing query
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransactionBlocksWithRetry(TransactionManager.java:224)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.execute(SentryStore.java:3272)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivilege(SentryStore.java:589)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:629)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:256)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:997)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:982)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:36)
> at
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: javax.jdo.JDOException: Exception thrown when executing query
> NestedThrowables:
> java.sql.BatchUpdateException: Duplicate entry '1153' for key 'PRIMARY'
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:596)
> at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:252)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.getRole(SentryStore.java:291)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivilegeCore(SentryStore.java:641)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.access$500(SentryStore.java:101)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore$11.execute(SentryStore.java:597)
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransaction(TransactionManager.java:148)
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransactionBlocksWithRetry(TransactionManager.java:214)
> ... 14 more
> Caused by: java.sql.BatchUpdateException: Duplicate entry '1153' for key
> 'PRIMARY'
> at
> com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1815)
> at
> com.mysql.jdbc.PreparedStatement.executeBatch(PreparedStatement.java:1277)
> at
> com.jolbox.bonecp.StatementHandle.executeBatch(StatementHandle.java:469)
> at
> org.datanucleus.store.rdbms.ParamLoggingPreparedStatement.executeBatch(ParamLoggingPreparedStatement.java:372)
> at
> org.datanucleus.store.rdbms.SQLController.processConnectionStatement(SQLController.java:683)
> at
> org.datanucleus.store.rdbms.SQLController.getStatementForQuery(SQLController.java:324)
> at
> org.datanucleus.store.rdbms.query.RDBMSQueryUtils.getPreparedStatementForQuery(RDBMSQueryUtils.java:217)
> at
> org.datanucleus.store.rdbms.query.JDOQLQuery.performExecute(JDOQLQuery.java:633)
> at org.datanucleus.store.query.Query.executeQuery(Query.java:1786)
> at org.datanucleus.store.query.Query.executeWithArray(Query.java:1672)
> at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:243)
> ... 20 more
> Caused by:
> com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
> Duplicate entry '1153' for key 'PRIMARY'
> at sun.reflect.GeneratedConstructorAccessor37.newInstance(Unknown
> Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at com.mysql.jdbc.Util.handleNewInstance(Util.java:377)
> at com.mysql.jdbc.Util.getInstance(Util.java:360)
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:971)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823)
> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2530)
> at
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1907)
> at
> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2141)
> at
> com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1773)
> ... 30 more
> at org.apache.sentry.service.thrift.Status.throwIfNotOk(Status.java:109)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyServiceClientDefaultImpl.grantPrivilege(SentryPolicyServiceClientDefaultImpl.java:388)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyServiceClientDefaultImpl.grantDatabasePrivilege(SentryPolicyServiceClientDefaultImpl.java:285)
> ... 28 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
