Vamsee Yarlagadda created SENTRY-1821:
-----------------------------------------
Summary: Transactions could fail to commit to the database under
load
Key: SENTRY-1821
URL: https://issues.apache.org/jira/browse/SENTRY-1821
Project: Sentry
Issue Type: Sub-task
Components: Sentry
Affects Versions: sentry-ha-redesign
Reporter: Vamsee Yarlagadda
I notice this problem when trying to issue some GRANT commands where the
transactions fail after exhausting the number of retries to commit to the
database.
Current load conditions:
* 20 parallel clients each issuing a GRANT command on MariaDB backed Sentry
server.
Even with this relatively small load, we came across this issue which could get
become a major problem under high load. Looking at the retry logic, we do have
static wait before retrying but most probably all the clients are waking up at
the same time which keeps hitting the same problem and thus exhausts all the
retries.
Instead of a static wait, we can think of implementing an exponential backoff
as a way to avoid this.
{code}
Caused by: java.lang.RuntimeException: Unknown error for request:
TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2,
requestorUserName:hive, roleName:gr_role_2017_06_27_16_48_10_0_0_8,
privileges:[TSentryPrivilege(privilegeScope:DATABASE, serverName:server1,
dbName:gr_db_2017_06_27_16_48_10, tableName:, URI:, action:*,
createTime:1498607304833, grantOption:FALSE, columnName:)]), message: The
transaction has reached max retry number, Exception thrown when executing
query. Server Stacktrace: java.lang.Exception: The transaction has reached max
retry number, Exception thrown when executing query
at
org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransactionBlocksWithRetry(TransactionManager.java:224)
at
org.apache.sentry.provider.db.service.persistent.SentryStore.execute(SentryStore.java:3272)
at
org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivilege(SentryStore.java:589)
at
org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:629)
at
org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:256)
at
org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:997)
at
org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:982)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
at
org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:36)
at
org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
at
org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: javax.jdo.JDOException: Exception thrown when executing query
NestedThrowables:
java.sql.BatchUpdateException: Duplicate entry '1153' for key 'PRIMARY'
at
org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:596)
at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:252)
at
org.apache.sentry.provider.db.service.persistent.SentryStore.getRole(SentryStore.java:291)
at
org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivilegeCore(SentryStore.java:641)
at
org.apache.sentry.provider.db.service.persistent.SentryStore.access$500(SentryStore.java:101)
at
org.apache.sentry.provider.db.service.persistent.SentryStore$11.execute(SentryStore.java:597)
at
org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransaction(TransactionManager.java:148)
at
org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransactionBlocksWithRetry(TransactionManager.java:214)
... 14 more
Caused by: java.sql.BatchUpdateException: Duplicate entry '1153' for key
'PRIMARY'
at
com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1815)
at
com.mysql.jdbc.PreparedStatement.executeBatch(PreparedStatement.java:1277)
at
com.jolbox.bonecp.StatementHandle.executeBatch(StatementHandle.java:469)
at
org.datanucleus.store.rdbms.ParamLoggingPreparedStatement.executeBatch(ParamLoggingPreparedStatement.java:372)
at
org.datanucleus.store.rdbms.SQLController.processConnectionStatement(SQLController.java:683)
at
org.datanucleus.store.rdbms.SQLController.getStatementForQuery(SQLController.java:324)
at
org.datanucleus.store.rdbms.query.RDBMSQueryUtils.getPreparedStatementForQuery(RDBMSQueryUtils.java:217)
at
org.datanucleus.store.rdbms.query.JDOQLQuery.performExecute(JDOQLQuery.java:633)
at org.datanucleus.store.query.Query.executeQuery(Query.java:1786)
at org.datanucleus.store.query.Query.executeWithArray(Query.java:1672)
at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:243)
... 20 more
Caused by:
com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException:
Duplicate entry '1153' for key 'PRIMARY'
at sun.reflect.GeneratedConstructorAccessor37.newInstance(Unknown
Source)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at com.mysql.jdbc.Util.handleNewInstance(Util.java:377)
at com.mysql.jdbc.Util.getInstance(Util.java:360)
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:971)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2530)
at
com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1907)
at
com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2141)
at
com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1773)
... 30 more
at org.apache.sentry.service.thrift.Status.throwIfNotOk(Status.java:109)
at
org.apache.sentry.provider.db.service.thrift.SentryPolicyServiceClientDefaultImpl.grantPrivilege(SentryPolicyServiceClientDefaultImpl.java:388)
at
org.apache.sentry.provider.db.service.thrift.SentryPolicyServiceClientDefaultImpl.grantDatabasePrivilege(SentryPolicyServiceClientDefaultImpl.java:285)
... 28 more
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
