Alexander Kolbasov created SENTRY-1855:
------------------------------------------
Summary: PERM/PATH transactions can fail to commit to the sentry
database under load
Key: SENTRY-1855
URL: https://issues.apache.org/jira/browse/SENTRY-1855
Project: Sentry
Issue Type: Sub-task
Components: Sentry
Affects Versions: sentry-ha-redesign
Reporter: Alexander Kolbasov
Assignee: Na Li
Looking at the latest stress runs, we noticed that some of the transactions
could fail to commit to the database (with Duplicate key exception) after
exhausting all the retries.
This problem has become more evident if we have more number of clients
connecting to Sentry to issue the permission updates. Was able to reproduce
consistently with 15 clients doing 100 operations each.
In the past we introduced exponential backoff (SENTRY-1821) so as part of test
run increased the defaults to 750ms sleep and 20 retries. But even after this,
the cluster still shows up the transaction failures. This change also increases
the latency of every transaction in sentry.
We need to revisit this and come up with a better way to solve this problem.
{code}
2017-07-13 13:18:14,449 ERROR
org.apache.sentry.provider.db.service.persistent.TransactionManager: The
transaction has reached max retry number, Exception thrown when executing query
javax.jdo.JDOException: Exception thrown when executing query
at
org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:596)
at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:252)
at
org.apache.sentry.provider.db.service.persistent.SentryStore.getRole(SentryStore.java:294)
at
org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivilegeCore(SentryStore.java:645)
at
org.apache.sentry.provider.db.service.persistent.SentryStore.access$500(SentryStore.java:101)
at
org.apache.sentry.provider.db.service.persistent.SentryStore$11.execute(SentryStore.java:601)
at
org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransaction(TransactionManager.java:159)
at
org.apache.sentry.provider.db.service.persistent.TransactionManager.access$100(TransactionManager.java:63)
at
org.apache.sentry.provider.db.service.persistent.TransactionManager$2.call(TransactionManager.java:213)
--
at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:971)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887)
at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823)
at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2530)
at
com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1907)
at
com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2141)
at
com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1773)
... 33 more
2017-07-13 13:18:14,450 ERROR
org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor:
Unknown error for request:
TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2,
requestorUserName:hive, roleName:2017_07_12_15_06_38_1_2_805,
privileges:[TSentryPrivilege(privilegeScope:DATABASE, serverName:server1,
dbName:2017_07_12_15_06_38_1_2, tableName:, URI:, action:*,
createTime:1499904401222, grantOption:FALSE, columnName:)]), message: The
transaction has reached max retry number, Exception thrown when executing query
java.lang.Exception: The transaction has reached max retry number, Exception
thrown when executing query
at
org.apache.sentry.provider.db.service.persistent.TransactionManager$ExponentialBackoff.execute(TransactionManager.java:255)
at
org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransactionBlocksWithRetry(TransactionManager.java:209)
at
org.apache.sentry.provider.db.service.persistent.SentryStore.execute(SentryStore.java:3330)
at
org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivilege(SentryStore.java:593)
at
org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:633)
at
org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:256)
at
org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:997)
at
org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:982)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
{code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
