[ 
https://issues.apache.org/jira/browse/SENTRY-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16118430#comment-16118430
 ] 

Alexander Kolbasov commented on SENTRY-1855:
--------------------------------------------

See https://dev.mysql.com/doc/refman/5.7/en/innodb-auto-increment-handling.html 
for MySQL handling of auto-increment

> PERM/PATH transactions can fail to commit to the sentry database under load
> ---------------------------------------------------------------------------
>
>                 Key: SENTRY-1855
>                 URL: https://issues.apache.org/jira/browse/SENTRY-1855
>             Project: Sentry
>          Issue Type: Sub-task
>          Components: Sentry
>    Affects Versions: 2.0.0
>            Reporter: Alexander Kolbasov
>            Assignee: Na Li
>             Fix For: 2.0.0
>
>         Attachments: SENTRY-1855.002-sentry-ha-redesign.patch, 
> SENTRY-1855.003-master.patch, SENTRY-1855.01-sentry-ha-redesign.patch
>
>
> Looking at the latest stress runs, we noticed that some of the transactions 
> could fail to commit to the database (with Duplicate key exception) after 
> exhausting all the retries.
> This problem has become more evident if we have more number of clients 
> connecting to Sentry to issue the permission updates. Was able to reproduce 
> consistently with 15 clients doing 100 operations each.
> In the past we introduced exponential backoff (SENTRY-1821) so as part of 
> test run increased the defaults to 750ms sleep and 20 retries. But even after 
> this, the cluster still shows up the transaction failures. This change also 
> increases the latency of every transaction in sentry.
> We need to revisit this and come up with a better way to solve this problem.
> {code}
> 2017-07-13 13:18:14,449 ERROR 
> org.apache.sentry.provider.db.service.persistent.TransactionManager: The 
> transaction has reached max retry number, Exception thrown when executing 
> query
> javax.jdo.JDOException: Exception thrown when executing query
>       at 
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:596)
>       at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:252)
>       at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.getRole(SentryStore.java:294)
>       at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivilegeCore(SentryStore.java:645)
>       at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.access$500(SentryStore.java:101)
>       at 
> org.apache.sentry.provider.db.service.persistent.SentryStore$11.execute(SentryStore.java:601)
>       at 
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransaction(TransactionManager.java:159)
>       at 
> org.apache.sentry.provider.db.service.persistent.TransactionManager.access$100(TransactionManager.java:63)
>       at 
> org.apache.sentry.provider.db.service.persistent.TransactionManager$2.call(TransactionManager.java:213)
> --
>       at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:971)
>       at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887)
>       at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823)
>       at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
>       at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
>       at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2530)
>       at 
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1907)
>       at 
> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2141)
>       at 
> com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1773)
>       ... 33 more
> 2017-07-13 13:18:14,450 ERROR 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: 
> Unknown error for request: 
> TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2, 
> requestorUserName:hive, roleName:2017_07_12_15_06_38_1_2_805, 
> privileges:[TSentryPrivilege(privilegeScope:DATABASE, serverName:server1, 
> dbName:2017_07_12_15_06_38_1_2, tableName:, URI:, action:*, 
> createTime:1499904401222, grantOption:FALSE, columnName:)]), message: The 
> transaction has reached max retry number, Exception thrown when executing 
> query
> java.lang.Exception: The transaction has reached max retry number, Exception 
> thrown when executing query
>       at 
> org.apache.sentry.provider.db.service.persistent.TransactionManager$ExponentialBackoff.execute(TransactionManager.java:255)
>       at 
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransactionBlocksWithRetry(TransactionManager.java:209)
>       at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.execute(SentryStore.java:3330)
>       at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivilege(SentryStore.java:593)
>       at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:633)
>       at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:256)
>       at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:997)
>       at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:982)
>       at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>       at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Reply via email to