[ 
https://issues.apache.org/jira/browse/SENTRY-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117952#comment-16117952
 ] 

Alexander Kolbasov commented on SENTRY-1855:
--------------------------------------------

The current behavior have some nice properties:

1) It guarantees that when change N is committed all prior changes are 
committed as well.
2) It associates every full snapshot with exact ID value corresponding to it.
3) It is guaranteed to work correctly, very simple, easy to understand and 
verify.

Essentially it provides a semantic of atomic increment operation. Proposed 
solution doesn't have this property, it doesn't guarantee that specific 
snapshot is associated with a consistent state and that all prior changes are 
committed. It is complicated and still doesn't guarantee correctness.

> PERM/PATH transactions can fail to commit to the sentry database under load
> ---------------------------------------------------------------------------
>
>                 Key: SENTRY-1855
>                 URL: https://issues.apache.org/jira/browse/SENTRY-1855
>             Project: Sentry
>          Issue Type: Sub-task
>          Components: Sentry
>    Affects Versions: 2.0.0
>            Reporter: Alexander Kolbasov
>            Assignee: Na Li
>             Fix For: 2.0.0
>
>         Attachments: SENTRY-1855.002-sentry-ha-redesign.patch, 
> SENTRY-1855.003-master.patch, SENTRY-1855.01-sentry-ha-redesign.patch
>
>
> Looking at the latest stress runs, we noticed that some of the transactions 
> could fail to commit to the database (with Duplicate key exception) after 
> exhausting all the retries.
> This problem has become more evident if we have more number of clients 
> connecting to Sentry to issue the permission updates. Was able to reproduce 
> consistently with 15 clients doing 100 operations each.
> In the past we introduced exponential backoff (SENTRY-1821) so as part of 
> test run increased the defaults to 750ms sleep and 20 retries. But even after 
> this, the cluster still shows up the transaction failures. This change also 
> increases the latency of every transaction in sentry.
> We need to revisit this and come up with a better way to solve this problem.
> {code}
> 2017-07-13 13:18:14,449 ERROR 
> org.apache.sentry.provider.db.service.persistent.TransactionManager: The 
> transaction has reached max retry number, Exception thrown when executing 
> query
> javax.jdo.JDOException: Exception thrown when executing query
>       at 
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:596)
>       at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:252)
>       at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.getRole(SentryStore.java:294)
>       at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivilegeCore(SentryStore.java:645)
>       at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.access$500(SentryStore.java:101)
>       at 
> org.apache.sentry.provider.db.service.persistent.SentryStore$11.execute(SentryStore.java:601)
>       at 
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransaction(TransactionManager.java:159)
>       at 
> org.apache.sentry.provider.db.service.persistent.TransactionManager.access$100(TransactionManager.java:63)
>       at 
> org.apache.sentry.provider.db.service.persistent.TransactionManager$2.call(TransactionManager.java:213)
> --
>       at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:971)
>       at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887)
>       at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823)
>       at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
>       at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
>       at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2530)
>       at 
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1907)
>       at 
> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2141)
>       at 
> com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1773)
>       ... 33 more
> 2017-07-13 13:18:14,450 ERROR 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: 
> Unknown error for request: 
> TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2, 
> requestorUserName:hive, roleName:2017_07_12_15_06_38_1_2_805, 
> privileges:[TSentryPrivilege(privilegeScope:DATABASE, serverName:server1, 
> dbName:2017_07_12_15_06_38_1_2, tableName:, URI:, action:*, 
> createTime:1499904401222, grantOption:FALSE, columnName:)]), message: The 
> transaction has reached max retry number, Exception thrown when executing 
> query
> java.lang.Exception: The transaction has reached max retry number, Exception 
> thrown when executing query
>       at 
> org.apache.sentry.provider.db.service.persistent.TransactionManager$ExponentialBackoff.execute(TransactionManager.java:255)
>       at 
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransactionBlocksWithRetry(TransactionManager.java:209)
>       at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.execute(SentryStore.java:3330)
>       at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivilege(SentryStore.java:593)
>       at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:633)
>       at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:256)
>       at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:997)
>       at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:982)
>       at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>       at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Reply via email to