[ https://issues.apache.org/jira/browse/SENTRY-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alexander Kolbasov updated SENTRY-1855: --------------------------------------- Status: Patch Available (was: Open) > PERM/PATH transactions can fail to commit to the sentry database under load > --------------------------------------------------------------------------- > > Key: SENTRY-1855 > URL: https://issues.apache.org/jira/browse/SENTRY-1855 > Project: Sentry > Issue Type: Sub-task > Components: Sentry > Affects Versions: sentry-ha-redesign > Reporter: Alexander Kolbasov > Assignee: Alexander Kolbasov > Fix For: sentry-ha-redesign > > Attachments: SENTRY-1855.01-sentry-ha-redesign.patch > > > Looking at the latest stress runs, we noticed that some of the transactions > could fail to commit to the database (with Duplicate key exception) after > exhausting all the retries. > This problem has become more evident if we have more number of clients > connecting to Sentry to issue the permission updates. Was able to reproduce > consistently with 15 clients doing 100 operations each. > In the past we introduced exponential backoff (SENTRY-1821) so as part of > test run increased the defaults to 750ms sleep and 20 retries. But even after > this, the cluster still shows up the transaction failures. This change also > increases the latency of every transaction in sentry. > We need to revisit this and come up with a better way to solve this problem. > {code} > 2017-07-13 13:18:14,449 ERROR > org.apache.sentry.provider.db.service.persistent.TransactionManager: The > transaction has reached max retry number, Exception thrown when executing > query > javax.jdo.JDOException: Exception thrown when executing query > at > org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:596) > at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:252) > at > org.apache.sentry.provider.db.service.persistent.SentryStore.getRole(SentryStore.java:294) > at > org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivilegeCore(SentryStore.java:645) > at > org.apache.sentry.provider.db.service.persistent.SentryStore.access$500(SentryStore.java:101) > at > org.apache.sentry.provider.db.service.persistent.SentryStore$11.execute(SentryStore.java:601) > at > org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransaction(TransactionManager.java:159) > at > org.apache.sentry.provider.db.service.persistent.TransactionManager.access$100(TransactionManager.java:63) > at > org.apache.sentry.provider.db.service.persistent.TransactionManager$2.call(TransactionManager.java:213) > -- > at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:971) > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887) > at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823) > at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435) > at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582) > at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2530) > at > com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1907) > at > com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2141) > at > com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1773) > ... 33 more > 2017-07-13 13:18:14,450 ERROR > org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: > Unknown error for request: > TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2, > requestorUserName:hive, roleName:2017_07_12_15_06_38_1_2_805, > privileges:[TSentryPrivilege(privilegeScope:DATABASE, serverName:server1, > dbName:2017_07_12_15_06_38_1_2, tableName:, URI:, action:*, > createTime:1499904401222, grantOption:FALSE, columnName:)]), message: The > transaction has reached max retry number, Exception thrown when executing > query > java.lang.Exception: The transaction has reached max retry number, Exception > thrown when executing query > at > org.apache.sentry.provider.db.service.persistent.TransactionManager$ExponentialBackoff.execute(TransactionManager.java:255) > at > org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransactionBlocksWithRetry(TransactionManager.java:209) > at > org.apache.sentry.provider.db.service.persistent.SentryStore.execute(SentryStore.java:3330) > at > org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivilege(SentryStore.java:593) > at > org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:633) > at > org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:256) > at > org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:997) > at > org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:982) > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) > {code} -- This message was sent by Atlassian JIRA (v6.4.14#64029)