[
https://issues.apache.org/jira/browse/SENTRY-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117706#comment-16117706
]
Alexander Kolbasov edited comment on SENTRY-1855 at 8/8/17 1:44 AM:
--------------------------------------------------------------------
How severe is the problem in the first place? When I checked with [~vamsee] it
seemed that we had 10 failed requests out of ~10K, so this doesn't look like a
big deal.
Getting back to the requirements:
+ R3.) Are the results consistent between Sentry and NN
The results should be *guaranteed* to be consistent from HMS to Sentry to the
Name Node. This means that if a user successfully performed an operation that
changed the permissions, he should *always* get correct results from both
Hive/Impala operations and the Name Node.
This is the first requirement. Only when we have consistency guarantees can we
discuss scalability, performance, etc.
+ C1.) Normal operations should not generate exceptions. The performance of
Sentry 2.0 should be similar to or better than Sentry 2.0
What does this mean? Do you mean 1.8 vs 2.0? How should you measure
performance? Should it be the same or better for all operations? This is a
great goal but I don't see it as a must-have. Note that we provide much
stronger guarantees in 2.0 then the ones that we provided in 1.8.
was (Author: akolb):
How severe is the problem in the first place? When I checked with [~vamsee] it
seemed that we had 10 failed requests out of ~10K, so this doesn't look like a
big deal.
Getting back to the requirements:
+ R3.) Are the results consistent between Sentry and NN
The results should be *guaranteed* to be consistent from HMS to Sentry to the
Name Node. This means that if a user successfully performed an operation that
changed the permissions, he should *always* get correct results from both
Hive/Impala operations and the Name Node.
This is the first request. Only when we have consistency guarantees can we
discuss scalability, performance, etc.
> PERM/PATH transactions can fail to commit to the sentry database under load
> ---------------------------------------------------------------------------
>
> Key: SENTRY-1855
> URL: https://issues.apache.org/jira/browse/SENTRY-1855
> Project: Sentry
> Issue Type: Sub-task
> Components: Sentry
> Affects Versions: 2.0.0
> Reporter: Alexander Kolbasov
> Assignee: Na Li
> Fix For: 2.0.0
>
> Attachments: SENTRY-1855.002-sentry-ha-redesign.patch,
> SENTRY-1855.003-master.patch, SENTRY-1855.01-sentry-ha-redesign.patch
>
>
> Looking at the latest stress runs, we noticed that some of the transactions
> could fail to commit to the database (with Duplicate key exception) after
> exhausting all the retries.
> This problem has become more evident if we have more number of clients
> connecting to Sentry to issue the permission updates. Was able to reproduce
> consistently with 15 clients doing 100 operations each.
> In the past we introduced exponential backoff (SENTRY-1821) so as part of
> test run increased the defaults to 750ms sleep and 20 retries. But even after
> this, the cluster still shows up the transaction failures. This change also
> increases the latency of every transaction in sentry.
> We need to revisit this and come up with a better way to solve this problem.
> {code}
> 2017-07-13 13:18:14,449 ERROR
> org.apache.sentry.provider.db.service.persistent.TransactionManager: The
> transaction has reached max retry number, Exception thrown when executing
> query
> javax.jdo.JDOException: Exception thrown when executing query
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:596)
> at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:252)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.getRole(SentryStore.java:294)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivilegeCore(SentryStore.java:645)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.access$500(SentryStore.java:101)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore$11.execute(SentryStore.java:601)
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransaction(TransactionManager.java:159)
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager.access$100(TransactionManager.java:63)
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager$2.call(TransactionManager.java:213)
> --
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:971)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823)
> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2530)
> at
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1907)
> at
> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2141)
> at
> com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1773)
> ... 33 more
> 2017-07-13 13:18:14,450 ERROR
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor:
> Unknown error for request:
> TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2,
> requestorUserName:hive, roleName:2017_07_12_15_06_38_1_2_805,
> privileges:[TSentryPrivilege(privilegeScope:DATABASE, serverName:server1,
> dbName:2017_07_12_15_06_38_1_2, tableName:, URI:, action:*,
> createTime:1499904401222, grantOption:FALSE, columnName:)]), message: The
> transaction has reached max retry number, Exception thrown when executing
> query
> java.lang.Exception: The transaction has reached max retry number, Exception
> thrown when executing query
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager$ExponentialBackoff.execute(TransactionManager.java:255)
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransactionBlocksWithRetry(TransactionManager.java:209)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.execute(SentryStore.java:3330)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivilege(SentryStore.java:593)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:633)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:256)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:997)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:982)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
