[
https://issues.apache.org/jira/browse/SENTRY-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16117056#comment-16117056
]
Na Li commented on SENTRY-1855:
-------------------------------
Hello,
Below is the latest proposed approach for permission change. Path change will
use the existing approach because there is only one thread to get HMS changes
and save into DB.
===============================
Requirements:
+ R1.) from sentry 1.8 to sentry 2.0 "do no harm", such as what's our upgrade
experience. Performance comparison
+ R2.) Is the approach scalable in terms of supporting multiple Sentry servers
and concurrent updates
+ R3.) Are the results consistent between Sentry and NN
+ R4.) Does it impact HMS-HA
Constrains from Requirements:
+ C1.) Normal operations should not generate exceptions. The performance of
Sentry 2.0 should be similar to or better than Sentry 2.0
+ C2.) The system should work reasonable well when having more Sentry servers
adding into the system and handling concurrent updates
+ C3.) The order Sentry applies the changes should be the same order NN applies
the changes, so NN will end up with the same result as Sentry and they behave
consistently
Current Approach for Permission Change:
+ The changeID is primary key and manually updated in Sentry application. The
current max(changeID) is read from DB, increased by 1, and then used as
primarykey for new change entry. If more than one threads are saving new change
entry with the same changeID, the transaction in onethread succeeds, such
transactions from other transactions fail, and go through retry with
exponential retry interval.
Benefits of the current approach
+ Satisfies R3.). The changeID is continuously increased. There is no hole in
changeID. Guaranteed consistency for both deltas and full snapshots.
+ Satisfied R4.). It should not affect HMS-HA
Issues with current approach
+ Violates R1.) because normal concurrent transaction could cause transaction
failure due to key conflict. When transaction fails, exponential retry interval
causes additional delay for transaction. When load reaches to certain point,
transaction fails to commit
+ Violates R2.) and it is not scalable. In our endurance test, we see
transaction failed to commit after max retry with two sentry servers
Proposed Approach for Permission Change:
+ The changeID is primary key and auto-incremented in DataStore. When a
transaction fails, it creates permanent hole in changeID. When a transaction
starts early but committed after transactions with larger changeID, it forms
temporary hole in changeID. Once this transaction is committed, the temporary
hole disappears. How long a temporary hole can exist is limited by how long a
transaction can be pending and still be committed successfully.
0) read <write_timeout> from configuration and set to datastore to timeout
transactions that take longer than that value. In this way, we can limit how
long a transaction can be pending and still be committed successfully.
<transaction buffer time> = <buffer factor> * <write_timeout>, and its default
value is 2. This is how long we go back of time to capture temporary holes in
changeID.
1.) Change the MSentryPermChange timestamp to be assigned by DB, not from
sentry server. In this way, we have a single source of truth for the timestamp
of the permission change entries.
1.a) Need to make sure the timestamp (using @CreateTimestamp annotation in
datanucleus) is set by DB, not be set by datanucleus using server time. This
can be done without using @CreateTimestamp if necessary by just doing a
default value of CURRENT_TIMESTAMP(6) on the database table and not setting
anything on the object. If the annotation can handle it, great, but we have to
make sure of the semantics.
2.) When Sentry sends changes to NN, it also includes the "current time in DB"
(Now).
3.) When NN asks for changes, it sends to Sentry the "last received 'current
time in DB'" (referred as Last_Now) together with the last processed changeID
for permission and [imageID, changeID] for path changes
4.) When Sentry receives the request, it sends all entries that are newer than
the time = Last_Now - <transaction buffer time>. NN applies the changes in
increasing order of changeID
Benefits of the proposed approach:
+ Satisfies requirements R1.), R2.), R3.) and R4.). It should not affect HMS-HA
Issues with Proposed Approach:
+ More complicated to implement than current approach is.
+ Require change protocol API between NN and Sentry to pass "current time in
DB"
+ Need to re-apply some old updates. May cause more overhead, but should be
reasonable. We can optimize how to reapply old updates. It will be done in next
release as the change will be local to NN.
Background on R3.)
There are several types of transaction orders: TO1) The order of transaction
starting time; TO2) The order of permission SQL execution time; TO3) The order
of transaction commit time; What order affects the result? NN should apply perm
changes in the order that determines the result, so it will behave consistently
with Sentry.
Our testing result shows that the TO2) is the order that determines the result.
Two transactions: T_1 First
End Result
Transaction Commit Order T_1: Transaction #1
T_1 First T_2 First S_1: SQL in Transaction #1
SQL Execution Order
S_1 First V_2 V_2 V_1: Value of Transaction #1
S_2 First V_1 V_1
Conclusion: SQL execution order determins the end result
The next question is "how to capture the order of permission SQL execution
order" One approach is to get the permission execution timestamp and save it to
the permission change entry. This brings two issues: a) It requires a lot of
code changes to do so. 2) It is possible to have two transactions on the same
authentication object with same timestamp even when it is on microsecond
granularity. In this case, timestamp fails to capture the SQL execution order.
Our test shows that when permission SQL execution is followed by permission
change log execution (that is when changeID is auto-incremented), the order of
changeID is exactly the same as the permission SQL execution. As a result, we
choose to apply changes at NN by the changeID order. This should satisfy R3.).
Two transactions: T_1 First
T_1 contains two SQL: S_1 followed by C_1
S_1: normal SQL query
C_1: log SQL change, primary key changeID is auto-increment
End Result
Log SQL Order
C_1 First C_2 First
SQL Execution Order
S_1 First C_1=1; C_2=2 C_1=1; C_2=2
S_2 First C_2=1; C_1=2 C_2=1; C_1=2
Conclusion: changeID is in the same order as the SQL execution order
Q & A
Question_1: Can we skip applying old updates? For example, the transactions
identified by the changeID are committed in the following timely order
[changeID_1, changeID_3, changeID_2]. When NN gets [changeID_1, changeID_3], it
applies them. When it gets [changeID_1, changeID_3, changeID_2], can it skips
re-applying [changeID: changeID_1, changeID_3], and only applies changeID_2?
That will avoid overhead of re-applying the old changes.
Answer_1: NO. The result is determined by SQL execution order and it is the
same as the changeID order. It is not the transaction commit order. The above
approach applies changes at NN in order of [changeID_1, changeID_3,
changeID_2], which is different from how Sentry applies the changes, which is
[changeID_1, changeID_2, changeID_3]. This will cause inconsistency between
Sentry and NN.
Question_2: Can we hold on applies some changes and until the temporary hole
finishes, and then apply changes in changeID order at NN?
Answer_2: the temporary hole could exist for a minute. Holding applying changes
for a minute may not be desirable. If we wait for every temporary hole for a
minute, the changes could backup, and NN will be out-of-sync from Sentry for a
while.
Thanks,
Lina
> PERM/PATH transactions can fail to commit to the sentry database under load
> ---------------------------------------------------------------------------
>
> Key: SENTRY-1855
> URL: https://issues.apache.org/jira/browse/SENTRY-1855
> Project: Sentry
> Issue Type: Sub-task
> Components: Sentry
> Affects Versions: 2.0.0
> Reporter: Alexander Kolbasov
> Assignee: Na Li
> Fix For: 2.0.0
>
> Attachments: SENTRY-1855.002-sentry-ha-redesign.patch,
> SENTRY-1855.003-master.patch, SENTRY-1855.01-sentry-ha-redesign.patch
>
>
> Looking at the latest stress runs, we noticed that some of the transactions
> could fail to commit to the database (with Duplicate key exception) after
> exhausting all the retries.
> This problem has become more evident if we have more number of clients
> connecting to Sentry to issue the permission updates. Was able to reproduce
> consistently with 15 clients doing 100 operations each.
> In the past we introduced exponential backoff (SENTRY-1821) so as part of
> test run increased the defaults to 750ms sleep and 20 retries. But even after
> this, the cluster still shows up the transaction failures. This change also
> increases the latency of every transaction in sentry.
> We need to revisit this and come up with a better way to solve this problem.
> {code}
> 2017-07-13 13:18:14,449 ERROR
> org.apache.sentry.provider.db.service.persistent.TransactionManager: The
> transaction has reached max retry number, Exception thrown when executing
> query
> javax.jdo.JDOException: Exception thrown when executing query
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:596)
> at org.datanucleus.api.jdo.JDOQuery.execute(JDOQuery.java:252)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.getRole(SentryStore.java:294)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivilegeCore(SentryStore.java:645)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.access$500(SentryStore.java:101)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore$11.execute(SentryStore.java:601)
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransaction(TransactionManager.java:159)
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager.access$100(TransactionManager.java:63)
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager$2.call(TransactionManager.java:213)
> --
> at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:971)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3887)
> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3823)
> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2435)
> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2582)
> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2530)
> at
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1907)
> at
> com.mysql.jdbc.PreparedStatement.executeUpdate(PreparedStatement.java:2141)
> at
> com.mysql.jdbc.PreparedStatement.executeBatchSerially(PreparedStatement.java:1773)
> ... 33 more
> 2017-07-13 13:18:14,450 ERROR
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor:
> Unknown error for request:
> TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2,
> requestorUserName:hive, roleName:2017_07_12_15_06_38_1_2_805,
> privileges:[TSentryPrivilege(privilegeScope:DATABASE, serverName:server1,
> dbName:2017_07_12_15_06_38_1_2, tableName:, URI:, action:*,
> createTime:1499904401222, grantOption:FALSE, columnName:)]), message: The
> transaction has reached max retry number, Exception thrown when executing
> query
> java.lang.Exception: The transaction has reached max retry number, Exception
> thrown when executing query
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager$ExponentialBackoff.execute(TransactionManager.java:255)
> at
> org.apache.sentry.provider.db.service.persistent.TransactionManager.executeTransactionBlocksWithRetry(TransactionManager.java:209)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.execute(SentryStore.java:3330)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivilege(SentryStore.java:593)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:633)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:256)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:997)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:982)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
