Hi,
Based on testing result, we found transactions fail to commit when running
2 sentry servers with 15 concurrent clients issuing 200 GRANTS/REVOKES
each. So the current approach of manually increasing changeID has serious
performance issue.
We need to develop a solution that has good performance and behave
correctly.
I list the following approaches and please feel free to provide your
feedback or add more approaches. We need to reach agreement on the solution
soon.
Approach 1.) NN asks for missing change
1.1) NN asks for the oldest changeID that is not processed.
1.2) Sentry server sends back the list including and above that requested
changeID. Sentry server sends back all changes in that list even when there
is a hole in the list.
1.3) When NN finds a hole(s), it puts following changes in a buffer, and
request for the changeID for the earliest hole. It retries several times
(for example 3).
1.3.1) If it gets the change of the hole, apply all changes up to next
hole.
1.3.2) if still does not get it, it skips the hole, and applies the
changes up to the next hole or end of the list.
1.4) Repeat 1.1)
For example
If NN gets N, N+2, N+3, it applies N and keep N+2, N+3 in buffer. Next time, it
asks for N+1. If Sentry has N+1, it sends NN the list N+1, N+2, N+3 (N+2,
N+3 are sent twice). NN applies all of them and moves on. If it does not
get N+1, NN retry next round. After number of retry, it assumes N+1 is
permanent hole, applies N+2 and N+3, and asks for N+4.
Pros: a) No change to NN to Sentry server protocol
b) No re-apply of the changes
Cons: a) Need to configure the retry number for a hole
b) Introduce delay between the changes and applying them at NN
when there is a hole. It could cause security issue.
c) Duplicate changes are sent to NN when there is a hole.
d) NN needs to maintain state.
Approach 2.) Sentry sends back old changes and NN replay
2.1) NN asks for the newest changeID that is not received.
2.2) Sentry server sends back the list including and above the requested
[changeID -X]. Sentry server sends back all changes in that list even when
there is a hole in the list. X is configurable parameter
2.3) NN applies all received changes .
2.4) Repeat 2.1)
For example:
Suppose X = 10. If NN gets N, N+2, N+3, it applies N, N+2, N+3. Next time,
it asks for N+4. Sentry gets entries equal of larger than N+4 -10 = N-6. If
Sentry has N+1, it sends NN the list N-6, N-5, N-4, N-3, N-2, N-1, N, N+1,
N+2, N+3. NN re-applies from N-6 to N, apply N+1,and re-apply from N+2 to
N+3.
Pros: a) No change to NN to Sentry server protocol
b) NN keeps no state
Cons: a) Need to configure the X. If X is too small, may miss changes. If X
is too big, too many duplicate changes to NN and re-apply
b) Need to make sure re-apply does not cause issue.
Thanks,
Lina
On Mon, Jul 24, 2017 at 5:19 PM, Alexander Kolbasov <[email protected]>
wrote:
>
> >
> > Reducing the time between reading max(changeID in DB) and transaction
> > commit will reduce the chance of key conflict. That is the whole point of
> > re-order the blocks.
> >
>
>
> Why would this affect anything? Whenever you read max(changeID) inside a
> transaction you should get exactly the same value since we are using
> repeatable-read transaction isolation level. You will get the value at the
> start of the transaction, not at the time you read it.
>
>
>
