Approach 3) Sentry sends continuous changes
3.1) NN asks for the oldest changeID that is not processed.
3.2) Sentry server sends back the list including and above that requested
changeID. Sentry server sends back all continuous changes in that list
starting from requested changeID. If the hole is at the front of the list,
send back a single change right after the hole.
3.3) When NN gets the single entry without requested ID, it request for
the changeID for the earliest hole. It retries several times (for example
3).
3.3.1) If it gets the change of the hole, apply all changes up to next
hole.
3.3.2) if still does not get it, it skips the hole, and applies the
changes up to the next hole or end of the list.
3.4) Repeat 3.1)
For example
If NN asks for N, and Sentry server has a list of N, N+2, N+3. NN gets N, it
applies N. Next time, it asks for N+1. If Sentry has N+1 by that time, it
sends NN the list N+1, N+2, N+3. NN applies all of them and moves on. If
Sentry server still does not have N+1, it sends N+2 to NN. NN knows there
is a hole (not there is no updates from N+1), NN retry next round. After
number of retry, it assumes N+1 is a permanent hole, applies N+2, and asks
for N+3.
Pros: a) No change to NN to Sentry server protocol
b) No re-apply of the changes
c) Reduce the duplicated changes sent to NN when there is a hole
d) Can detect if there is a hole (skip it after multiple retry) or
there is no changes from that changeID, so keep on asking for that changeID
Cons: a) Need to configure the retry number for a hole
b) Introduce delay between the changes and applying them at NN
when there is a hole. It could cause security issue.
c) Duplicate changes are sent to NN when there is a hole.
d) NN needs to maintain state.
On Wed, Jul 26, 2017 at 5:26 PM, Na Li <lina...@cloudera.com> wrote:
> Hi,
>
> Based on testing result, we found transactions fail to commit when running
> 2 sentry servers with 15 concurrent clients issuing 200 GRANTS/REVOKES
> each. So the current approach of manually increasing changeID has serious
> performance issue.
>
> We need to develop a solution that has good performance and behave
> correctly.
>
> I list the following approaches and please feel free to provide your
> feedback or add more approaches. We need to reach agreement on the solution
> soon.
>
> Approach 1.) NN asks for missing change
> 1.1) NN asks for the oldest changeID that is not processed.
> 1.2) Sentry server sends back the list including and above that requested
> changeID. Sentry server sends back all changes in that list even when there
> is a hole in the list.
> 1.3) When NN finds a hole(s), it puts following changes in a buffer, and
> request for the changeID for the earliest hole. It retries several times
> (for example 3).
> 1.3.1) If it gets the change of the hole, apply all changes up to next
> hole.
> 1.3.2) if still does not get it, it skips the hole, and applies the
> changes up to the next hole or end of the list.
> 1.4) Repeat 1.1)
> For example
> If NN gets N, N+2, N+3, it applies N and keep N+2, N+3 in buffer. Next
> time, it asks for N+1. If Sentry has N+1, it sends NN the list N+1, N+2,
> N+3 (N+2, N+3 are sent twice). NN applies all of them and moves on. If it
> does not get N+1, NN retry next round. After number of retry, it assumes
> N+1 is permanent hole, applies N+2 and N+3, and asks for N+4.
>
> Pros: a) No change to NN to Sentry server protocol
> b) No re-apply of the changes
>
> Cons: a) Need to configure the retry number for a hole
> b) Introduce delay between the changes and applying them at NN
> when there is a hole. It could cause security issue.
> c) Duplicate changes are sent to NN when there is a hole.
> d) NN needs to maintain state.
>
> Approach 2.) Sentry sends back old changes and NN replay
> 2.1) NN asks for the newest changeID that is not received.
> 2.2) Sentry server sends back the list including and above the requested
> [changeID -X]. Sentry server sends back all changes in that list even when
> there is a hole in the list. X is configurable parameter
> 2.3) NN applies all received changes .
> 2.4) Repeat 2.1)
> For example:
> Suppose X = 10. If NN gets N, N+2, N+3, it applies N, N+2, N+3. Next time,
> it asks for N+4. Sentry gets entries equal of larger than N+4 -10 = N-6. If
> Sentry has N+1, it sends NN the list N-6, N-5, N-4, N-3, N-2, N-1, N, N+1,
> N+2, N+3. NN re-applies from N-6 to N, apply N+1,and re-apply from N+2 to
> N+3.
>
> Pros: a) No change to NN to Sentry server protocol
> b) NN keeps no state
>
> Cons: a) Need to configure the X. If X is too small, may miss changes. If
> X is too big, too many duplicate changes to NN and re-apply
> b) Need to make sure re-apply does not cause issue.
>
> Thanks,
>
> Lina
>
>
> On Mon, Jul 24, 2017 at 5:19 PM, Alexander Kolbasov <ak...@cloudera.com>
> wrote:
>
>>
>> >
>> > Reducing the time between reading max(changeID in DB) and transaction
>> > commit will reduce the chance of key conflict. That is the whole point
>> of
>> > re-order the blocks.
>> >
>>
>>
>> Why would this affect anything? Whenever you read max(changeID) inside a
>> transaction you should get exactly the same value since we are using
>> repeatable-read transaction isolation level. You will get the value at the
>> start of the transaction, not at the time you read it.
>>
>>
>>
>
