To avoid NN out of sync with Sentry, if NN has skip multiple holes within a time frame, say skipped 10 holes in a day, it will request for full snapshot by asking changeID being 0. Those parameters are configurable
On Thu, Jul 27, 2017 at 10:09 AM, Na Li <[email protected]> wrote: > Approach 3) Sentry sends continuous changes > 3.1) NN asks for the oldest changeID that is not processed. > 3.2) Sentry server sends back the list including and above that requested > changeID. Sentry server sends back all continuous changes in that list > starting from requested changeID. If the hole is at the front of the list, > send back a single change right after the hole. > 3.3) When NN gets the single entry without requested ID, it request for > the changeID for the earliest hole. It retries several times (for example > 3). > 3.3.1) If it gets the change of the hole, apply all changes up to next > hole. > 3.3.2) if still does not get it, it skips the hole, and applies the > changes up to the next hole or end of the list. > 3.4) Repeat 3.1) > For example > If NN asks for N, and Sentry server has a list of N, N+2, N+3. NN gets N, it > applies N. Next time, it asks for N+1. If Sentry has N+1 by that time, it > sends NN the list N+1, N+2, N+3. NN applies all of them and moves on. If > Sentry server still does not have N+1, it sends N+2 to NN. NN knows there > is a hole (not there is no updates from N+1), NN retry next round. After > number of retry, it assumes N+1 is a permanent hole, applies N+2, and > asks for N+3. > > Pros: a) No change to NN to Sentry server protocol > b) No re-apply of the changes > c) Reduce the duplicated changes sent to NN when there is a hole > d) Can detect if there is a hole (skip it after multiple retry) > or there is no changes from that changeID, so keep on asking for that > changeID > > Cons: a) Need to configure the retry number for a hole > b) Introduce delay between the changes and applying them at NN > when there is a hole. It could cause security issue. > c) Duplicate changes are sent to NN when there is a hole. > d) NN needs to maintain state. > > On Wed, Jul 26, 2017 at 5:26 PM, Na Li <[email protected]> wrote: > >> Hi, >> >> Based on testing result, we found transactions fail to commit when running >> 2 sentry servers with 15 concurrent clients issuing 200 GRANTS/REVOKES >> each. So the current approach of manually increasing changeID has serious >> performance issue. >> >> We need to develop a solution that has good performance and behave >> correctly. >> >> I list the following approaches and please feel free to provide your >> feedback or add more approaches. We need to reach agreement on the solution >> soon. >> >> Approach 1.) NN asks for missing change >> 1.1) NN asks for the oldest changeID that is not processed. >> 1.2) Sentry server sends back the list including and above that requested >> changeID. Sentry server sends back all changes in that list even when there >> is a hole in the list. >> 1.3) When NN finds a hole(s), it puts following changes in a buffer, and >> request for the changeID for the earliest hole. It retries several times >> (for example 3). >> 1.3.1) If it gets the change of the hole, apply all changes up to next >> hole. >> 1.3.2) if still does not get it, it skips the hole, and applies the >> changes up to the next hole or end of the list. >> 1.4) Repeat 1.1) >> For example >> If NN gets N, N+2, N+3, it applies N and keep N+2, N+3 in buffer. Next >> time, it asks for N+1. If Sentry has N+1, it sends NN the list N+1, N+2, >> N+3 (N+2, N+3 are sent twice). NN applies all of them and moves on. If it >> does not get N+1, NN retry next round. After number of retry, it assumes >> N+1 is permanent hole, applies N+2 and N+3, and asks for N+4. >> >> Pros: a) No change to NN to Sentry server protocol >> b) No re-apply of the changes >> >> Cons: a) Need to configure the retry number for a hole >> b) Introduce delay between the changes and applying them at NN >> when there is a hole. It could cause security issue. >> c) Duplicate changes are sent to NN when there is a hole. >> d) NN needs to maintain state. >> >> Approach 2.) Sentry sends back old changes and NN replay >> 2.1) NN asks for the newest changeID that is not received. >> 2.2) Sentry server sends back the list including and above the requested >> [changeID -X]. Sentry server sends back all changes in that list even when >> there is a hole in the list. X is configurable parameter >> 2.3) NN applies all received changes . >> 2.4) Repeat 2.1) >> For example: >> Suppose X = 10. If NN gets N, N+2, N+3, it applies N, N+2, N+3. Next >> time, it asks for N+4. Sentry gets entries equal of larger than N+4 -10 = >> N-6. If Sentry has N+1, it sends NN the list N-6, N-5, N-4, N-3, N-2, N-1, >> N, N+1, N+2, N+3. NN re-applies from N-6 to N, apply N+1,and re-apply from >> N+2 to N+3. >> >> Pros: a) No change to NN to Sentry server protocol >> b) NN keeps no state >> >> Cons: a) Need to configure the X. If X is too small, may miss changes. If >> X is too big, too many duplicate changes to NN and re-apply >> b) Need to make sure re-apply does not cause issue. >> >> Thanks, >> >> Lina >> >> >> On Mon, Jul 24, 2017 at 5:19 PM, Alexander Kolbasov <[email protected]> >> wrote: >> >>> >>> > >>> > Reducing the time between reading max(changeID in DB) and transaction >>> > commit will reduce the chance of key conflict. That is the whole point >>> of >>> > re-order the blocks. >>> > >>> >>> >>> Why would this affect anything? Whenever you read max(changeID) inside a >>> transaction you should get exactly the same value since we are using >>> repeatable-read transaction isolation level. You will get the value at the >>> start of the transaction, not at the time you read it. >>> >>> >>> >> >
