Re: SENTRY-1855: PERM/PATH transactions can fail to commit to the sentry database under load

Hello, Below is the latest proposed approach for permission change. Path change will use the existing approach because there is only one thread to get HMS changes and save into DB. === Requirements: + R1.) from sentry 1.8 to sentry 2.0 "do no harm", such as what's

Re: Review Request 61192: SENTRY-1855: PERM PATH transactions can fail to commit to the sentry database under load

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61192/ --- (Updated July 28, 2017, 11:16 p.m.) Review request for sentry, Alexander

Re: Review Request 61192: SENTRY-1855: PERM PATH transactions can fail to commit to the sentry database under load

> On July 28, 2017, 3:44 a.m., Na Li wrote: > > sentry-provider/sentry-provider-db/src/test/java/org/apache/sentry/provider/db/service/model/TestMSentryUtil.java > > Line 70 (original), 70 (patched) > > > > > > the

Re: Review Request 61192: SENTRY-1855: PERM PATH transactions can fail to commit to the sentry database under load

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61192/#review181721 ---

Re: Review Request 61192: SENTRY-1855: PERM PATH transactions can fail to commit to the sentry database under load

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/61192/#review181643 ---

Re: SENTRY-1855: PERM/PATH transactions can fail to commit to the sentry database under load

To avoid NN out of sync with Sentry, if NN has skip multiple holes within a time frame, say skipped 10 holes in a day, it will request for full snapshot by asking changeID being 0. Those parameters are configurable On Thu, Jul 27, 2017 at 10:09 AM, Na Li wrote: > Approach

Re: SENTRY-1855: PERM/PATH transactions can fail to commit to the sentry database under load

Approach 3) Sentry sends continuous changes 3.1) NN asks for the oldest changeID that is not processed. 3.2) Sentry server sends back the list including and above that requested changeID. Sentry server sends back all continuous changes in that list starting from requested changeID. If the hole is

Re: SENTRY-1855: PERM/PATH transactions can fail to commit to the sentry database under load

Hi, Based on testing result, we found transactions fail to commit when running 2 sentry servers with 15 concurrent clients issuing 200 GRANTS/REVOKES each. So the current approach of manually increasing changeID has serious performance issue. We need to develop a solution that has good

Re: SENTRY-1855: PERM/PATH transactions can fail to commit to the sentry database under load

Lia, I don’t understand why it would be any different, but there is no harm in trying. > On Jul 24, 2017, at 11:50 PM, Na Li wrote: > > Sasha, > > "How do you know that it improves performance? Do you see the improvement > in your experiments? If you do see

Re: SENTRY-1855: PERM/PATH transactions can fail to commit to the sentry database under load

Sasha, "How do you know that it improves performance? Do you see the improvement in your experiments? If you do see improvements, let’s do the change." [Lina] I have not run the test. Vamsee will run the test on cluster 30 minutes later today for current approach with re-order transactions. So we

Re: SENTRY-1855: PERM/PATH transactions can fail to commit to the sentry database under load

Lina, > On Jul 24, 2017, at 10:20 PM, Na Li wrote: > > Sasha, > > I found the following approach that could improve the performance of the > current approach and second approach a lot with minor code change How do you know that it improves performance? Do you see the

Re: SENTRY-1855: PERM/PATH transactions can fail to commit to the sentry database under load

Sasha, I found the following approach that could improve the performance of the current approach and second approach a lot with minor code change Right now, the execution order of the transaction blocks in a transaction is 1. Delta transaction block to save the perm change and path change 2. The

Re: SENTRY-1855: PERM/PATH transactions can fail to commit to the sentry database under load

Sasha, I realize a serious issue in approach 2.2) that it does not function correctly. So I re-iterate the approaches and their pros and cons. This leaves us the option 2.1) and 2.3). The question is if the performance of approach 2.1) acceptable for now, and we can work on the long term

SENTRY-1855: PERM/PATH transactions can fail to commit to the sentry database under load

Hello, Can you provide your feedback on this issue? We need to make decision soon. 1) Summary The current approach updates changeID (primary key for permission change and path change) manually. In a single transaction, the code reads the max changeID from DB, increases it by one, and save the