[
https://issues.apache.org/jira/browse/SENTRY-2437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667445#comment-16667445
]
Arjun Mishra edited comment on SENTRY-2437 at 10/29/18 4:58 PM:
----------------------------------------------------------------
Below is the summary of the impact of having a single transaction. We are
seeing improvement of over 20 times faster.
The test was done on a table with 1800 columns and a grant was made to each of
those columns in a single statement like _GRANT SELECT(co1, col2, ....,
col1800) ON TABLE_
{noformat}
Current Sentry Code:
HDFS Sync ON
Start Time : 2018-10-29 08:26:15,519 INFO
org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST
RUN: Start grant
Retry(1) : 2018-10-29 08:29:35,663 INFO
org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST
RUN: Start grant
Retry(2) : 2018-10-29 08:32:55,815 INFO
org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST
RUN: Start grant
End Time(1) : 2018-10-29 08:34:05,662 INFO
org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST
RUN: Completed grant
End Time(2) : 2018-10-29 08:34:06,992 INFO
org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST
RUN: Completed grant
End Time(3) : 2018-10-29 08:35:25,798 INFO
org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST
RUN: Completed grant
Total Time : 554.91 sec. After all retries were completed
HDFS Sync OFF
Start Time : 2018-10-29 08:42:13,944 INFO
org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST
RUN: Start grant
Retry(1) : 2018-10-29 08:45:34,141 INFO
org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST
RUN: Start grant
Retry(2) : 2018-10-29 08:48:54,283 INFO
org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST
RUN: Start grant
End Time(1) : 2018-10-29 08:49:58,434 INFO
org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST
RUN: Completed grant
End Time(2) : 2018-10-29 08:50:15,245 INFO
org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST
RUN: Completed grant
End Time(3) : 2018-10-29 08:50:18,654 INFO
org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST
RUN: Completed grant
Total Time : 489.059 sec. After all retries were completed
Code change with a single transaction:
HDFS Sync ON
Start Time : 2018-10-29 09:46:19,260 INFO
org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST
RUN: Start grant
End Time : 2018-10-29 09:46:40,767 INFO
org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST
RUN: Completed grant
Total Time : 26.426 sec
IMPROVEMENT : Over 20 times faster
HDFS Sync OFF
Start Time : 2018-10-29 09:53:37,795 INFO
org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST
RUN: Start grant
End Time : 2018-10-29 09:53:56,433 INFO
org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST
RUN: Completed grant
Total Time : 22.932 sec
IMPROVEMENT : Over 20 times faster
{noformat}
was (Author: arjunmishra13):
Below is the summary of the impact of having a single transaction. We are
seeing improvement of over 20 times faster. The test was done on a table with
1800 columns and a grant was made to each of those columns in a single
statement like _GRANT SELECT(co1, col2, ...., col1800) ON TABLE_
{noformat}
Current Sentry Code:
HDFS Sync ON
Start Time : 2018-10-29 08:26:15,519 INFO
org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST
RUN: Start grant
Retry(1) : 2018-10-29 08:29:35,663 INFO
org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST
RUN: Start grant
Retry(2) : 2018-10-29 08:32:55,815 INFO
org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST
RUN: Start grant
End Time(1) : 2018-10-29 08:34:05,662 INFO
org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST
RUN: Completed grant
End Time(2) : 2018-10-29 08:34:06,992 INFO
org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST
RUN: Completed grant
End Time(3) : 2018-10-29 08:35:25,798 INFO
org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST
RUN: Completed grant
Total Time : 554.91 sec. After all retries were completed
HDFS Sync OFF
Start Time : 2018-10-29 08:42:13,944 INFO
org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST
RUN: Start grant
Retry(1) : 2018-10-29 08:45:34,141 INFO
org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST
RUN: Start grant
Retry(2) : 2018-10-29 08:48:54,283 INFO
org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST
RUN: Start grant
End Time(1) : 2018-10-29 08:49:58,434 INFO
org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST
RUN: Completed grant
End Time(2) : 2018-10-29 08:50:15,245 INFO
org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST
RUN: Completed grant
End Time(3) : 2018-10-29 08:50:18,654 INFO
org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST
RUN: Completed grant
Total Time : 489.059 sec. After all retries were completed
Code change with a single transaction:
HDFS Sync ON
Start Time : 2018-10-29 09:46:19,260 INFO
org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST
RUN: Start grant
End Time : 2018-10-29 09:46:40,767 INFO
org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST
RUN: Completed grant
Total Time : 26.426 sec
IMPROVEMENT : Over 20 times faster
HDFS Sync OFF
Start Time : 2018-10-29 09:53:37,795 INFO
org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST
RUN: Start grant
End Time : 2018-10-29 09:53:56,433 INFO
org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST
RUN: Completed grant
Total Time : 22.932 sec
IMPROVEMENT : Over 20 times faster
{noformat}
> When granting privileges a single transaction per grant causes long delays
> --------------------------------------------------------------------------
>
> Key: SENTRY-2437
> URL: https://issues.apache.org/jira/browse/SENTRY-2437
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
> Reporter: Arjun Mishra
> Assignee: Arjun Mishra
> Priority: Major
>
> Currently sentry creates a transaction for each TSentryPrivilege object it
> needs to grant. If the list of privileges is very large creating a single
> transaction for each significantly affects performance. This is particularly
> impactful for tables with large columns and if a user grants privileges to
> many of those columns
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
