[ https://issues.apache.org/jira/browse/SENTRY-2437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16667445#comment-16667445 ]
Arjun Mishra edited comment on SENTRY-2437 at 10/29/18 4:58 PM: ---------------------------------------------------------------- Below is the summary of the impact of having a single transaction. We are seeing improvement of over 20 times faster. The test was done on a table with 1800 columns and a grant was made to each of those columns in a single statement like _GRANT SELECT(co1, col2, ...., col1800) ON TABLE_ {noformat} Current Sentry Code: HDFS Sync ON Start Time : 2018-10-29 08:26:15,519 INFO org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST RUN: Start grant Retry(1) : 2018-10-29 08:29:35,663 INFO org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST RUN: Start grant Retry(2) : 2018-10-29 08:32:55,815 INFO org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST RUN: Start grant End Time(1) : 2018-10-29 08:34:05,662 INFO org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST RUN: Completed grant End Time(2) : 2018-10-29 08:34:06,992 INFO org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST RUN: Completed grant End Time(3) : 2018-10-29 08:35:25,798 INFO org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST RUN: Completed grant Total Time : 554.91 sec. After all retries were completed HDFS Sync OFF Start Time : 2018-10-29 08:42:13,944 INFO org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST RUN: Start grant Retry(1) : 2018-10-29 08:45:34,141 INFO org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST RUN: Start grant Retry(2) : 2018-10-29 08:48:54,283 INFO org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST RUN: Start grant End Time(1) : 2018-10-29 08:49:58,434 INFO org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST RUN: Completed grant End Time(2) : 2018-10-29 08:50:15,245 INFO org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST RUN: Completed grant End Time(3) : 2018-10-29 08:50:18,654 INFO org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST RUN: Completed grant Total Time : 489.059 sec. After all retries were completed Code change with a single transaction: HDFS Sync ON Start Time : 2018-10-29 09:46:19,260 INFO org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST RUN: Start grant End Time : 2018-10-29 09:46:40,767 INFO org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST RUN: Completed grant Total Time : 26.426 sec IMPROVEMENT : Over 20 times faster HDFS Sync OFF Start Time : 2018-10-29 09:53:37,795 INFO org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST RUN: Start grant End Time : 2018-10-29 09:53:56,433 INFO org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST RUN: Completed grant Total Time : 22.932 sec IMPROVEMENT : Over 20 times faster {noformat} was (Author: arjunmishra13): Below is the summary of the impact of having a single transaction. We are seeing improvement of over 20 times faster. The test was done on a table with 1800 columns and a grant was made to each of those columns in a single statement like _GRANT SELECT(co1, col2, ...., col1800) ON TABLE_ {noformat} Current Sentry Code: HDFS Sync ON Start Time : 2018-10-29 08:26:15,519 INFO org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST RUN: Start grant Retry(1) : 2018-10-29 08:29:35,663 INFO org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST RUN: Start grant Retry(2) : 2018-10-29 08:32:55,815 INFO org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST RUN: Start grant End Time(1) : 2018-10-29 08:34:05,662 INFO org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST RUN: Completed grant End Time(2) : 2018-10-29 08:34:06,992 INFO org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST RUN: Completed grant End Time(3) : 2018-10-29 08:35:25,798 INFO org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST RUN: Completed grant Total Time : 554.91 sec. After all retries were completed HDFS Sync OFF Start Time : 2018-10-29 08:42:13,944 INFO org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST RUN: Start grant Retry(1) : 2018-10-29 08:45:34,141 INFO org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST RUN: Start grant Retry(2) : 2018-10-29 08:48:54,283 INFO org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST RUN: Start grant End Time(1) : 2018-10-29 08:49:58,434 INFO org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST RUN: Completed grant End Time(2) : 2018-10-29 08:50:15,245 INFO org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST RUN: Completed grant End Time(3) : 2018-10-29 08:50:18,654 INFO org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST RUN: Completed grant Total Time : 489.059 sec. After all retries were completed Code change with a single transaction: HDFS Sync ON Start Time : 2018-10-29 09:46:19,260 INFO org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST RUN: Start grant End Time : 2018-10-29 09:46:40,767 INFO org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST RUN: Completed grant Total Time : 26.426 sec IMPROVEMENT : Over 20 times faster HDFS Sync OFF Start Time : 2018-10-29 09:53:37,795 INFO org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST RUN: Start grant End Time : 2018-10-29 09:53:56,433 INFO org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor: TEST RUN: Completed grant Total Time : 22.932 sec IMPROVEMENT : Over 20 times faster {noformat} > When granting privileges a single transaction per grant causes long delays > -------------------------------------------------------------------------- > > Key: SENTRY-2437 > URL: https://issues.apache.org/jira/browse/SENTRY-2437 > Project: Sentry > Issue Type: Bug > Components: Sentry > Reporter: Arjun Mishra > Assignee: Arjun Mishra > Priority: Major > > Currently sentry creates a transaction for each TSentryPrivilege object it > needs to grant. If the list of privileges is very large creating a single > transaction for each significantly affects performance. This is particularly > impactful for tables with large columns and if a user grants privileges to > many of those columns -- This message was sent by Atlassian JIRA (v7.6.3#76005)