[jira] [Comment Edited] (SENTRY-1453) Enable passing sentry client cache configs from kafka conf
[ https://issues.apache.org/jira/browse/SENTRY-1453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15450323#comment-15450323 ] Sravya Tirukkovalur edited comment on SENTRY-1453 at 8/30/16 10:27 PM: --- [~singhashish] So there are some properties needed by Sentry binding of Kafka. Is the plan of this jira to have these sentry properties defined in kafka's configuration xml instead of sentry-site.xml available to Kafka deamon? From user point of view, it seems like all sentry properties need to be defined in sentry-site.xml? Not sure I follow why some of them needed to be defined in Kafka's config (except for the sentry plugin end point classes) was (Author: sravya): [~singhashish] So there are some properties needed by Sentry binding of Kafka. Is the plan of this jira to have these sentry properties defined in kafka's configuration xml instead of sentry-site.xml available to Kafka deamon? > Enable passing sentry client cache configs from kafka conf > -- > > Key: SENTRY-1453 > URL: https://issues.apache.org/jira/browse/SENTRY-1453 > Project: Sentry > Issue Type: Improvement > Components: kafka-integration >Reporter: Ashish K Singh >Assignee: Ashish K Singh > Attachments: SENTRY-1453.001.patch, SENTRY-1453.002.patch, > SENTRY-1453.003.patch > > > As it currently stands user will have to set some sentry related configs in > Kafka's config and some in Sentry's configs. Configuration can be eased by > allowing users to pass all related configs in one place. One way is to enable > passing sentry client cache configs from Kafka conf. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (SENTRY-1422) JDO deadlocks while processing grant while a background thread processes Notificationlogs
[
https://issues.apache.org/jira/browse/SENTRY-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15450712#comment-15450712
]
Alexander Kolbasov edited comment on SENTRY-1422 at 8/31/16 1:33 AM:
-
It is interesting to look at the use of SENTRY_ROLE_DB_PRIVILEGE_MAP and
CENTRY_DB_PRIVILEGE.
I only see references to SENTRY_ROLE_DB_PRIVILEGE_MAP for joins as part of the
definition of MSentryRole - no one uses it directly except
removeOrphanedPrivileges():
{code}
private void removeOrphanedPrivileges() throws SentryStandbyException {
final String privDB = "SENTRY_DB_PRIVILEGE";
final String privId = "DB_PRIVILEGE_ID";
final String mapDB = "SENTRY_ROLE_DB_PRIVILEGE_MAP";
final String privFilter =
"select " + privId +
" from " + privDB + " p" +
" where not exists (" +
" select 1 from " + mapDB + " d" +
" where p." + privId + " != d." + privId +
" )";
{code}
This is used here:
{code}
transaction.begin();
transaction.setRollbackOnly(); // Makes the tx read-only
act.checkSqlFencing(pm);
Query query = pm.newQuery("javax.jdo.query.SQL", privFilter);
query.setClass(MSentryPrivilege.class);
List results = (List) query.execute();
{code}
The definition for MSentryPrivilege has “privileges” field which does join with
SENTRY_ROLE_DB_PRIVILEGE_MAP.
Can this cause our deadlock troubles?
If we look at potential holders of SENTRY_ROLE_DB_PRIVILEGE_MAP we can see that
outside of the use above it is only used indirectly in joins (see package.jdo
file).
I also see the following interesting thing:
MSentryRole does join with SENTRY_ROLE_DB_PRIVILEGE_MAP. It also references
collection of MSentryPrivilege objects which has “roles” fields which is a
collection of MSentryRole objects. This looks somewhat suspicious - like a
circular dependency - can this cause any deadlocks?
Here is the package.jdo file for references:
{code}
{code}
was (Author: akolb):
It is interesting to look at the use of SENTRY_ROLE_DB_PRIVILEGE_MAP and
CENTRY_DB_PRIVILEGE.
I only see references to SENTRY_ROLE_DB_PRIVILEGE_MAP for joins as part of the
definition of MSentryRole - no one uses it directly except
removeOrphanedPrivileges():
{code}
private void removeOrphanedPrivileges() throws SentryStandbyException {
final String privDB = "SENTRY_DB_PRIVILEGE";
final String privId = "DB_PRIVILEGE_ID";
final String mapDB = "SENTRY_ROLE_DB_PRIVILEGE_MAP";
final String privFilter =
"select " + privId +
" from " + privDB + " p" +
" where not exists (" +
" select 1 from " + mapDB + " d" +
" where p." + privId + " != d." + privId +
" )";
{code}
This is used here:
{code}
transaction.begin();
transaction.setRollbackOnly(); // Makes the tx read-only
act.checkSqlFencing(pm);
Query query = pm.newQuery("javax.jdo.query.SQL", privFilter);
query.setClass(MSentryPrivilege.class);
List results = (List) query.execute();
{code}
The definition for MSentryPrivilege has “privileges” field which does join with
SENTRY_ROLE_DB_PRIVILEGE_MAP.
Can this cause our deadlock troubles?
If we look at potential holders of SENTRY_ROLE_DB_PRIVILEGE_MAP we can see that
outside of the use abovew
[jira] [Commented] (SENTRY-1453) Enable passing sentry client cache configs from kafka conf
[ https://issues.apache.org/jira/browse/SENTRY-1453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15450438#comment-15450438 ] Sravya Tirukkovalur commented on SENTRY-1453: - Just synced up with [~singhashish] offline. It is better to restrict client side sentry-site for basic configs required to connect to sentry service. All other client specific tuning parameters are best setup in the component as these can be configured differently in each client. +1 for the patch. > Enable passing sentry client cache configs from kafka conf > -- > > Key: SENTRY-1453 > URL: https://issues.apache.org/jira/browse/SENTRY-1453 > Project: Sentry > Issue Type: Improvement > Components: kafka-integration >Reporter: Ashish K Singh >Assignee: Ashish K Singh > Attachments: SENTRY-1453.001.patch, SENTRY-1453.002.patch, > SENTRY-1453.003.patch > > > As it currently stands user will have to set some sentry related configs in > Kafka's config and some in Sentry's configs. Configuration can be eased by > allowing users to pass all related configs in one place. One way is to enable > passing sentry client cache configs from Kafka conf. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SENTRY-1260) Improve error handling - ArrayIndexOutOfBoundsException in PathsUpdate.parsePath can cause MetastoreCacheInitializer intialization to fail
[
https://issues.apache.org/jira/browse/SENTRY-1260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15448390#comment-15448390
]
Venkat Sambath commented on SENTRY-1260:
Please let us know on which CDH version is this fix shipped.
> Improve error handling - ArrayIndexOutOfBoundsException in
> PathsUpdate.parsePath can cause MetastoreCacheInitializer intialization to
> fail
> --
>
> Key: SENTRY-1260
> URL: https://issues.apache.org/jira/browse/SENTRY-1260
> Project: Sentry
> Issue Type: Bug
>Reporter: Sravya Tirukkovalur
>Assignee: Sravya Tirukkovalur
> Attachments: SENTRY-1260.0.patch, SENTRY-1260.1.patch,
> SENTRY-1260.2.patch, SENTRY-1260.3.patch
>
>
> We should handle this gracefully. Instead of just throwing
> ArrayIndexOutOfBound, logging the path would be helpful
> {noformat}
> 2016-04-12 14:53:28,924 ERROR org.apache.sentry.hdfs.MetastorePlugin: [main]:
> Could not create Initial AuthzPaths or HMSHandler !!
> java.lang.RuntimeException: java.lang.ArrayIndexOutOfBoundsException: 1
> at
> org.apache.sentry.hdfs.MetastoreCacheInitializer.createInitialUpdate(MetastoreCacheInitializer.java:242)
> at org.apache.sentry.hdfs.MetastorePlugin$1.run(MetastorePlugin.java:160)
> at org.apache.sentry.hdfs.MetastorePlugin.(MetastorePlugin.java:197)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.sentry.binding.metastore.SentryMetastorePostEventListener.(SentryMetastorePostEventListener.java:78)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.getMetaStoreListeners(MetaStoreUtils.java:1439)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:485)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.(RetryingHMSHandler.java:78)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:84)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5775)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:5770)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore.startMetaStore(HiveMetaStore.java:6022)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore.main(HiveMetaStore.java:5947)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.lang.ArrayIndexOutOfBoundsException: 1
> at org.apache.sentry.hdfs.PathsUpdate.parsePath(PathsUpdate.java:109)
> at
> org.apache.sentry.hdfs.MetastoreCacheInitializer$DbTask.doTask(MetastoreCacheInitializer.java:176)
> at
> org.apache.sentry.hdfs.MetastoreCacheInitializer$BaseTask.call(MetastoreCacheInitializer.java:61)
> at
> org.apache.sentry.hdfs.MetastoreCacheInitializer$BaseTask.call(MetastoreCacheInitializer.java:53)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SENTRY-1422) JDO deadlocks while processing grant while a background thread processes Notificationlogs
[
https://issues.apache.org/jira/browse/SENTRY-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15448324#comment-15448324
]
Colin Ma commented on SENTRY-1422:
--
[~hahao], transaction will hold the exclusive locks during the transaction.
[refer
this|https://db.apache.org/derby/docs/10.0/manuals/develop/develop74.html]
So I think after 2 transactions get the exclusive locks, the following requests
for shard lock will be rejected and we get the deadlock. Sorry for my example
above doesn't make this clearly.
I also agree with Alexander that the different sequence of read and write will
cause the deadlock.
> JDO deadlocks while processing grant while a background thread processes
> Notificationlogs
> -
>
> Key: SENTRY-1422
> URL: https://issues.apache.org/jira/browse/SENTRY-1422
> Project: Sentry
> Issue Type: Sub-task
> Components: Hdfs Plugin
>Reporter: Sravya Tirukkovalur
>Assignee: Colin Ma
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1422.008-sentry-ha-redesign.patch,
> SENTRY-1422.009-sentry-ha-redesign.patch,
> SENTRY-1422.010-sentry-ha-redesign.patch
>
>
> As I was working on Sentry-1321. I see that test case
> TestDbPrivilegeCleanupOnDrop#testDropObjects fails while "Granting select on
> table" with following deadlock.
> {code}
> java.sql.SQLException: Error while processing statement: FAILED: Execution
> Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Unknown error for
> request: TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2,
> requestorUserName:admin1, roleName:select_tbl1,
> privileges:[TSentryPrivilege(privilegeScope:TABLE, serverName:server1,
> dbName:db_2, tableName:tb_1, URI:, action:select, createTime:1469562185573,
> grantOption:FALSE, columnName:)]), message: Iteration request failed : SELECT
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
> FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?. Server
> Stacktrace: javax.jdo.JDODataStoreException: Iteration request failed :
> SELECT 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
> FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
> at
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitTransaction(SentryStore.java:279)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitUpdateTransaction(SentryStore.java:261)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:458)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:261)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
> at
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> NestedThrowablesStackTrace:
> java.sql.SQLTransactionRollbackException: A lock could not be obtained due to
> a deadlock, cycle of locks and waiters is:
> Lock : ROW, SENTRY_ROLE_DB_PRIVILEGE_MAP, (1,10)
> Waiting XID : {712, S} , SENTRY, SELECT
>
