[jira] [Updated] (SENTRY-1422) JDO deadlocks while processing grant while a background thread processes Notificationlogs

[Colin Ma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Ma updated SENTRY-1422:
-
Attachment: SENTRY-1422.020-sentry-ha-redesign.patch

> JDO deadlocks while processing grant while a background thread processes 
> Notificationlogs
> -
>
> Key: SENTRY-1422
> URL: https://issues.apache.org/jira/browse/SENTRY-1422
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Hdfs Plugin
>Reporter: Sravya Tirukkovalur
>Assignee: Colin Ma
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1422.013-sentry-ha-redesign.patch, 
> SENTRY-1422.014-sentry-ha-redesign.patch, 
> SENTRY-1422.015-sentry-ha-redesign.patch, 
> SENTRY-1422.016-sentry-ha-redesign.patch, 
> SENTRY-1422.017-sentry-ha-redesign.patch, 
> SENTRY-1422.018-sentry-ha-redesign.patch, 
> SENTRY-1422.019-sentry-ha-redesign.patch, 
> SENTRY-1422.020-sentry-ha-redesign.patch, sentry_deadlock.png, 
> testDeadLock.patch
>
>
> As I was working on Sentry-1321. I see that test case 
> TestDbPrivilegeCleanupOnDrop#testDropObjects fails while "Granting select on 
> table" with following deadlock.
> {code}
> java.sql.SQLException: Error while processing statement: FAILED: Execution 
> Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Unknown error for 
> request: TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2, 
> requestorUserName:admin1, roleName:select_tbl1, 
> privileges:[TSentryPrivilege(privilegeScope:TABLE, serverName:server1, 
> dbName:db_2, tableName:tb_1, URI:, action:select, createTime:1469562185573, 
> grantOption:FALSE, columnName:)]), message: Iteration request failed : SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?. Server 
> Stacktrace: javax.jdo.JDODataStoreException: Iteration request failed : 
> SELECT 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
>   at 
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
>   at 
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitTransaction(SentryStore.java:279)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitUpdateTransaction(SentryStore.java:261)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:458)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:261)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
>   at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
>   at 
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
>   at 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> NestedThrowablesStackTrace:
> java.sql.SQLTransactionRollbackException: A lock could not be obtained due to 
> a deadlock, cycle of locks and waiters is:
> Lock : ROW, SENTRY_ROLE_DB_PRIVILEGE_MAP, (1,10)
>   Waiting XID : {712, S} , SENTRY, SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> 

[jira] [Updated] (SENTRY-1422) JDO deadlocks while processing grant while a background thread processes Notificationlogs

[Colin Ma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Ma updated SENTRY-1422:
-
Attachment: SENTRY-1422.019-sentry-ha-redesign.patch

> JDO deadlocks while processing grant while a background thread processes 
> Notificationlogs
> -
>
> Key: SENTRY-1422
> URL: https://issues.apache.org/jira/browse/SENTRY-1422
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Hdfs Plugin
>Reporter: Sravya Tirukkovalur
>Assignee: Colin Ma
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1422.013-sentry-ha-redesign.patch, 
> SENTRY-1422.014-sentry-ha-redesign.patch, 
> SENTRY-1422.015-sentry-ha-redesign.patch, 
> SENTRY-1422.016-sentry-ha-redesign.patch, 
> SENTRY-1422.017-sentry-ha-redesign.patch, 
> SENTRY-1422.018-sentry-ha-redesign.patch, 
> SENTRY-1422.019-sentry-ha-redesign.patch, sentry_deadlock.png, 
> testDeadLock.patch
>
>
> As I was working on Sentry-1321. I see that test case 
> TestDbPrivilegeCleanupOnDrop#testDropObjects fails while "Granting select on 
> table" with following deadlock.
> {code}
> java.sql.SQLException: Error while processing statement: FAILED: Execution 
> Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Unknown error for 
> request: TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2, 
> requestorUserName:admin1, roleName:select_tbl1, 
> privileges:[TSentryPrivilege(privilegeScope:TABLE, serverName:server1, 
> dbName:db_2, tableName:tb_1, URI:, action:select, createTime:1469562185573, 
> grantOption:FALSE, columnName:)]), message: Iteration request failed : SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?. Server 
> Stacktrace: javax.jdo.JDODataStoreException: Iteration request failed : 
> SELECT 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
>   at 
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
>   at 
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitTransaction(SentryStore.java:279)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitUpdateTransaction(SentryStore.java:261)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:458)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:261)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
>   at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
>   at 
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
>   at 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> NestedThrowablesStackTrace:
> java.sql.SQLTransactionRollbackException: A lock could not be obtained due to 
> a deadlock, cycle of locks and waiters is:
> Lock : ROW, SENTRY_ROLE_DB_PRIVILEGE_MAP, (1,10)
>   Waiting XID : {712, S} , SENTRY, SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 

[jira] [Updated] (SENTRY-1422) JDO deadlocks while processing grant while a background thread processes Notificationlogs

[Colin Ma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Ma updated SENTRY-1422:
-
Attachment: (was: SENTRY-1422.012-sentry-ha-redesign.patch)

> JDO deadlocks while processing grant while a background thread processes 
> Notificationlogs
> -
>
> Key: SENTRY-1422
> URL: https://issues.apache.org/jira/browse/SENTRY-1422
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Hdfs Plugin
>Reporter: Sravya Tirukkovalur
>Assignee: Colin Ma
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1422.013-sentry-ha-redesign.patch, 
> SENTRY-1422.014-sentry-ha-redesign.patch, 
> SENTRY-1422.015-sentry-ha-redesign.patch, 
> SENTRY-1422.016-sentry-ha-redesign.patch, 
> SENTRY-1422.017-sentry-ha-redesign.patch, 
> SENTRY-1422.018-sentry-ha-redesign.patch, sentry_deadlock.png, 
> testDeadLock.patch
>
>
> As I was working on Sentry-1321. I see that test case 
> TestDbPrivilegeCleanupOnDrop#testDropObjects fails while "Granting select on 
> table" with following deadlock.
> {code}
> java.sql.SQLException: Error while processing statement: FAILED: Execution 
> Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Unknown error for 
> request: TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2, 
> requestorUserName:admin1, roleName:select_tbl1, 
> privileges:[TSentryPrivilege(privilegeScope:TABLE, serverName:server1, 
> dbName:db_2, tableName:tb_1, URI:, action:select, createTime:1469562185573, 
> grantOption:FALSE, columnName:)]), message: Iteration request failed : SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?. Server 
> Stacktrace: javax.jdo.JDODataStoreException: Iteration request failed : 
> SELECT 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
>   at 
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
>   at 
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitTransaction(SentryStore.java:279)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitUpdateTransaction(SentryStore.java:261)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:458)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:261)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
>   at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
>   at 
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
>   at 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> NestedThrowablesStackTrace:
> java.sql.SQLTransactionRollbackException: A lock could not be obtained due to 
> a deadlock, cycle of locks and waiters is:
> Lock : ROW, SENTRY_ROLE_DB_PRIVILEGE_MAP, (1,10)
>   Waiting XID : {712, S} , SENTRY, SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 

[jira] [Updated] (SENTRY-1422) JDO deadlocks while processing grant while a background thread processes Notificationlogs

[Colin Ma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Ma updated SENTRY-1422:
-
Attachment: (was: SENTRY-1422.010-sentry-ha-redesign.patch)

> JDO deadlocks while processing grant while a background thread processes 
> Notificationlogs
> -
>
> Key: SENTRY-1422
> URL: https://issues.apache.org/jira/browse/SENTRY-1422
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Hdfs Plugin
>Reporter: Sravya Tirukkovalur
>Assignee: Colin Ma
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1422.013-sentry-ha-redesign.patch, 
> SENTRY-1422.014-sentry-ha-redesign.patch, 
> SENTRY-1422.015-sentry-ha-redesign.patch, 
> SENTRY-1422.016-sentry-ha-redesign.patch, 
> SENTRY-1422.017-sentry-ha-redesign.patch, 
> SENTRY-1422.018-sentry-ha-redesign.patch, sentry_deadlock.png, 
> testDeadLock.patch
>
>
> As I was working on Sentry-1321. I see that test case 
> TestDbPrivilegeCleanupOnDrop#testDropObjects fails while "Granting select on 
> table" with following deadlock.
> {code}
> java.sql.SQLException: Error while processing statement: FAILED: Execution 
> Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Unknown error for 
> request: TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2, 
> requestorUserName:admin1, roleName:select_tbl1, 
> privileges:[TSentryPrivilege(privilegeScope:TABLE, serverName:server1, 
> dbName:db_2, tableName:tb_1, URI:, action:select, createTime:1469562185573, 
> grantOption:FALSE, columnName:)]), message: Iteration request failed : SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?. Server 
> Stacktrace: javax.jdo.JDODataStoreException: Iteration request failed : 
> SELECT 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
>   at 
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
>   at 
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitTransaction(SentryStore.java:279)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitUpdateTransaction(SentryStore.java:261)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:458)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:261)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
>   at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
>   at 
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
>   at 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> NestedThrowablesStackTrace:
> java.sql.SQLTransactionRollbackException: A lock could not be obtained due to 
> a deadlock, cycle of locks and waiters is:
> Lock : ROW, SENTRY_ROLE_DB_PRIVILEGE_MAP, (1,10)
>   Waiting XID : {712, S} , SENTRY, SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 

[jira] [Updated] (SENTRY-1422) JDO deadlocks while processing grant while a background thread processes Notificationlogs

[Colin Ma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Ma updated SENTRY-1422:
-
Attachment: (was: SENTRY-1422.008-sentry-ha-redesign.patch)

> JDO deadlocks while processing grant while a background thread processes 
> Notificationlogs
> -
>
> Key: SENTRY-1422
> URL: https://issues.apache.org/jira/browse/SENTRY-1422
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Hdfs Plugin
>Reporter: Sravya Tirukkovalur
>Assignee: Colin Ma
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1422.010-sentry-ha-redesign.patch, 
> SENTRY-1422.011-sentry-ha-redesign.patch, 
> SENTRY-1422.012-sentry-ha-redesign.patch, 
> SENTRY-1422.013-sentry-ha-redesign.patch, 
> SENTRY-1422.014-sentry-ha-redesign.patch, 
> SENTRY-1422.015-sentry-ha-redesign.patch, 
> SENTRY-1422.016-sentry-ha-redesign.patch, 
> SENTRY-1422.017-sentry-ha-redesign.patch, 
> SENTRY-1422.018-sentry-ha-redesign.patch, sentry_deadlock.png, 
> testDeadLock.patch
>
>
> As I was working on Sentry-1321. I see that test case 
> TestDbPrivilegeCleanupOnDrop#testDropObjects fails while "Granting select on 
> table" with following deadlock.
> {code}
> java.sql.SQLException: Error while processing statement: FAILED: Execution 
> Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Unknown error for 
> request: TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2, 
> requestorUserName:admin1, roleName:select_tbl1, 
> privileges:[TSentryPrivilege(privilegeScope:TABLE, serverName:server1, 
> dbName:db_2, tableName:tb_1, URI:, action:select, createTime:1469562185573, 
> grantOption:FALSE, columnName:)]), message: Iteration request failed : SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?. Server 
> Stacktrace: javax.jdo.JDODataStoreException: Iteration request failed : 
> SELECT 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
>   at 
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
>   at 
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitTransaction(SentryStore.java:279)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitUpdateTransaction(SentryStore.java:261)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:458)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:261)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
>   at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
>   at 
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
>   at 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> NestedThrowablesStackTrace:
> java.sql.SQLTransactionRollbackException: A lock could not be obtained due to 
> a deadlock, cycle of locks and waiters is:
> Lock : ROW, SENTRY_ROLE_DB_PRIVILEGE_MAP, (1,10)
>   Waiting XID : {712, S} , SENTRY, SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> 

[jira] [Updated] (SENTRY-1422) JDO deadlocks while processing grant while a background thread processes Notificationlogs

[Colin Ma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Ma updated SENTRY-1422:
-
Attachment: SENTRY-1422.018-sentry-ha-redesign.patch

> JDO deadlocks while processing grant while a background thread processes 
> Notificationlogs
> -
>
> Key: SENTRY-1422
> URL: https://issues.apache.org/jira/browse/SENTRY-1422
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Hdfs Plugin
>Reporter: Sravya Tirukkovalur
>Assignee: Colin Ma
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1422.010-sentry-ha-redesign.patch, 
> SENTRY-1422.011-sentry-ha-redesign.patch, 
> SENTRY-1422.012-sentry-ha-redesign.patch, 
> SENTRY-1422.013-sentry-ha-redesign.patch, 
> SENTRY-1422.014-sentry-ha-redesign.patch, 
> SENTRY-1422.015-sentry-ha-redesign.patch, 
> SENTRY-1422.016-sentry-ha-redesign.patch, 
> SENTRY-1422.017-sentry-ha-redesign.patch, 
> SENTRY-1422.018-sentry-ha-redesign.patch, sentry_deadlock.png, 
> testDeadLock.patch
>
>
> As I was working on Sentry-1321. I see that test case 
> TestDbPrivilegeCleanupOnDrop#testDropObjects fails while "Granting select on 
> table" with following deadlock.
> {code}
> java.sql.SQLException: Error while processing statement: FAILED: Execution 
> Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Unknown error for 
> request: TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2, 
> requestorUserName:admin1, roleName:select_tbl1, 
> privileges:[TSentryPrivilege(privilegeScope:TABLE, serverName:server1, 
> dbName:db_2, tableName:tb_1, URI:, action:select, createTime:1469562185573, 
> grantOption:FALSE, columnName:)]), message: Iteration request failed : SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?. Server 
> Stacktrace: javax.jdo.JDODataStoreException: Iteration request failed : 
> SELECT 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
>   at 
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
>   at 
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitTransaction(SentryStore.java:279)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitUpdateTransaction(SentryStore.java:261)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:458)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:261)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
>   at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
>   at 
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
>   at 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> NestedThrowablesStackTrace:
> java.sql.SQLTransactionRollbackException: A lock could not be obtained due to 
> a deadlock, cycle of locks and waiters is:
> Lock : ROW, SENTRY_ROLE_DB_PRIVILEGE_MAP, (1,10)
>   Waiting XID : {712, S} , SENTRY, SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> 

[jira] [Updated] (SENTRY-1422) JDO deadlocks while processing grant while a background thread processes Notificationlogs

[Colin Ma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Ma updated SENTRY-1422:
-
Attachment: SENTRY-1422.017-sentry-ha-redesign.patch

> JDO deadlocks while processing grant while a background thread processes 
> Notificationlogs
> -
>
> Key: SENTRY-1422
> URL: https://issues.apache.org/jira/browse/SENTRY-1422
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Hdfs Plugin
>Reporter: Sravya Tirukkovalur
>Assignee: Colin Ma
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1422.008-sentry-ha-redesign.patch, 
> SENTRY-1422.009-sentry-ha-redesign.patch, 
> SENTRY-1422.010-sentry-ha-redesign.patch, 
> SENTRY-1422.011-sentry-ha-redesign.patch, 
> SENTRY-1422.012-sentry-ha-redesign.patch, 
> SENTRY-1422.013-sentry-ha-redesign.patch, 
> SENTRY-1422.014-sentry-ha-redesign.patch, 
> SENTRY-1422.015-sentry-ha-redesign.patch, 
> SENTRY-1422.016-sentry-ha-redesign.patch, 
> SENTRY-1422.017-sentry-ha-redesign.patch, sentry_deadlock.png, 
> testDeadLock.patch
>
>
> As I was working on Sentry-1321. I see that test case 
> TestDbPrivilegeCleanupOnDrop#testDropObjects fails while "Granting select on 
> table" with following deadlock.
> {code}
> java.sql.SQLException: Error while processing statement: FAILED: Execution 
> Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Unknown error for 
> request: TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2, 
> requestorUserName:admin1, roleName:select_tbl1, 
> privileges:[TSentryPrivilege(privilegeScope:TABLE, serverName:server1, 
> dbName:db_2, tableName:tb_1, URI:, action:select, createTime:1469562185573, 
> grantOption:FALSE, columnName:)]), message: Iteration request failed : SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?. Server 
> Stacktrace: javax.jdo.JDODataStoreException: Iteration request failed : 
> SELECT 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
>   at 
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
>   at 
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitTransaction(SentryStore.java:279)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitUpdateTransaction(SentryStore.java:261)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:458)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:261)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
>   at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
>   at 
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
>   at 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> NestedThrowablesStackTrace:
> java.sql.SQLTransactionRollbackException: A lock could not be obtained due to 
> a deadlock, cycle of locks and waiters is:
> Lock : ROW, SENTRY_ROLE_DB_PRIVILEGE_MAP, (1,10)
>   Waiting XID : {712, S} , SENTRY, SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> 

[jira] [Updated] (SENTRY-1422) JDO deadlocks while processing grant while a background thread processes Notificationlogs

[Colin Ma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Ma updated SENTRY-1422:
-
Attachment: SENTRY-1422.015-sentry-ha-redesign.patch

> JDO deadlocks while processing grant while a background thread processes 
> Notificationlogs
> -
>
> Key: SENTRY-1422
> URL: https://issues.apache.org/jira/browse/SENTRY-1422
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Hdfs Plugin
>Reporter: Sravya Tirukkovalur
>Assignee: Colin Ma
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1422.008-sentry-ha-redesign.patch, 
> SENTRY-1422.009-sentry-ha-redesign.patch, 
> SENTRY-1422.010-sentry-ha-redesign.patch, 
> SENTRY-1422.011-sentry-ha-redesign.patch, 
> SENTRY-1422.012-sentry-ha-redesign.patch, 
> SENTRY-1422.013-sentry-ha-redesign.patch, 
> SENTRY-1422.014-sentry-ha-redesign.patch, 
> SENTRY-1422.015-sentry-ha-redesign.patch, sentry_deadlock.png, 
> testDeadLock.patch
>
>
> As I was working on Sentry-1321. I see that test case 
> TestDbPrivilegeCleanupOnDrop#testDropObjects fails while "Granting select on 
> table" with following deadlock.
> {code}
> java.sql.SQLException: Error while processing statement: FAILED: Execution 
> Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Unknown error for 
> request: TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2, 
> requestorUserName:admin1, roleName:select_tbl1, 
> privileges:[TSentryPrivilege(privilegeScope:TABLE, serverName:server1, 
> dbName:db_2, tableName:tb_1, URI:, action:select, createTime:1469562185573, 
> grantOption:FALSE, columnName:)]), message: Iteration request failed : SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?. Server 
> Stacktrace: javax.jdo.JDODataStoreException: Iteration request failed : 
> SELECT 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
>   at 
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
>   at 
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitTransaction(SentryStore.java:279)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitUpdateTransaction(SentryStore.java:261)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:458)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:261)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
>   at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
>   at 
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
>   at 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> NestedThrowablesStackTrace:
> java.sql.SQLTransactionRollbackException: A lock could not be obtained due to 
> a deadlock, cycle of locks and waiters is:
> Lock : ROW, SENTRY_ROLE_DB_PRIVILEGE_MAP, (1,10)
>   Waiting XID : {712, S} , SENTRY, SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> 

[jira] [Updated] (SENTRY-1422) JDO deadlocks while processing grant while a background thread processes Notificationlogs

[Colin Ma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Ma updated SENTRY-1422:
-
Attachment: SENTRY-1422.015-sentry-ha-redesign.patch

> JDO deadlocks while processing grant while a background thread processes 
> Notificationlogs
> -
>
> Key: SENTRY-1422
> URL: https://issues.apache.org/jira/browse/SENTRY-1422
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Hdfs Plugin
>Reporter: Sravya Tirukkovalur
>Assignee: Colin Ma
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1422.008-sentry-ha-redesign.patch, 
> SENTRY-1422.009-sentry-ha-redesign.patch, 
> SENTRY-1422.010-sentry-ha-redesign.patch, 
> SENTRY-1422.011-sentry-ha-redesign.patch, 
> SENTRY-1422.012-sentry-ha-redesign.patch, 
> SENTRY-1422.013-sentry-ha-redesign.patch, 
> SENTRY-1422.014-sentry-ha-redesign.patch, 
> SENTRY-1422.015-sentry-ha-redesign.patch, sentry_deadlock.png, 
> testDeadLock.patch
>
>
> As I was working on Sentry-1321. I see that test case 
> TestDbPrivilegeCleanupOnDrop#testDropObjects fails while "Granting select on 
> table" with following deadlock.
> {code}
> java.sql.SQLException: Error while processing statement: FAILED: Execution 
> Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Unknown error for 
> request: TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2, 
> requestorUserName:admin1, roleName:select_tbl1, 
> privileges:[TSentryPrivilege(privilegeScope:TABLE, serverName:server1, 
> dbName:db_2, tableName:tb_1, URI:, action:select, createTime:1469562185573, 
> grantOption:FALSE, columnName:)]), message: Iteration request failed : SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?. Server 
> Stacktrace: javax.jdo.JDODataStoreException: Iteration request failed : 
> SELECT 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
>   at 
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
>   at 
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitTransaction(SentryStore.java:279)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitUpdateTransaction(SentryStore.java:261)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:458)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:261)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
>   at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
>   at 
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
>   at 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> NestedThrowablesStackTrace:
> java.sql.SQLTransactionRollbackException: A lock could not be obtained due to 
> a deadlock, cycle of locks and waiters is:
> Lock : ROW, SENTRY_ROLE_DB_PRIVILEGE_MAP, (1,10)
>   Waiting XID : {712, S} , SENTRY, SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> 

[jira] [Updated] (SENTRY-1422) JDO deadlocks while processing grant while a background thread processes Notificationlogs

[Colin Ma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Ma updated SENTRY-1422:
-
Attachment: SENTRY-1422.013-sentry-ha-redesign.patch

> JDO deadlocks while processing grant while a background thread processes 
> Notificationlogs
> -
>
> Key: SENTRY-1422
> URL: https://issues.apache.org/jira/browse/SENTRY-1422
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Hdfs Plugin
>Reporter: Sravya Tirukkovalur
>Assignee: Colin Ma
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1422.008-sentry-ha-redesign.patch, 
> SENTRY-1422.009-sentry-ha-redesign.patch, 
> SENTRY-1422.010-sentry-ha-redesign.patch, 
> SENTRY-1422.011-sentry-ha-redesign.patch, 
> SENTRY-1422.012-sentry-ha-redesign.patch, 
> SENTRY-1422.013-sentry-ha-redesign.patch, sentry_deadlock.png, 
> testDeadLock.patch
>
>
> As I was working on Sentry-1321. I see that test case 
> TestDbPrivilegeCleanupOnDrop#testDropObjects fails while "Granting select on 
> table" with following deadlock.
> {code}
> java.sql.SQLException: Error while processing statement: FAILED: Execution 
> Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Unknown error for 
> request: TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2, 
> requestorUserName:admin1, roleName:select_tbl1, 
> privileges:[TSentryPrivilege(privilegeScope:TABLE, serverName:server1, 
> dbName:db_2, tableName:tb_1, URI:, action:select, createTime:1469562185573, 
> grantOption:FALSE, columnName:)]), message: Iteration request failed : SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?. Server 
> Stacktrace: javax.jdo.JDODataStoreException: Iteration request failed : 
> SELECT 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
>   at 
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
>   at 
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitTransaction(SentryStore.java:279)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitUpdateTransaction(SentryStore.java:261)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:458)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:261)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
>   at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
>   at 
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
>   at 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> NestedThrowablesStackTrace:
> java.sql.SQLTransactionRollbackException: A lock could not be obtained due to 
> a deadlock, cycle of locks and waiters is:
> Lock : ROW, SENTRY_ROLE_DB_PRIVILEGE_MAP, (1,10)
>   Waiting XID : {712, S} , SENTRY, SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> 

[jira] [Updated] (SENTRY-1422) JDO deadlocks while processing grant while a background thread processes Notificationlogs

[Colin Ma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Ma updated SENTRY-1422:
-
Attachment: SENTRY-1422.012-sentry-ha-redesign.patch

> JDO deadlocks while processing grant while a background thread processes 
> Notificationlogs
> -
>
> Key: SENTRY-1422
> URL: https://issues.apache.org/jira/browse/SENTRY-1422
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Hdfs Plugin
>Reporter: Sravya Tirukkovalur
>Assignee: Colin Ma
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1422.008-sentry-ha-redesign.patch, 
> SENTRY-1422.009-sentry-ha-redesign.patch, 
> SENTRY-1422.010-sentry-ha-redesign.patch, 
> SENTRY-1422.011-sentry-ha-redesign.patch, 
> SENTRY-1422.012-sentry-ha-redesign.patch, sentry_deadlock.png, 
> testDeadLock.patch
>
>
> As I was working on Sentry-1321. I see that test case 
> TestDbPrivilegeCleanupOnDrop#testDropObjects fails while "Granting select on 
> table" with following deadlock.
> {code}
> java.sql.SQLException: Error while processing statement: FAILED: Execution 
> Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Unknown error for 
> request: TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2, 
> requestorUserName:admin1, roleName:select_tbl1, 
> privileges:[TSentryPrivilege(privilegeScope:TABLE, serverName:server1, 
> dbName:db_2, tableName:tb_1, URI:, action:select, createTime:1469562185573, 
> grantOption:FALSE, columnName:)]), message: Iteration request failed : SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?. Server 
> Stacktrace: javax.jdo.JDODataStoreException: Iteration request failed : 
> SELECT 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
>   at 
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
>   at 
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitTransaction(SentryStore.java:279)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitUpdateTransaction(SentryStore.java:261)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:458)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:261)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
>   at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
>   at 
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
>   at 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> NestedThrowablesStackTrace:
> java.sql.SQLTransactionRollbackException: A lock could not be obtained due to 
> a deadlock, cycle of locks and waiters is:
> Lock : ROW, SENTRY_ROLE_DB_PRIVILEGE_MAP, (1,10)
>   Waiting XID : {712, S} , SENTRY, SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE 

[jira] [Updated] (SENTRY-1422) JDO deadlocks while processing grant while a background thread processes Notificationlogs

[Colin Ma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Ma updated SENTRY-1422:
-
Attachment: SENTRY-1422.011-sentry-ha-redesign.patch

Update patch with the retry mechanism for SentryStore.

> JDO deadlocks while processing grant while a background thread processes 
> Notificationlogs
> -
>
> Key: SENTRY-1422
> URL: https://issues.apache.org/jira/browse/SENTRY-1422
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Hdfs Plugin
>Reporter: Sravya Tirukkovalur
>Assignee: Colin Ma
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1422.008-sentry-ha-redesign.patch, 
> SENTRY-1422.009-sentry-ha-redesign.patch, 
> SENTRY-1422.010-sentry-ha-redesign.patch, 
> SENTRY-1422.011-sentry-ha-redesign.patch, sentry_deadlock.png, 
> testDeadLock.patch
>
>
> As I was working on Sentry-1321. I see that test case 
> TestDbPrivilegeCleanupOnDrop#testDropObjects fails while "Granting select on 
> table" with following deadlock.
> {code}
> java.sql.SQLException: Error while processing statement: FAILED: Execution 
> Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Unknown error for 
> request: TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2, 
> requestorUserName:admin1, roleName:select_tbl1, 
> privileges:[TSentryPrivilege(privilegeScope:TABLE, serverName:server1, 
> dbName:db_2, tableName:tb_1, URI:, action:select, createTime:1469562185573, 
> grantOption:FALSE, columnName:)]), message: Iteration request failed : SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?. Server 
> Stacktrace: javax.jdo.JDODataStoreException: Iteration request failed : 
> SELECT 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
>   at 
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
>   at 
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitTransaction(SentryStore.java:279)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitUpdateTransaction(SentryStore.java:261)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:458)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:261)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
>   at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
>   at 
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
>   at 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> NestedThrowablesStackTrace:
> java.sql.SQLTransactionRollbackException: A lock could not be obtained due to 
> a deadlock, cycle of locks and waiters is:
> Lock : ROW, SENTRY_ROLE_DB_PRIVILEGE_MAP, (1,10)
>   Waiting XID : {712, S} , SENTRY, SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = 

[jira] [Updated] (SENTRY-1422) JDO deadlocks while processing grant while a background thread processes Notificationlogs

[Colin Ma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Ma updated SENTRY-1422:
-
Attachment: testDeadLock.patch

> JDO deadlocks while processing grant while a background thread processes 
> Notificationlogs
> -
>
> Key: SENTRY-1422
> URL: https://issues.apache.org/jira/browse/SENTRY-1422
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Hdfs Plugin
>Reporter: Sravya Tirukkovalur
>Assignee: Colin Ma
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1422.008-sentry-ha-redesign.patch, 
> SENTRY-1422.009-sentry-ha-redesign.patch, 
> SENTRY-1422.010-sentry-ha-redesign.patch, sentry_deadlock.png, 
> testDeadLock.patch
>
>
> As I was working on Sentry-1321. I see that test case 
> TestDbPrivilegeCleanupOnDrop#testDropObjects fails while "Granting select on 
> table" with following deadlock.
> {code}
> java.sql.SQLException: Error while processing statement: FAILED: Execution 
> Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Unknown error for 
> request: TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2, 
> requestorUserName:admin1, roleName:select_tbl1, 
> privileges:[TSentryPrivilege(privilegeScope:TABLE, serverName:server1, 
> dbName:db_2, tableName:tb_1, URI:, action:select, createTime:1469562185573, 
> grantOption:FALSE, columnName:)]), message: Iteration request failed : SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?. Server 
> Stacktrace: javax.jdo.JDODataStoreException: Iteration request failed : 
> SELECT 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
>   at 
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
>   at 
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitTransaction(SentryStore.java:279)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitUpdateTransaction(SentryStore.java:261)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:458)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:261)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
>   at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
>   at 
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
>   at 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> NestedThrowablesStackTrace:
> java.sql.SQLTransactionRollbackException: A lock could not be obtained due to 
> a deadlock, cycle of locks and waiters is:
> Lock : ROW, SENTRY_ROLE_DB_PRIVILEGE_MAP, (1,10)
>   Waiting XID : {712, S} , SENTRY, SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
>   Granted XID : {704, X} 
> Lock : ROW, SENTRY_DB_PRIVILEGE, (1,11)
>   Waiting XID : {704, S} , 

[jira] [Updated] (SENTRY-1422) JDO deadlocks while processing grant while a background thread processes Notificationlogs

[Alexander Kolbasov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexander Kolbasov updated SENTRY-1422:
---
Attachment: sentry_deadlock.png

> JDO deadlocks while processing grant while a background thread processes 
> Notificationlogs
> -
>
> Key: SENTRY-1422
> URL: https://issues.apache.org/jira/browse/SENTRY-1422
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Hdfs Plugin
>Reporter: Sravya Tirukkovalur
>Assignee: Colin Ma
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1422.008-sentry-ha-redesign.patch, 
> SENTRY-1422.009-sentry-ha-redesign.patch, 
> SENTRY-1422.010-sentry-ha-redesign.patch, sentry_deadlock.png
>
>
> As I was working on Sentry-1321. I see that test case 
> TestDbPrivilegeCleanupOnDrop#testDropObjects fails while "Granting select on 
> table" with following deadlock.
> {code}
> java.sql.SQLException: Error while processing statement: FAILED: Execution 
> Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Unknown error for 
> request: TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2, 
> requestorUserName:admin1, roleName:select_tbl1, 
> privileges:[TSentryPrivilege(privilegeScope:TABLE, serverName:server1, 
> dbName:db_2, tableName:tb_1, URI:, action:select, createTime:1469562185573, 
> grantOption:FALSE, columnName:)]), message: Iteration request failed : SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?. Server 
> Stacktrace: javax.jdo.JDODataStoreException: Iteration request failed : 
> SELECT 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
>   at 
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
>   at 
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitTransaction(SentryStore.java:279)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitUpdateTransaction(SentryStore.java:261)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:458)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:261)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
>   at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
>   at 
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
>   at 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> NestedThrowablesStackTrace:
> java.sql.SQLTransactionRollbackException: A lock could not be obtained due to 
> a deadlock, cycle of locks and waiters is:
> Lock : ROW, SENTRY_ROLE_DB_PRIVILEGE_MAP, (1,10)
>   Waiting XID : {712, S} , SENTRY, SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
>   Granted XID : {704, X} 
> Lock : ROW, SENTRY_DB_PRIVILEGE, (1,11)
>   Waiting XID : {704, S} , 

[jira] [Updated] (SENTRY-1422) JDO deadlocks while processing grant while a background thread processes Notificationlogs

[Colin Ma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Ma updated SENTRY-1422:
-
Attachment: SENTRY-1422.010-sentry-ha-redesign.patch

> JDO deadlocks while processing grant while a background thread processes 
> Notificationlogs
> -
>
> Key: SENTRY-1422
> URL: https://issues.apache.org/jira/browse/SENTRY-1422
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Hdfs Plugin
>Reporter: Sravya Tirukkovalur
>Assignee: Colin Ma
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1422.008-sentry-ha-redesign.patch, 
> SENTRY-1422.009-sentry-ha-redesign.patch, 
> SENTRY-1422.010-sentry-ha-redesign.patch
>
>
> As I was working on Sentry-1321. I see that test case 
> TestDbPrivilegeCleanupOnDrop#testDropObjects fails while "Granting select on 
> table" with following deadlock.
> {code}
> java.sql.SQLException: Error while processing statement: FAILED: Execution 
> Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Unknown error for 
> request: TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2, 
> requestorUserName:admin1, roleName:select_tbl1, 
> privileges:[TSentryPrivilege(privilegeScope:TABLE, serverName:server1, 
> dbName:db_2, tableName:tb_1, URI:, action:select, createTime:1469562185573, 
> grantOption:FALSE, columnName:)]), message: Iteration request failed : SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?. Server 
> Stacktrace: javax.jdo.JDODataStoreException: Iteration request failed : 
> SELECT 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
>   at 
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
>   at 
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitTransaction(SentryStore.java:279)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitUpdateTransaction(SentryStore.java:261)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:458)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:261)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
>   at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
>   at 
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
>   at 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> NestedThrowablesStackTrace:
> java.sql.SQLTransactionRollbackException: A lock could not be obtained due to 
> a deadlock, cycle of locks and waiters is:
> Lock : ROW, SENTRY_ROLE_DB_PRIVILEGE_MAP, (1,10)
>   Waiting XID : {712, S} , SENTRY, SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
>   Granted XID : {704, X} 
> Lock : ROW, SENTRY_DB_PRIVILEGE, (1,11)
>   Waiting XID : {704, S} , SENTRY, SELECT 
> 

[jira] [Updated] (SENTRY-1422) JDO deadlocks while processing grant while a background thread processes Notificationlogs

[Colin Ma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Ma updated SENTRY-1422:
-
Attachment: (was: SENTRY-1422.007-sentry-ha-redesign.patch)

> JDO deadlocks while processing grant while a background thread processes 
> Notificationlogs
> -
>
> Key: SENTRY-1422
> URL: https://issues.apache.org/jira/browse/SENTRY-1422
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Hdfs Plugin
>Reporter: Sravya Tirukkovalur
>Assignee: Colin Ma
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1422.008-sentry-ha-redesign.patch, 
> SENTRY-1422.009-sentry-ha-redesign.patch, 
> SENTRY-1422.010-sentry-ha-redesign.patch
>
>
> As I was working on Sentry-1321. I see that test case 
> TestDbPrivilegeCleanupOnDrop#testDropObjects fails while "Granting select on 
> table" with following deadlock.
> {code}
> java.sql.SQLException: Error while processing statement: FAILED: Execution 
> Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Unknown error for 
> request: TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2, 
> requestorUserName:admin1, roleName:select_tbl1, 
> privileges:[TSentryPrivilege(privilegeScope:TABLE, serverName:server1, 
> dbName:db_2, tableName:tb_1, URI:, action:select, createTime:1469562185573, 
> grantOption:FALSE, columnName:)]), message: Iteration request failed : SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?. Server 
> Stacktrace: javax.jdo.JDODataStoreException: Iteration request failed : 
> SELECT 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
>   at 
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
>   at 
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitTransaction(SentryStore.java:279)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitUpdateTransaction(SentryStore.java:261)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:458)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:261)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
>   at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
>   at 
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
>   at 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> NestedThrowablesStackTrace:
> java.sql.SQLTransactionRollbackException: A lock could not be obtained due to 
> a deadlock, cycle of locks and waiters is:
> Lock : ROW, SENTRY_ROLE_DB_PRIVILEGE_MAP, (1,10)
>   Waiting XID : {712, S} , SENTRY, SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
>   Granted XID : {704, X} 
> Lock : ROW, SENTRY_DB_PRIVILEGE, (1,11)
>   Waiting XID : {704, S} , SENTRY, 

[jira] [Updated] (SENTRY-1422) JDO deadlocks while processing grant while a background thread processes Notificationlogs

[Colin Ma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Ma updated SENTRY-1422:
-
Attachment: SENTRY-1422.009-sentry-ha-redesign.patch

> JDO deadlocks while processing grant while a background thread processes 
> Notificationlogs
> -
>
> Key: SENTRY-1422
> URL: https://issues.apache.org/jira/browse/SENTRY-1422
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Hdfs Plugin
>Reporter: Sravya Tirukkovalur
>Assignee: Colin Ma
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1422.007-sentry-ha-redesign.patch, 
> SENTRY-1422.008-sentry-ha-redesign.patch, 
> SENTRY-1422.009-sentry-ha-redesign.patch
>
>
> As I was working on Sentry-1321. I see that test case 
> TestDbPrivilegeCleanupOnDrop#testDropObjects fails while "Granting select on 
> table" with following deadlock.
> {code}
> java.sql.SQLException: Error while processing statement: FAILED: Execution 
> Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Unknown error for 
> request: TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2, 
> requestorUserName:admin1, roleName:select_tbl1, 
> privileges:[TSentryPrivilege(privilegeScope:TABLE, serverName:server1, 
> dbName:db_2, tableName:tb_1, URI:, action:select, createTime:1469562185573, 
> grantOption:FALSE, columnName:)]), message: Iteration request failed : SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?. Server 
> Stacktrace: javax.jdo.JDODataStoreException: Iteration request failed : 
> SELECT 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
>   at 
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
>   at 
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitTransaction(SentryStore.java:279)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitUpdateTransaction(SentryStore.java:261)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:458)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:261)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
>   at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
>   at 
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
>   at 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> NestedThrowablesStackTrace:
> java.sql.SQLTransactionRollbackException: A lock could not be obtained due to 
> a deadlock, cycle of locks and waiters is:
> Lock : ROW, SENTRY_ROLE_DB_PRIVILEGE_MAP, (1,10)
>   Waiting XID : {712, S} , SENTRY, SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
>   Granted XID : {704, X} 
> Lock : ROW, SENTRY_DB_PRIVILEGE, (1,11)
>   Waiting XID : {704, S} , SENTRY, SELECT 
> 

[jira] [Updated] (SENTRY-1422) JDO deadlocks while processing grant while a background thread processes Notificationlogs

[Colin Ma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Ma updated SENTRY-1422:
-
Attachment: (was: SENTRY-1422.006-sentry-ha-redesign.patch)

> JDO deadlocks while processing grant while a background thread processes 
> Notificationlogs
> -
>
> Key: SENTRY-1422
> URL: https://issues.apache.org/jira/browse/SENTRY-1422
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Hdfs Plugin
>Reporter: Sravya Tirukkovalur
>Assignee: Colin Ma
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1422.007-sentry-ha-redesign.patch, 
> SENTRY-1422.008-sentry-ha-redesign.patch
>
>
> As I was working on Sentry-1321. I see that test case 
> TestDbPrivilegeCleanupOnDrop#testDropObjects fails while "Granting select on 
> table" with following deadlock.
> {code}
> java.sql.SQLException: Error while processing statement: FAILED: Execution 
> Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Unknown error for 
> request: TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2, 
> requestorUserName:admin1, roleName:select_tbl1, 
> privileges:[TSentryPrivilege(privilegeScope:TABLE, serverName:server1, 
> dbName:db_2, tableName:tb_1, URI:, action:select, createTime:1469562185573, 
> grantOption:FALSE, columnName:)]), message: Iteration request failed : SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?. Server 
> Stacktrace: javax.jdo.JDODataStoreException: Iteration request failed : 
> SELECT 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
>   at 
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
>   at 
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitTransaction(SentryStore.java:279)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitUpdateTransaction(SentryStore.java:261)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:458)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:261)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
>   at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
>   at 
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
>   at 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> NestedThrowablesStackTrace:
> java.sql.SQLTransactionRollbackException: A lock could not be obtained due to 
> a deadlock, cycle of locks and waiters is:
> Lock : ROW, SENTRY_ROLE_DB_PRIVILEGE_MAP, (1,10)
>   Waiting XID : {712, S} , SENTRY, SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
>   Granted XID : {704, X} 
> Lock : ROW, SENTRY_DB_PRIVILEGE, (1,11)
>   Waiting XID : {704, S} , SENTRY, SELECT 
> 

[jira] [Updated] (SENTRY-1422) JDO deadlocks while processing grant while a background thread processes Notificationlogs

[Colin Ma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Ma updated SENTRY-1422:
-
Attachment: (was: SENTRY-1422.005-sentry-ha-redesign.patch)

> JDO deadlocks while processing grant while a background thread processes 
> Notificationlogs
> -
>
> Key: SENTRY-1422
> URL: https://issues.apache.org/jira/browse/SENTRY-1422
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Hdfs Plugin
>Reporter: Sravya Tirukkovalur
>Assignee: Colin Ma
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1422.006-sentry-ha-redesign.patch, 
> SENTRY-1422.007-sentry-ha-redesign.patch, 
> SENTRY-1422.008-sentry-ha-redesign.patch
>
>
> As I was working on Sentry-1321. I see that test case 
> TestDbPrivilegeCleanupOnDrop#testDropObjects fails while "Granting select on 
> table" with following deadlock.
> {code}
> java.sql.SQLException: Error while processing statement: FAILED: Execution 
> Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Unknown error for 
> request: TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2, 
> requestorUserName:admin1, roleName:select_tbl1, 
> privileges:[TSentryPrivilege(privilegeScope:TABLE, serverName:server1, 
> dbName:db_2, tableName:tb_1, URI:, action:select, createTime:1469562185573, 
> grantOption:FALSE, columnName:)]), message: Iteration request failed : SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?. Server 
> Stacktrace: javax.jdo.JDODataStoreException: Iteration request failed : 
> SELECT 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
>   at 
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
>   at 
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitTransaction(SentryStore.java:279)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitUpdateTransaction(SentryStore.java:261)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:458)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:261)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
>   at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
>   at 
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
>   at 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> NestedThrowablesStackTrace:
> java.sql.SQLTransactionRollbackException: A lock could not be obtained due to 
> a deadlock, cycle of locks and waiters is:
> Lock : ROW, SENTRY_ROLE_DB_PRIVILEGE_MAP, (1,10)
>   Waiting XID : {712, S} , SENTRY, SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
>   Granted XID : {704, X} 
> Lock : ROW, SENTRY_DB_PRIVILEGE, (1,11)
>   Waiting XID : {704, S} , SENTRY, 

[jira] [Updated] (SENTRY-1422) JDO deadlocks while processing grant while a background thread processes Notificationlogs

[Colin Ma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Ma updated SENTRY-1422:
-
Attachment: (was: SENTRY-1422.002-sentry-ha-redesign.patch)

> JDO deadlocks while processing grant while a background thread processes 
> Notificationlogs
> -
>
> Key: SENTRY-1422
> URL: https://issues.apache.org/jira/browse/SENTRY-1422
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Hdfs Plugin
>Reporter: Sravya Tirukkovalur
>Assignee: Colin Ma
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1422.004-sentry-ha-redesign.patch, 
> SENTRY-1422.005-sentry-ha-redesign.patch, 
> SENTRY-1422.006-sentry-ha-redesign.patch, 
> SENTRY-1422.007-sentry-ha-redesign.patch
>
>
> As I was working on Sentry-1321. I see that test case 
> TestDbPrivilegeCleanupOnDrop#testDropObjects fails while "Granting select on 
> table" with following deadlock.
> {code}
> java.sql.SQLException: Error while processing statement: FAILED: Execution 
> Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Unknown error for 
> request: TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2, 
> requestorUserName:admin1, roleName:select_tbl1, 
> privileges:[TSentryPrivilege(privilegeScope:TABLE, serverName:server1, 
> dbName:db_2, tableName:tb_1, URI:, action:select, createTime:1469562185573, 
> grantOption:FALSE, columnName:)]), message: Iteration request failed : SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?. Server 
> Stacktrace: javax.jdo.JDODataStoreException: Iteration request failed : 
> SELECT 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
>   at 
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
>   at 
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitTransaction(SentryStore.java:279)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitUpdateTransaction(SentryStore.java:261)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:458)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:261)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
>   at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
>   at 
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
>   at 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> NestedThrowablesStackTrace:
> java.sql.SQLTransactionRollbackException: A lock could not be obtained due to 
> a deadlock, cycle of locks and waiters is:
> Lock : ROW, SENTRY_ROLE_DB_PRIVILEGE_MAP, (1,10)
>   Waiting XID : {712, S} , SENTRY, SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
>   Granted XID : {704, X} 
> Lock : ROW, SENTRY_DB_PRIVILEGE, 

[jira] [Updated] (SENTRY-1422) JDO deadlocks while processing grant while a background thread processes Notificationlogs

[Colin Ma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Ma updated SENTRY-1422:
-
Attachment: SENTRY-1422.006-sentry-ha-redesign.patch

> JDO deadlocks while processing grant while a background thread processes 
> Notificationlogs
> -
>
> Key: SENTRY-1422
> URL: https://issues.apache.org/jira/browse/SENTRY-1422
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Hdfs Plugin
>Reporter: Sravya Tirukkovalur
>Assignee: Colin Ma
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1422.001-sentry-ha-redesign.patch, 
> SENTRY-1422.002-sentry-ha-redesign.patch, 
> SENTRY-1422.003-sentry-ha-redesign.patch, 
> SENTRY-1422.004-sentry-ha-redesign.patch, 
> SENTRY-1422.005-sentry-ha-redesign.patch, 
> SENTRY-1422.006-sentry-ha-redesign.patch
>
>
> As I was working on Sentry-1321. I see that test case 
> TestDbPrivilegeCleanupOnDrop#testDropObjects fails while "Granting select on 
> table" with following deadlock.
> {code}
> java.sql.SQLException: Error while processing statement: FAILED: Execution 
> Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Unknown error for 
> request: TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2, 
> requestorUserName:admin1, roleName:select_tbl1, 
> privileges:[TSentryPrivilege(privilegeScope:TABLE, serverName:server1, 
> dbName:db_2, tableName:tb_1, URI:, action:select, createTime:1469562185573, 
> grantOption:FALSE, columnName:)]), message: Iteration request failed : SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?. Server 
> Stacktrace: javax.jdo.JDODataStoreException: Iteration request failed : 
> SELECT 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
>   at 
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
>   at 
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitTransaction(SentryStore.java:279)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitUpdateTransaction(SentryStore.java:261)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:458)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:261)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
>   at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
>   at 
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
>   at 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> NestedThrowablesStackTrace:
> java.sql.SQLTransactionRollbackException: A lock could not be obtained due to 
> a deadlock, cycle of locks and waiters is:
> Lock : ROW, SENTRY_ROLE_DB_PRIVILEGE_MAP, (1,10)
>   Waiting XID : {712, S} , SENTRY, SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE 

[jira] [Updated] (SENTRY-1422) JDO deadlocks while processing grant while a background thread processes Notificationlogs

[Colin Ma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Ma updated SENTRY-1422:
-
Attachment: SENTRY-1422.005-sentry-ha-redesign.patch

> JDO deadlocks while processing grant while a background thread processes 
> Notificationlogs
> -
>
> Key: SENTRY-1422
> URL: https://issues.apache.org/jira/browse/SENTRY-1422
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Hdfs Plugin
>Reporter: Sravya Tirukkovalur
>Assignee: Colin Ma
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1422.001-sentry-ha-redesign.patch, 
> SENTRY-1422.002-sentry-ha-redesign.patch, 
> SENTRY-1422.003-sentry-ha-redesign.patch, 
> SENTRY-1422.004-sentry-ha-redesign.patch, 
> SENTRY-1422.005-sentry-ha-redesign.patch
>
>
> As I was working on Sentry-1321. I see that test case 
> TestDbPrivilegeCleanupOnDrop#testDropObjects fails while "Granting select on 
> table" with following deadlock.
> {code}
> java.sql.SQLException: Error while processing statement: FAILED: Execution 
> Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Unknown error for 
> request: TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2, 
> requestorUserName:admin1, roleName:select_tbl1, 
> privileges:[TSentryPrivilege(privilegeScope:TABLE, serverName:server1, 
> dbName:db_2, tableName:tb_1, URI:, action:select, createTime:1469562185573, 
> grantOption:FALSE, columnName:)]), message: Iteration request failed : SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?. Server 
> Stacktrace: javax.jdo.JDODataStoreException: Iteration request failed : 
> SELECT 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
>   at 
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
>   at 
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitTransaction(SentryStore.java:279)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitUpdateTransaction(SentryStore.java:261)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:458)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:261)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
>   at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
>   at 
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
>   at 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> NestedThrowablesStackTrace:
> java.sql.SQLTransactionRollbackException: A lock could not be obtained due to 
> a deadlock, cycle of locks and waiters is:
> Lock : ROW, SENTRY_ROLE_DB_PRIVILEGE_MAP, (1,10)
>   Waiting XID : {712, S} , SENTRY, SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
>   Granted XID : {704, X} 
> 

[jira] [Updated] (SENTRY-1422) JDO deadlocks while processing grant while a background thread processes Notificationlogs

[Colin Ma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Ma updated SENTRY-1422:
-
Attachment: SENTRY-1422.004-sentry-ha-redesign.patch

> JDO deadlocks while processing grant while a background thread processes 
> Notificationlogs
> -
>
> Key: SENTRY-1422
> URL: https://issues.apache.org/jira/browse/SENTRY-1422
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Hdfs Plugin
>Reporter: Sravya Tirukkovalur
>Assignee: Colin Ma
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1422.001-sentry-ha-redesign.patch, 
> SENTRY-1422.002-sentry-ha-redesign.patch, 
> SENTRY-1422.003-sentry-ha-redesign.patch, 
> SENTRY-1422.004-sentry-ha-redesign.patch
>
>
> As I was working on Sentry-1321. I see that test case 
> TestDbPrivilegeCleanupOnDrop#testDropObjects fails while "Granting select on 
> table" with following deadlock.
> {code}
> java.sql.SQLException: Error while processing statement: FAILED: Execution 
> Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Unknown error for 
> request: TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2, 
> requestorUserName:admin1, roleName:select_tbl1, 
> privileges:[TSentryPrivilege(privilegeScope:TABLE, serverName:server1, 
> dbName:db_2, tableName:tb_1, URI:, action:select, createTime:1469562185573, 
> grantOption:FALSE, columnName:)]), message: Iteration request failed : SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?. Server 
> Stacktrace: javax.jdo.JDODataStoreException: Iteration request failed : 
> SELECT 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
>   at 
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
>   at 
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitTransaction(SentryStore.java:279)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitUpdateTransaction(SentryStore.java:261)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:458)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:261)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
>   at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
>   at 
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
>   at 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> NestedThrowablesStackTrace:
> java.sql.SQLTransactionRollbackException: A lock could not be obtained due to 
> a deadlock, cycle of locks and waiters is:
> Lock : ROW, SENTRY_ROLE_DB_PRIVILEGE_MAP, (1,10)
>   Waiting XID : {712, S} , SENTRY, SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
>   Granted XID : {704, X} 
> Lock : ROW, SENTRY_DB_PRIVILEGE, (1,11)
>   

[jira] [Updated] (SENTRY-1422) JDO deadlocks while processing grant while a background thread processes Notificationlogs

[Colin Ma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Ma updated SENTRY-1422:
-
Attachment: SENTRY-1422.003-sentry-ha-redesign.patch

> JDO deadlocks while processing grant while a background thread processes 
> Notificationlogs
> -
>
> Key: SENTRY-1422
> URL: https://issues.apache.org/jira/browse/SENTRY-1422
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Hdfs Plugin
>Reporter: Sravya Tirukkovalur
>Assignee: Colin Ma
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1422.001-sentry-ha-redesign.patch, 
> SENTRY-1422.002-sentry-ha-redesign.patch, 
> SENTRY-1422.003-sentry-ha-redesign.patch
>
>
> As I was working on Sentry-1321. I see that test case 
> TestDbPrivilegeCleanupOnDrop#testDropObjects fails while "Granting select on 
> table" with following deadlock.
> {code}
> java.sql.SQLException: Error while processing statement: FAILED: Execution 
> Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Unknown error for 
> request: TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2, 
> requestorUserName:admin1, roleName:select_tbl1, 
> privileges:[TSentryPrivilege(privilegeScope:TABLE, serverName:server1, 
> dbName:db_2, tableName:tb_1, URI:, action:select, createTime:1469562185573, 
> grantOption:FALSE, columnName:)]), message: Iteration request failed : SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?. Server 
> Stacktrace: javax.jdo.JDODataStoreException: Iteration request failed : 
> SELECT 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
>   at 
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
>   at 
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitTransaction(SentryStore.java:279)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitUpdateTransaction(SentryStore.java:261)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:458)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:261)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
>   at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
>   at 
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
>   at 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> NestedThrowablesStackTrace:
> java.sql.SQLTransactionRollbackException: A lock could not be obtained due to 
> a deadlock, cycle of locks and waiters is:
> Lock : ROW, SENTRY_ROLE_DB_PRIVILEGE_MAP, (1,10)
>   Waiting XID : {712, S} , SENTRY, SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
>   Granted XID : {704, X} 
> Lock : ROW, SENTRY_DB_PRIVILEGE, (1,11)
>   Waiting XID : {704, S} , SENTRY, SELECT 
> 

[jira] [Updated] (SENTRY-1422) JDO deadlocks while processing grant while a background thread processes Notificationlogs

[Colin Ma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Ma updated SENTRY-1422:
-
Attachment: SENTRY-1422.002-sentry-ha-redesign.patch

> JDO deadlocks while processing grant while a background thread processes 
> Notificationlogs
> -
>
> Key: SENTRY-1422
> URL: https://issues.apache.org/jira/browse/SENTRY-1422
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Hdfs Plugin
>Reporter: Sravya Tirukkovalur
>Assignee: Colin Ma
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1422.001-sentry-ha-redesign.patch, 
> SENTRY-1422.002-sentry-ha-redesign.patch
>
>
> As I was working on Sentry-1321. I see that test case 
> TestDbPrivilegeCleanupOnDrop#testDropObjects fails while "Granting select on 
> table" with following deadlock.
> {code}
> java.sql.SQLException: Error while processing statement: FAILED: Execution 
> Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Unknown error for 
> request: TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2, 
> requestorUserName:admin1, roleName:select_tbl1, 
> privileges:[TSentryPrivilege(privilegeScope:TABLE, serverName:server1, 
> dbName:db_2, tableName:tb_1, URI:, action:select, createTime:1469562185573, 
> grantOption:FALSE, columnName:)]), message: Iteration request failed : SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?. Server 
> Stacktrace: javax.jdo.JDODataStoreException: Iteration request failed : 
> SELECT 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
>   at 
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
>   at 
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitTransaction(SentryStore.java:279)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitUpdateTransaction(SentryStore.java:261)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:458)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:261)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
>   at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
>   at 
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
>   at 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> NestedThrowablesStackTrace:
> java.sql.SQLTransactionRollbackException: A lock could not be obtained due to 
> a deadlock, cycle of locks and waiters is:
> Lock : ROW, SENTRY_ROLE_DB_PRIVILEGE_MAP, (1,10)
>   Waiting XID : {712, S} , SENTRY, SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
>   Granted XID : {704, X} 
> Lock : ROW, SENTRY_DB_PRIVILEGE, (1,11)
>   Waiting XID : {704, S} , SENTRY, SELECT 
> 

[jira] [Updated] (SENTRY-1422) JDO deadlocks while processing grant while a background thread processes Notificationlogs

[Colin Ma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/SENTRY-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin Ma updated SENTRY-1422:
-
Attachment: SENTRY-1422.001-sentry-ha-redesign.patch

> JDO deadlocks while processing grant while a background thread processes 
> Notificationlogs
> -
>
> Key: SENTRY-1422
> URL: https://issues.apache.org/jira/browse/SENTRY-1422
> Project: Sentry
>  Issue Type: Sub-task
>  Components: Hdfs Plugin
>Reporter: Sravya Tirukkovalur
>Assignee: Colin Ma
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1422.001-sentry-ha-redesign.patch
>
>
> As I was working on Sentry-1321. I see that test case 
> TestDbPrivilegeCleanupOnDrop#testDropObjects fails while "Granting select on 
> table" with following deadlock.
> {code}
> java.sql.SQLException: Error while processing statement: FAILED: Execution 
> Error, return code 1 from 
> org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Unknown error for 
> request: TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2, 
> requestorUserName:admin1, roleName:select_tbl1, 
> privileges:[TSentryPrivilege(privilegeScope:TABLE, serverName:server1, 
> dbName:db_2, tableName:tb_1, URI:, action:select, createTime:1469562185573, 
> grantOption:FALSE, columnName:)]), message: Iteration request failed : SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?. Server 
> Stacktrace: javax.jdo.JDODataStoreException: Iteration request failed : 
> SELECT 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
>   at 
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
>   at 
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitTransaction(SentryStore.java:279)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitUpdateTransaction(SentryStore.java:261)
>   at 
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:458)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:261)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
>   at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
>   at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
>   at 
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
>   at 
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
>   at 
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:745)
> NestedThrowablesStackTrace:
> java.sql.SQLTransactionRollbackException: A lock could not be obtained due to 
> a deadlock, cycle of locks and waiters is:
> Lock : ROW, SENTRY_ROLE_DB_PRIVILEGE_MAP, (1,10)
>   Waiting XID : {712, S} , SENTRY, SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
>  FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON 
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
>   Granted XID : {704, X} 
> Lock : ROW, SENTRY_DB_PRIVILEGE, (1,11)
>   Waiting XID : {704, S} , SENTRY, SELECT 
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS 
>