[jira] [Comment Edited] (SENTRY-1422) JDO deadlocks while processing grant while a background thread processes Notificationlogs
[
https://issues.apache.org/jira/browse/SENTRY-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15532152#comment-15532152
]
Colin Ma edited comment on SENTRY-1422 at 9/30/16 6:39 AM:
---
[~akolb], here is the investigation result for this problem:
1. How to reproduce?
Create 2 privileges and 1 role, grant 2 privileges independently in one thread,
revoke 2 privileges independently in other thread. The exception for deadlock
will be thrown.
2. What's the root cause?
For example, one thread grant privilege1 to role and get the X lock for
privilege1, the other thread revoke privilege2 from role and get the X lock for
privilege2. In both transaction, when do the commit in datanucleus, there has a
select SQL to query the privilege1 and privilege2, but the locks are kept by
different transaction. So the deadlock happened. The following log shows where
call the select:
{code}
at
org.datanucleus.state.AbstractStateManager.loadFieldsInFetchPlan(AbstractStateManager.java:1064)
at
org.datanucleus.ExecutionContextImpl.performDetachAllOnTxnEndPreparation(ExecutionContextImpl.java:4574)
at
org.datanucleus.ExecutionContextImpl.preCommit(ExecutionContextImpl.java:4259)
{code}
3. Try to set datanucleus.DetachAllOnCommit=false, but this will impact current
implementation, how about adding retry mechanism to fix this problem?
What do you think?
was (Author: colinma):
[~akolb], here is the investigation result for this problem:
1. How to reproduce?
Create 2 privileges and 1 role, grant 2 privileges independently in one thread,
revoke 2 privileges independently in other thread. The exception for deadlock
will be thrown.
2. What's the root cause?
For example, one thread grant privilege1 to role and get the X lock for
privilege1, the other thread revoke privilege2 from role and get the X lock for
privilege2. In both transaction, when do the commit in datanucleus, there has a
select SQL to query the privilege1 and privilege2, but the locks are kept by
different transaction. So the deadlock happened. The following log shows where
call the select:
{code}
at
org.datanucleus.state.AbstractStateManager.loadFieldsInFetchPlan(AbstractStateManager.java:1064)
at
org.datanucleus.ExecutionContextImpl.performDetachAllOnTxnEndPreparation(ExecutionContextImpl.java:4574)
at
org.datanucleus.ExecutionContextImpl.preCommit(ExecutionContextImpl.java:4259)
{code}
3. Suggest solution: go through the SentryStore, we don't use the instances of
MPrivileges, MRole after transaction, so, set
datanucleus.DetachAllOnCommit=false can avoid this deadlock problem.
What do you think?
> JDO deadlocks while processing grant while a background thread processes
> Notificationlogs
> -
>
> Key: SENTRY-1422
> URL: https://issues.apache.org/jira/browse/SENTRY-1422
> Project: Sentry
> Issue Type: Sub-task
> Components: Hdfs Plugin
>Reporter: Sravya Tirukkovalur
>Assignee: Colin Ma
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1422.008-sentry-ha-redesign.patch,
> SENTRY-1422.009-sentry-ha-redesign.patch,
> SENTRY-1422.010-sentry-ha-redesign.patch, sentry_deadlock.png
>
>
> As I was working on Sentry-1321. I see that test case
> TestDbPrivilegeCleanupOnDrop#testDropObjects fails while "Granting select on
> table" with following deadlock.
> {code}
> java.sql.SQLException: Error while processing statement: FAILED: Execution
> Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Unknown error for
> request: TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2,
> requestorUserName:admin1, roleName:select_tbl1,
> privileges:[TSentryPrivilege(privilegeScope:TABLE, serverName:server1,
> dbName:db_2, tableName:tb_1, URI:, action:select, createTime:1469562185573,
> grantOption:FALSE, columnName:)]), message: Iteration request failed : SELECT
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
> FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?. Server
> Stacktrace: javax.jdo.JDODataStoreException: Iteration request failed :
> SELECT 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
> FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID W
[jira] [Comment Edited] (SENTRY-1422) JDO deadlocks while processing grant while a background thread processes Notificationlogs
[
https://issues.apache.org/jira/browse/SENTRY-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15532152#comment-15532152
]
Colin Ma edited comment on SENTRY-1422 at 9/29/16 8:29 AM:
---
[~akolb], here is the investigation result for this problem:
1. How to reproduce?
Create 2 privileges and 1 role, grant 2 privileges independently in one thread,
revoke 2 privileges independently in other thread. The exception for deadlock
will be thrown.
2. What's the root cause?
For example, one thread grant privilege1 to role and get the X lock for
privilege1, the other thread revoke privilege2 from role and get the X lock for
privilege2. In both transaction, when do the commit in datanucleus, there has a
select SQL to query the privilege1 and privilege2, but the locks are kept by
different transaction. So the deadlock happened. The following log shows where
call the select:
{code}
at
org.datanucleus.state.AbstractStateManager.loadFieldsInFetchPlan(AbstractStateManager.java:1064)
at
org.datanucleus.ExecutionContextImpl.performDetachAllOnTxnEndPreparation(ExecutionContextImpl.java:4574)
at
org.datanucleus.ExecutionContextImpl.preCommit(ExecutionContextImpl.java:4259)
{code}
3. Suggest solution: go through the SentryStore, we don't use the instances of
MPrivileges, MRole after transaction, so, set
datanucleus.DetachAllOnCommit=false can avoid this deadlock problem.
What do you think?
was (Author: colinma):
[~akolb], here is the investigation result for this problems:
1. How to reproduce?
Create 2 privileges and 1 role, grant 2 privileges independently in one thread,
revoke 2 privileges independently in other thread. The exception for deadlock
will be thrown.
2. What's the root cause?
For example, one thread grant privilege1 to role and get the X lock for
privilege1, the other thread revoke privilege2 from role and get the X lock for
privilege2. In both transaction, when do the commit in datanucleus, there has a
select SQL to query the privilege1 and privilege2, but the locks are kept by
different transaction. So the deadlock happened. The following log shows where
call the select:
{code}
at
org.datanucleus.state.AbstractStateManager.loadFieldsInFetchPlan(AbstractStateManager.java:1064)
at
org.datanucleus.ExecutionContextImpl.performDetachAllOnTxnEndPreparation(ExecutionContextImpl.java:4574)
at
org.datanucleus.ExecutionContextImpl.preCommit(ExecutionContextImpl.java:4259)
{code}
3. Suggest solution: go through the SentryStore, we don't use the instances of
MPrivileges, MRole after transaction, so, set
datanucleus.DetachAllOnCommit=false can avoid this deadlock problem.
What do you think?
> JDO deadlocks while processing grant while a background thread processes
> Notificationlogs
> -
>
> Key: SENTRY-1422
> URL: https://issues.apache.org/jira/browse/SENTRY-1422
> Project: Sentry
> Issue Type: Sub-task
> Components: Hdfs Plugin
>Reporter: Sravya Tirukkovalur
>Assignee: Colin Ma
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1422.008-sentry-ha-redesign.patch,
> SENTRY-1422.009-sentry-ha-redesign.patch,
> SENTRY-1422.010-sentry-ha-redesign.patch, sentry_deadlock.png
>
>
> As I was working on Sentry-1321. I see that test case
> TestDbPrivilegeCleanupOnDrop#testDropObjects fails while "Granting select on
> table" with following deadlock.
> {code}
> java.sql.SQLException: Error while processing statement: FAILED: Execution
> Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Unknown error for
> request: TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2,
> requestorUserName:admin1, roleName:select_tbl1,
> privileges:[TSentryPrivilege(privilegeScope:TABLE, serverName:server1,
> dbName:db_2, tableName:tb_1, URI:, action:select, createTime:1469562185573,
> grantOption:FALSE, columnName:)]), message: Iteration request failed : SELECT
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
> FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?. Server
> Stacktrace: javax.jdo.JDODataStoreException: Iteration request failed :
> SELECT 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
> FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 O
[jira] [Comment Edited] (SENTRY-1422) JDO deadlocks while processing grant while a background thread processes Notificationlogs
[
https://issues.apache.org/jira/browse/SENTRY-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15509132#comment-15509132
]
Jerry Chen edited comment on SENTRY-1422 at 9/21/16 8:03 AM:
-
[~colinma] I agree to Alex that the cycle lock was caused by two transactions
both need to lock two rows of data but they happen acquiring them in different
order.
The best solution to this problem is to avoid this order in transactions.
Please try to figure whether it is possible.
If it is not possible, I suggest to use the proper handling of the exception
and retry. For example wait for a random milliseconds of time and retry to
avoid the contention.
was (Author: jerrychenhf):
[~colinma] I agree to Alex that the cycle lock was caused by two transactions
both need to lock two rows of data by they happen acquiring them in different
order.
The best solution to this problem is to avoid this order in transactions.
Please try to figure whether it is possible.
If it is not possible, I suggest to use the proper handling of the exception
and retry. For example wait for a random milliseconds of time and retry to
avoid the contention.
> JDO deadlocks while processing grant while a background thread processes
> Notificationlogs
> -
>
> Key: SENTRY-1422
> URL: https://issues.apache.org/jira/browse/SENTRY-1422
> Project: Sentry
> Issue Type: Sub-task
> Components: Hdfs Plugin
>Reporter: Sravya Tirukkovalur
>Assignee: Colin Ma
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1422.008-sentry-ha-redesign.patch,
> SENTRY-1422.009-sentry-ha-redesign.patch,
> SENTRY-1422.010-sentry-ha-redesign.patch, sentry_deadlock.png
>
>
> As I was working on Sentry-1321. I see that test case
> TestDbPrivilegeCleanupOnDrop#testDropObjects fails while "Granting select on
> table" with following deadlock.
> {code}
> java.sql.SQLException: Error while processing statement: FAILED: Execution
> Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Unknown error for
> request: TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2,
> requestorUserName:admin1, roleName:select_tbl1,
> privileges:[TSentryPrivilege(privilegeScope:TABLE, serverName:server1,
> dbName:db_2, tableName:tb_1, URI:, action:select, createTime:1469562185573,
> grantOption:FALSE, columnName:)]), message: Iteration request failed : SELECT
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
> FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?. Server
> Stacktrace: javax.jdo.JDODataStoreException: Iteration request failed :
> SELECT 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
> FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
> at
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitTransaction(SentryStore.java:279)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitUpdateTransaction(SentryStore.java:261)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:458)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:261)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
> at
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
> at
> org.apache.thrift.server.TThreadPoolServer$Work
[jira] [Comment Edited] (SENTRY-1422) JDO deadlocks while processing grant while a background thread processes Notificationlogs
[
https://issues.apache.org/jira/browse/SENTRY-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15508726#comment-15508726
]
Alexander Kolbasov edited comment on SENTRY-1422 at 9/21/16 4:46 AM:
-
I like the idea of making SentryStore a singleton because conceptually it is.
But I don't like the use of synchronized() to fix DB synchronization issues. We
shouldn't protect DB with mutexes.
I run some experiments with Datanucleus and could not reproduce the deadlock by
simply modifying left or right side of M:N relationship, but I can easily
reproduce the deadlock whenever I access two objects (represented by two
different tables) in the reversed order. For example, for two objects X and Y,
if one thread calls
{code}
x.addY(y)
y.addX(x)
{code}
and another calls
{code}
y.addX(x);
x.addY(y);
{code}
the deadlock is very easy to reproduce.
So it seems that we do have order inversion between privileges and roles which
we need to find.
was (Author: akolb):
I like the idea of making SentryStore a singleton because conceptually it is.
But I don't like the use of synchronized() to fix DB synchronization issues. We
shouldn't protect DB with mutexes.
I run some experiments with Datanucleus and could not reproduce the deadlock by
simply modifying left or right side of M:N relationship, but I can easily
reproduce the deadlock whenever I access two objects (represented by two
different tables) in the reversed order. For example, for two objects X and Y,
if one thread calls
{code}
x.addY(y)
y.addX(x)
{code}
and another calls
{code}
y.addX(x)
} the deadlock is very easy to reproduce.
So it seems that we do have order inversion between privileges and roles which
we need to find.
> JDO deadlocks while processing grant while a background thread processes
> Notificationlogs
> -
>
> Key: SENTRY-1422
> URL: https://issues.apache.org/jira/browse/SENTRY-1422
> Project: Sentry
> Issue Type: Sub-task
> Components: Hdfs Plugin
>Reporter: Sravya Tirukkovalur
>Assignee: Colin Ma
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1422.008-sentry-ha-redesign.patch,
> SENTRY-1422.009-sentry-ha-redesign.patch,
> SENTRY-1422.010-sentry-ha-redesign.patch, sentry_deadlock.png
>
>
> As I was working on Sentry-1321. I see that test case
> TestDbPrivilegeCleanupOnDrop#testDropObjects fails while "Granting select on
> table" with following deadlock.
> {code}
> java.sql.SQLException: Error while processing statement: FAILED: Execution
> Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Unknown error for
> request: TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2,
> requestorUserName:admin1, roleName:select_tbl1,
> privileges:[TSentryPrivilege(privilegeScope:TABLE, serverName:server1,
> dbName:db_2, tableName:tb_1, URI:, action:select, createTime:1469562185573,
> grantOption:FALSE, columnName:)]), message: Iteration request failed : SELECT
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
> FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?. Server
> Stacktrace: javax.jdo.JDODataStoreException: Iteration request failed :
> SELECT 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
> FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
> at
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitTransaction(SentryStore.java:279)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitUpdateTransaction(SentryStore.java:261)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:458)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:261)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
> at
> org.apa
[jira] [Comment Edited] (SENTRY-1422) JDO deadlocks while processing grant while a background thread processes Notificationlogs
[
https://issues.apache.org/jira/browse/SENTRY-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15508726#comment-15508726
]
Alexander Kolbasov edited comment on SENTRY-1422 at 9/21/16 4:45 AM:
-
I like the idea of making SentryStore a singleton because conceptually it is.
But I don't like the use of synchronized() to fix DB synchronization issues. We
shouldn't protect DB with mutexes.
I run some experiments with Datanucleus and could not reproduce the deadlock by
simply modifying left or right side of M:N relationship, but I can easily
reproduce the deadlock whenever I access two objects (represented by two
different tables) in the reversed order. For example, for two objects X and Y,
if one thread calls
{code}
x.addY(y)
y.addX(x)
{code}
and another calls
{code}
y.addX(x)
} the deadlock is very easy to reproduce.
So it seems that we do have order inversion between privileges and roles which
we need to find.
was (Author: akolb):
I like the idea of making SentryStore a singleton because conceptually it is.
But I don't like the use of synchronized() to fix DB synchronization issues. We
shouldn't protect DB with mutexes.
I run some experiments with Datanucleus and could not reproduce the deadlock by
simply modifying left or right side of M:N relationship, but I can easily
reproduce the deadlock whenever I access two objects (represented by two
different tables) in the reversed order. For example, for two objects X and Y,
if one thread calls {X.addY(y)} and another calls {Y.addX(x)} the deadlock is
very easy to reproduce.
So it seems that we do have order inversion between privileges and roles which
we need to find.
> JDO deadlocks while processing grant while a background thread processes
> Notificationlogs
> -
>
> Key: SENTRY-1422
> URL: https://issues.apache.org/jira/browse/SENTRY-1422
> Project: Sentry
> Issue Type: Sub-task
> Components: Hdfs Plugin
>Reporter: Sravya Tirukkovalur
>Assignee: Colin Ma
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1422.008-sentry-ha-redesign.patch,
> SENTRY-1422.009-sentry-ha-redesign.patch,
> SENTRY-1422.010-sentry-ha-redesign.patch, sentry_deadlock.png
>
>
> As I was working on Sentry-1321. I see that test case
> TestDbPrivilegeCleanupOnDrop#testDropObjects fails while "Granting select on
> table" with following deadlock.
> {code}
> java.sql.SQLException: Error while processing statement: FAILED: Execution
> Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Unknown error for
> request: TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2,
> requestorUserName:admin1, roleName:select_tbl1,
> privileges:[TSentryPrivilege(privilegeScope:TABLE, serverName:server1,
> dbName:db_2, tableName:tb_1, URI:, action:select, createTime:1469562185573,
> grantOption:FALSE, columnName:)]), message: Iteration request failed : SELECT
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
> FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?. Server
> Stacktrace: javax.jdo.JDODataStoreException: Iteration request failed :
> SELECT 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
> FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
> at
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitTransaction(SentryStore.java:279)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitUpdateTransaction(SentryStore.java:261)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:458)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:261)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPoli
[jira] [Comment Edited] (SENTRY-1422) JDO deadlocks while processing grant while a background thread processes Notificationlogs
[
https://issues.apache.org/jira/browse/SENTRY-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15508726#comment-15508726
]
Alexander Kolbasov edited comment on SENTRY-1422 at 9/21/16 4:43 AM:
-
I like the idea of making SentryStore a singleton because conceptually it is.
But I don't like the use of synchronized() to fix DB synchronization issues. We
shouldn't protect DB with mutexes.
I run some experiments with Datanucleus and could not reproduce the deadlock by
simply modifying left or right side of M:N relationship, but I can easily
reproduce the deadlock whenever I access two objects (represented by two
different tables) in the reversed order. For example, for two objects X and Y,
if one thread calls {X.addY(y)} and another calls Y.addX(x) the deadlock is
very easy to reproduce.
So it seems that we do have order inversion between privileges and roles which
we need to find.
was (Author: akolb):
I like the idea of making SentryStore a singleton because conceptually it is.
But I don't like the use of synchronized() to fix DB synchronization issues. We
shouldn't protect DB with mutexes.
I run some experiments with Datanucleus and could not reproduce the deadlock by
simply modifying left or right side of M:N relationship, but I can easily
reproduce the deadlock whenever I access two objects (represented by two
different tables) in the reversed order. For example, for two objects X and Y,
if one thread calls X.addY(y) and another calls Y.addX(x) the deadlock is very
easy to reproduce.
So it seems that we do have order inversion between privileges and roles which
we need to find.
> JDO deadlocks while processing grant while a background thread processes
> Notificationlogs
> -
>
> Key: SENTRY-1422
> URL: https://issues.apache.org/jira/browse/SENTRY-1422
> Project: Sentry
> Issue Type: Sub-task
> Components: Hdfs Plugin
>Reporter: Sravya Tirukkovalur
>Assignee: Colin Ma
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1422.008-sentry-ha-redesign.patch,
> SENTRY-1422.009-sentry-ha-redesign.patch,
> SENTRY-1422.010-sentry-ha-redesign.patch, sentry_deadlock.png
>
>
> As I was working on Sentry-1321. I see that test case
> TestDbPrivilegeCleanupOnDrop#testDropObjects fails while "Granting select on
> table" with following deadlock.
> {code}
> java.sql.SQLException: Error while processing statement: FAILED: Execution
> Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Unknown error for
> request: TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2,
> requestorUserName:admin1, roleName:select_tbl1,
> privileges:[TSentryPrivilege(privilegeScope:TABLE, serverName:server1,
> dbName:db_2, tableName:tb_1, URI:, action:select, createTime:1469562185573,
> grantOption:FALSE, columnName:)]), message: Iteration request failed : SELECT
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
> FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?. Server
> Stacktrace: javax.jdo.JDODataStoreException: Iteration request failed :
> SELECT 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
> FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
> at
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitTransaction(SentryStore.java:279)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitUpdateTransaction(SentryStore.java:261)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:458)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:261)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_ro
[jira] [Comment Edited] (SENTRY-1422) JDO deadlocks while processing grant while a background thread processes Notificationlogs
[
https://issues.apache.org/jira/browse/SENTRY-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15508726#comment-15508726
]
Alexander Kolbasov edited comment on SENTRY-1422 at 9/21/16 4:44 AM:
-
I like the idea of making SentryStore a singleton because conceptually it is.
But I don't like the use of synchronized() to fix DB synchronization issues. We
shouldn't protect DB with mutexes.
I run some experiments with Datanucleus and could not reproduce the deadlock by
simply modifying left or right side of M:N relationship, but I can easily
reproduce the deadlock whenever I access two objects (represented by two
different tables) in the reversed order. For example, for two objects X and Y,
if one thread calls {X.addY(y)} and another calls {Y.addX(x)} the deadlock is
very easy to reproduce.
So it seems that we do have order inversion between privileges and roles which
we need to find.
was (Author: akolb):
I like the idea of making SentryStore a singleton because conceptually it is.
But I don't like the use of synchronized() to fix DB synchronization issues. We
shouldn't protect DB with mutexes.
I run some experiments with Datanucleus and could not reproduce the deadlock by
simply modifying left or right side of M:N relationship, but I can easily
reproduce the deadlock whenever I access two objects (represented by two
different tables) in the reversed order. For example, for two objects X and Y,
if one thread calls {X.addY(y)} and another calls Y.addX(x) the deadlock is
very easy to reproduce.
So it seems that we do have order inversion between privileges and roles which
we need to find.
> JDO deadlocks while processing grant while a background thread processes
> Notificationlogs
> -
>
> Key: SENTRY-1422
> URL: https://issues.apache.org/jira/browse/SENTRY-1422
> Project: Sentry
> Issue Type: Sub-task
> Components: Hdfs Plugin
>Reporter: Sravya Tirukkovalur
>Assignee: Colin Ma
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1422.008-sentry-ha-redesign.patch,
> SENTRY-1422.009-sentry-ha-redesign.patch,
> SENTRY-1422.010-sentry-ha-redesign.patch, sentry_deadlock.png
>
>
> As I was working on Sentry-1321. I see that test case
> TestDbPrivilegeCleanupOnDrop#testDropObjects fails while "Granting select on
> table" with following deadlock.
> {code}
> java.sql.SQLException: Error while processing statement: FAILED: Execution
> Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Unknown error for
> request: TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2,
> requestorUserName:admin1, roleName:select_tbl1,
> privileges:[TSentryPrivilege(privilegeScope:TABLE, serverName:server1,
> dbName:db_2, tableName:tb_1, URI:, action:select, createTime:1469562185573,
> grantOption:FALSE, columnName:)]), message: Iteration request failed : SELECT
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
> FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?. Server
> Stacktrace: javax.jdo.JDODataStoreException: Iteration request failed :
> SELECT 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
> FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
> at
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitTransaction(SentryStore.java:279)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitUpdateTransaction(SentryStore.java:261)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:458)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:261)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentr
[jira] [Comment Edited] (SENTRY-1422) JDO deadlocks while processing grant while a background thread processes Notificationlogs
[
https://issues.apache.org/jira/browse/SENTRY-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15485728#comment-15485728
]
Alexander Kolbasov edited comment on SENTRY-1422 at 9/13/16 12:13 AM:
--
I attached a small png file that shows deadlock when accessing MSentryRole and
MSentryPrivilege from two different threads.
Is it possible to break the deadlock by removing set of roles from
MSentryPrivilege?
was (Author: akolb):
I attached a small png file that shows deadlock when accessing MSentryRole and
MSentryPrivilege from two different threads.
> JDO deadlocks while processing grant while a background thread processes
> Notificationlogs
> -
>
> Key: SENTRY-1422
> URL: https://issues.apache.org/jira/browse/SENTRY-1422
> Project: Sentry
> Issue Type: Sub-task
> Components: Hdfs Plugin
>Reporter: Sravya Tirukkovalur
>Assignee: Colin Ma
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1422.008-sentry-ha-redesign.patch,
> SENTRY-1422.009-sentry-ha-redesign.patch,
> SENTRY-1422.010-sentry-ha-redesign.patch, sentry_deadlock.png
>
>
> As I was working on Sentry-1321. I see that test case
> TestDbPrivilegeCleanupOnDrop#testDropObjects fails while "Granting select on
> table" with following deadlock.
> {code}
> java.sql.SQLException: Error while processing statement: FAILED: Execution
> Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Unknown error for
> request: TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2,
> requestorUserName:admin1, roleName:select_tbl1,
> privileges:[TSentryPrivilege(privilegeScope:TABLE, serverName:server1,
> dbName:db_2, tableName:tb_1, URI:, action:select, createTime:1469562185573,
> grantOption:FALSE, columnName:)]), message: Iteration request failed : SELECT
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
> FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?. Server
> Stacktrace: javax.jdo.JDODataStoreException: Iteration request failed :
> SELECT 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
> FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
> at
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitTransaction(SentryStore.java:279)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitUpdateTransaction(SentryStore.java:261)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:458)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:261)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
> at
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> NestedThrowablesStackTrace:
> java.sql.SQLTransactionRollbackException: A lock could not be obtained due to
> a deadlock, cycle of locks and waiters is:
> Lock : ROW, SENTRY_ROLE_DB_PRIVILEGE_MAP, (1,10)
> Waiting XID : {712, S} , SENTRY, SELECT
> 'org.apache.sentry.provider.db.service.model.
[jira] [Comment Edited] (SENTRY-1422) JDO deadlocks while processing grant while a background thread processes Notificationlogs
[
https://issues.apache.org/jira/browse/SENTRY-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15450712#comment-15450712
]
Alexander Kolbasov edited comment on SENTRY-1422 at 8/31/16 1:33 AM:
-
It is interesting to look at the use of SENTRY_ROLE_DB_PRIVILEGE_MAP and
CENTRY_DB_PRIVILEGE.
I only see references to SENTRY_ROLE_DB_PRIVILEGE_MAP for joins as part of the
definition of MSentryRole - no one uses it directly except
removeOrphanedPrivileges():
{code}
private void removeOrphanedPrivileges() throws SentryStandbyException {
final String privDB = "SENTRY_DB_PRIVILEGE";
final String privId = "DB_PRIVILEGE_ID";
final String mapDB = "SENTRY_ROLE_DB_PRIVILEGE_MAP";
final String privFilter =
"select " + privId +
" from " + privDB + " p" +
" where not exists (" +
" select 1 from " + mapDB + " d" +
" where p." + privId + " != d." + privId +
" )";
{code}
This is used here:
{code}
transaction.begin();
transaction.setRollbackOnly(); // Makes the tx read-only
act.checkSqlFencing(pm);
Query query = pm.newQuery("javax.jdo.query.SQL", privFilter);
query.setClass(MSentryPrivilege.class);
List results = (List) query.execute();
{code}
The definition for MSentryPrivilege has “privileges” field which does join with
SENTRY_ROLE_DB_PRIVILEGE_MAP.
Can this cause our deadlock troubles?
If we look at potential holders of SENTRY_ROLE_DB_PRIVILEGE_MAP we can see that
outside of the use above it is only used indirectly in joins (see package.jdo
file).
I also see the following interesting thing:
MSentryRole does join with SENTRY_ROLE_DB_PRIVILEGE_MAP. It also references
collection of MSentryPrivilege objects which has “roles” fields which is a
collection of MSentryRole objects. This looks somewhat suspicious - like a
circular dependency - can this cause any deadlocks?
Here is the package.jdo file for references:
{code}
{code}
was (Author: akolb):
It is interesting to look at the use of SENTRY_ROLE_DB_PRIVILEGE_MAP and
CENTRY_DB_PRIVILEGE.
I only see references to SENTRY_ROLE_DB_PRIVILEGE_MAP for joins as part of the
definition of MSentryRole - no one uses it directly except
removeOrphanedPrivileges():
{code}
private void removeOrphanedPrivileges() throws SentryStandbyException {
final String privDB = "SENTRY_DB_PRIVILEGE";
final String privId = "DB_PRIVILEGE_ID";
final String mapDB = "SENTRY_ROLE_DB_PRIVILEGE_MAP";
final String privFilter =
"select " + privId +
" from " + privDB + " p" +
" where not exists (" +
" select 1 from " + mapDB + " d" +
" where p." + privId + " != d." + privId +
" )";
{code}
This is used here:
{code}
transaction.begin();
transaction.setRollbackOnly(); // Makes the tx read-only
act.checkSqlFencing(pm);
Query query = pm.newQuery("javax.jdo.query.SQL", privFilter);
query.setClass(MSentryPrivilege.class);
List results = (List) query.execute();
{code}
The definition for MSentryPrivilege has “privileges” field which does join with
SENTRY_ROLE_DB_PRIVILEGE_MAP.
Can this cause our deadlock troubles?
If we look at potential holders of SENTRY_ROLE_DB_PRIVILEGE_MAP we can see that
outside of
