[jira] [Updated] (SENTRY-1422) JDO deadlocks while processing grant while a background thread processes Notificationlogs
[
https://issues.apache.org/jira/browse/SENTRY-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Ma updated SENTRY-1422:
-
Attachment: SENTRY-1422.006-sentry-ha-redesign.patch
> JDO deadlocks while processing grant while a background thread processes
> Notificationlogs
> -
>
> Key: SENTRY-1422
> URL: https://issues.apache.org/jira/browse/SENTRY-1422
> Project: Sentry
> Issue Type: Sub-task
> Components: Hdfs Plugin
>Reporter: Sravya Tirukkovalur
>Assignee: Colin Ma
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1422.001-sentry-ha-redesign.patch,
> SENTRY-1422.002-sentry-ha-redesign.patch,
> SENTRY-1422.003-sentry-ha-redesign.patch,
> SENTRY-1422.004-sentry-ha-redesign.patch,
> SENTRY-1422.005-sentry-ha-redesign.patch,
> SENTRY-1422.006-sentry-ha-redesign.patch
>
>
> As I was working on Sentry-1321. I see that test case
> TestDbPrivilegeCleanupOnDrop#testDropObjects fails while "Granting select on
> table" with following deadlock.
> {code}
> java.sql.SQLException: Error while processing statement: FAILED: Execution
> Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Unknown error for
> request: TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2,
> requestorUserName:admin1, roleName:select_tbl1,
> privileges:[TSentryPrivilege(privilegeScope:TABLE, serverName:server1,
> dbName:db_2, tableName:tb_1, URI:, action:select, createTime:1469562185573,
> grantOption:FALSE, columnName:)]), message: Iteration request failed : SELECT
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
> FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?. Server
> Stacktrace: javax.jdo.JDODataStoreException: Iteration request failed :
> SELECT 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
> FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
> at
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitTransaction(SentryStore.java:279)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitUpdateTransaction(SentryStore.java:261)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:458)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:261)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
> at
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> NestedThrowablesStackTrace:
> java.sql.SQLTransactionRollbackException: A lock could not be obtained due to
> a deadlock, cycle of locks and waiters is:
> Lock : ROW, SENTRY_ROLE_DB_PRIVILEGE_MAP, (1,10)
> Waiting XID : {712, S} , SENTRY, SELECT
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
> FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE
[jira] [Commented] (SENTRY-996) Faulty loop logic in HdfsHAClientInvocationHandler + HAClientInvocationHandler
[
https://issues.apache.org/jira/browse/SENTRY-996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15434108#comment-15434108
]
Hadoop QA commented on SENTRY-996:
--
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12824053/SENTRY-996-002.patch
against master.
{color:red}Overall:{color} -1 due to 3 errors
{color:red}ERROR:{color} mvn test exited 1
{color:red}ERROR:{color} Failed:
org.apache.sentry.tests.e2e.dbprovider.TestDbPrivilegesAtFunctionScope
{color:red}ERROR:{color} Failed:
org.apache.sentry.provider.db.generic.tools.TestSentryShellSolr
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/1977/console
This message is automatically generated.
> Faulty loop logic in HdfsHAClientInvocationHandler +
> HAClientInvocationHandler
> ---
>
> Key: SENTRY-996
> URL: https://issues.apache.org/jira/browse/SENTRY-996
> Project: Sentry
> Issue Type: Bug
>Reporter: Colm O hEigeartaigh
>Assignee: Ke Jia
> Attachments: SENTRY-996-002.patch, SENTRY-996.001.patch
>
>
> As discussed on the review board here: https://reviews.apache.org/r/41349/
> Both the HdfsHAClientInvocationHandler + HAClientInvocationHandler contain a
> while(true) loop that looks like it was designed to catch certain exceptions
> + retry. However, the logic is faulty in that it will always return after the
> first iteration, not matter what (as picked up by PMD).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SENTRY-1450) Have privilege converter set by Kafka binding
[ https://issues.apache.org/jira/browse/SENTRY-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15434050#comment-15434050 ] Ashish K Singh commented on SENTRY-1450: Thanks guys. > Have privilege converter set by Kafka binding > -- > > Key: SENTRY-1450 > URL: https://issues.apache.org/jira/browse/SENTRY-1450 > Project: Sentry > Issue Type: Improvement >Reporter: Ashish K Singh >Assignee: Ashish K Singh > Attachments: SENTRY-1450.001.patch > > > Have privilege converter set by Kafka binding. This will avoid users from > having to specify one extra config, which as of now can only take one value > for Kafka plugin, when enabling caching. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SENTRY-1450) Have privilege converter set by Kafka binding
[ https://issues.apache.org/jira/browse/SENTRY-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish K Singh updated SENTRY-1450: --- Attachment: (was: SENTRY-1450.001.diff) > Have privilege converter set by Kafka binding > -- > > Key: SENTRY-1450 > URL: https://issues.apache.org/jira/browse/SENTRY-1450 > Project: Sentry > Issue Type: Improvement >Reporter: Ashish K Singh >Assignee: Ashish K Singh > Attachments: SENTRY-1450.001.patch > > > Have privilege converter set by Kafka binding. This will avoid users from > having to specify one extra config, which as of now can only take one value > for Kafka plugin, when enabling caching. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (SENTRY-1450) Have privilege converter set by Kafka binding
[
https://issues.apache.org/jira/browse/SENTRY-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15434039#comment-15434039
]
Dapeng Sun edited comment on SENTRY-1450 at 8/24/16 1:44 AM:
-
Hi [~singhashish], the patch LGTM, I uploaded a patch with name
{{SENTRY-1450.001.patch}} to trigger Jenkins
was (Author: dapengsun):
Hi [~singhashish], the patch LGTM, could you uploaded a patch with name
{{SENTRY-1450.001.patch}} to trigger Jenkins?
> Have privilege converter set by Kafka binding
> --
>
> Key: SENTRY-1450
> URL: https://issues.apache.org/jira/browse/SENTRY-1450
> Project: Sentry
> Issue Type: Improvement
>Reporter: Ashish K Singh
>Assignee: Ashish K Singh
> Attachments: SENTRY-1450.001.diff, SENTRY-1450.001.patch
>
>
> Have privilege converter set by Kafka binding. This will avoid users from
> having to specify one extra config, which as of now can only take one value
> for Kafka plugin, when enabling caching.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SENTRY-1450) Have privilege converter set by Kafka binding
[
https://issues.apache.org/jira/browse/SENTRY-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15434039#comment-15434039
]
Dapeng Sun commented on SENTRY-1450:
Hi [~singhashish], the patch LGTM, could you uploaded a patch with name
{{SENTRY-1450.001.patch}} to trigger Jenkins?
> Have privilege converter set by Kafka binding
> --
>
> Key: SENTRY-1450
> URL: https://issues.apache.org/jira/browse/SENTRY-1450
> Project: Sentry
> Issue Type: Improvement
>Reporter: Ashish K Singh
>Assignee: Ashish K Singh
> Attachments: SENTRY-1450.001.diff
>
>
> Have privilege converter set by Kafka binding. This will avoid users from
> having to specify one extra config, which as of now can only take one value
> for Kafka plugin, when enabling caching.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SENTRY-1450) Have privilege converter set by Kafka binding
[ https://issues.apache.org/jira/browse/SENTRY-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15434036#comment-15434036 ] Hao Hao commented on SENTRY-1450: - +1 if all tests pass. Thanks [~singhashish]! > Have privilege converter set by Kafka binding > -- > > Key: SENTRY-1450 > URL: https://issues.apache.org/jira/browse/SENTRY-1450 > Project: Sentry > Issue Type: Improvement >Reporter: Ashish K Singh >Assignee: Ashish K Singh > Attachments: SENTRY-1450.001.diff > > > Have privilege converter set by Kafka binding. This will avoid users from > having to specify one extra config, which as of now can only take one value > for Kafka plugin, when enabling caching. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SENTRY-1422) JDO deadlocks while processing grant while a background thread processes Notificationlogs
[
https://issues.apache.org/jira/browse/SENTRY-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Ma updated SENTRY-1422:
-
Attachment: SENTRY-1422.005-sentry-ha-redesign.patch
> JDO deadlocks while processing grant while a background thread processes
> Notificationlogs
> -
>
> Key: SENTRY-1422
> URL: https://issues.apache.org/jira/browse/SENTRY-1422
> Project: Sentry
> Issue Type: Sub-task
> Components: Hdfs Plugin
>Reporter: Sravya Tirukkovalur
>Assignee: Colin Ma
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1422.001-sentry-ha-redesign.patch,
> SENTRY-1422.002-sentry-ha-redesign.patch,
> SENTRY-1422.003-sentry-ha-redesign.patch,
> SENTRY-1422.004-sentry-ha-redesign.patch,
> SENTRY-1422.005-sentry-ha-redesign.patch
>
>
> As I was working on Sentry-1321. I see that test case
> TestDbPrivilegeCleanupOnDrop#testDropObjects fails while "Granting select on
> table" with following deadlock.
> {code}
> java.sql.SQLException: Error while processing statement: FAILED: Execution
> Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Unknown error for
> request: TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2,
> requestorUserName:admin1, roleName:select_tbl1,
> privileges:[TSentryPrivilege(privilegeScope:TABLE, serverName:server1,
> dbName:db_2, tableName:tb_1, URI:, action:select, createTime:1469562185573,
> grantOption:FALSE, columnName:)]), message: Iteration request failed : SELECT
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
> FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?. Server
> Stacktrace: javax.jdo.JDODataStoreException: Iteration request failed :
> SELECT 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
> FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
> at
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitTransaction(SentryStore.java:279)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitUpdateTransaction(SentryStore.java:261)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:458)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:261)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
> at
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> NestedThrowablesStackTrace:
> java.sql.SQLTransactionRollbackException: A lock could not be obtained due to
> a deadlock, cycle of locks and waiters is:
> Lock : ROW, SENTRY_ROLE_DB_PRIVILEGE_MAP, (1,10)
> Waiting XID : {712, S} , SENTRY, SELECT
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
> FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
> Granted XID : {704, X}
>
[jira] [Updated] (SENTRY-1450) Have privilege converter set by Kafka binding
[ https://issues.apache.org/jira/browse/SENTRY-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish K Singh updated SENTRY-1450: --- Status: Patch Available (was: Open) > Have privilege converter set by Kafka binding > -- > > Key: SENTRY-1450 > URL: https://issues.apache.org/jira/browse/SENTRY-1450 > Project: Sentry > Issue Type: Improvement >Reporter: Ashish K Singh >Assignee: Ashish K Singh > Attachments: SENTRY-1450.001.diff > > > Have privilege converter set by Kafka binding. This will avoid users from > having to specify one extra config, which as of now can only take one value > for Kafka plugin, when enabling caching. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SENTRY-1450) Have privilege converter set by Kafka binding
[ https://issues.apache.org/jira/browse/SENTRY-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish K Singh updated SENTRY-1450: --- Attachment: SENTRY-1450.001.diff > Have privilege converter set by Kafka binding > -- > > Key: SENTRY-1450 > URL: https://issues.apache.org/jira/browse/SENTRY-1450 > Project: Sentry > Issue Type: Improvement >Reporter: Ashish K Singh >Assignee: Ashish K Singh > Attachments: SENTRY-1450.001.diff > > > Have privilege converter set by Kafka binding. This will avoid users from > having to specify one extra config, which as of now can only take one value > for Kafka plugin, when enabling caching. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (SENTRY-1450) Have privilege converter set by Kafka binding
Ashish K Singh created SENTRY-1450: -- Summary: Have privilege converter set by Kafka binding Key: SENTRY-1450 URL: https://issues.apache.org/jira/browse/SENTRY-1450 Project: Sentry Issue Type: Improvement Reporter: Ashish K Singh Assignee: Ashish K Singh Have privilege converter set by Kafka binding. This will avoid users from having to specify one extra config, which as of now can only take one value for Kafka plugin, when enabling caching. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SENTRY-1422) JDO deadlocks while processing grant while a background thread processes Notificationlogs
[
https://issues.apache.org/jira/browse/SENTRY-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15434000#comment-15434000
]
Colin Ma commented on SENTRY-1422:
--
Hi, [~sravya], for the deadlock problem, it caused by multiple SentryStore
instances (from SentryPolicyStoreProcessor and HMSFollower) do the write
methods in the same time. And you're right, the test changes are for fencing.
[~hahao], I'm still working on the test fix, and will add the link for RB later.
> JDO deadlocks while processing grant while a background thread processes
> Notificationlogs
> -
>
> Key: SENTRY-1422
> URL: https://issues.apache.org/jira/browse/SENTRY-1422
> Project: Sentry
> Issue Type: Sub-task
> Components: Hdfs Plugin
>Reporter: Sravya Tirukkovalur
>Assignee: Colin Ma
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1422.001-sentry-ha-redesign.patch,
> SENTRY-1422.002-sentry-ha-redesign.patch,
> SENTRY-1422.003-sentry-ha-redesign.patch,
> SENTRY-1422.004-sentry-ha-redesign.patch
>
>
> As I was working on Sentry-1321. I see that test case
> TestDbPrivilegeCleanupOnDrop#testDropObjects fails while "Granting select on
> table" with following deadlock.
> {code}
> java.sql.SQLException: Error while processing statement: FAILED: Execution
> Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Unknown error for
> request: TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2,
> requestorUserName:admin1, roleName:select_tbl1,
> privileges:[TSentryPrivilege(privilegeScope:TABLE, serverName:server1,
> dbName:db_2, tableName:tb_1, URI:, action:select, createTime:1469562185573,
> grantOption:FALSE, columnName:)]), message: Iteration request failed : SELECT
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
> FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?. Server
> Stacktrace: javax.jdo.JDODataStoreException: Iteration request failed :
> SELECT 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
> FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
> at
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitTransaction(SentryStore.java:279)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitUpdateTransaction(SentryStore.java:261)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:458)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:261)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
> at
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> NestedThrowablesStackTrace:
> java.sql.SQLTransactionRollbackException: A lock could not be obtained due to
> a deadlock, cycle of locks and waiters is:
> Lock : ROW, SENTRY_ROLE_DB_PRIVILEGE_MAP, (1,10)
> Waiting XID : {712, S} , SENTRY, SELECT
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS
>
[jira] [Updated] (SENTRY-1371) Rework Sentry start up and Hive state fetch
[ https://issues.apache.org/jira/browse/SENTRY-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hao Hao updated SENTRY-1371: Resolution: Fixed Status: Resolved (was: Patch Available) > Rework Sentry start up and Hive state fetch > --- > > Key: SENTRY-1371 > URL: https://issues.apache.org/jira/browse/SENTRY-1371 > Project: Sentry > Issue Type: Sub-task > Components: Service >Reporter: Colin P. McCabe >Assignee: Hao Hao > Fix For: sentry-ha-redesign > > Attachments: SENTRY-1371.001-sentry-ha-redesign.patch, > SENTRY-1371.002-sentry-ha-redesign.patch, > SENTRY-1371.003-sentry-ha-redesign.patch > > > Rework Sentry start up and Hive state fetch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SENTRY-1371) Rework Sentry start up and Hive state fetch
[ https://issues.apache.org/jira/browse/SENTRY-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433991#comment-15433991 ] Hao Hao commented on SENTRY-1371: - Committed it to feature branch as the failure test case is related to Sentry-1422. > Rework Sentry start up and Hive state fetch > --- > > Key: SENTRY-1371 > URL: https://issues.apache.org/jira/browse/SENTRY-1371 > Project: Sentry > Issue Type: Sub-task > Components: Service >Reporter: Colin P. McCabe >Assignee: Hao Hao > Fix For: sentry-ha-redesign > > Attachments: SENTRY-1371.001-sentry-ha-redesign.patch, > SENTRY-1371.002-sentry-ha-redesign.patch, > SENTRY-1371.003-sentry-ha-redesign.patch > > > Rework Sentry start up and Hive state fetch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SENTRY-1447) When s3 is configured as HDFS defaultFS and Hive Warehouse Dir, need to fix some e2e test failures. For example, TestDbHdfsMaxGroups.java.
[
https://issues.apache.org/jira/browse/SENTRY-1447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433986#comment-15433986
]
Hadoop QA commented on SENTRY-1447:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12825156/SENTRY-1447.1.patch
against master.
{color:green}Overall:{color} +1 all checks pass
{color:green}SUCCESS:{color} all tests passed
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/1975/console
This message is automatically generated.
> When s3 is configured as HDFS defaultFS and Hive Warehouse Dir, need to fix
> some e2e test failures. For example, TestDbHdfsMaxGroups.java.
> --
>
> Key: SENTRY-1447
> URL: https://issues.apache.org/jira/browse/SENTRY-1447
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 1.8.0
>Reporter: Anne Yu
>Assignee: Anne Yu
> Fix For: 1.8.0
>
> Attachments: SENTRY-1447.0.patch, SENTRY-1447.1.patch
>
>
> org.apache.sentry.tests.e2e.dbprovider.TestDbExportImportPrivileges.testExportImportPrivileges
> org.apache.sentry.tests.e2e.dbprovider.TestDbOperationsPart2.testExportImport
> org.apache.sentry.tests.e2e.hdfs.TestDbHdfsExtMaxGroups.testIntDbTblMaxAclsWithGroups
> org.apache.sentry.tests.e2e.hdfs.TestDbHdfsExtMaxGroups.testIntColMaxAclsWithGroups
> org.apache.sentry.tests.e2e.hdfs.TestDbHdfsExtMaxGroups.testIntDbTblMaxAclsWithGroupsNoPar
> org.apache.sentry.tests.e2e.hdfs.TestDbHdfsMaxGroups.testIntDbTblMaxAclsWithGroups
> org.apache.sentry.tests.e2e.hdfs.TestDbHdfsMaxGroups.testIntDbTblMaxAclsWithGroupsNoPar
> org.apache.sentry.tests.e2e.hdfs.TestDbHdfsMaxGroups.testIntColMaxAclsWithGroups
> org.apache.sentry.tests.e2e.hive.fs.TestHiveWarehouseOnExtFs.TestCreateTable
> org.apache.sentry.tests.e2e.hive.fs.TestTableOnExtFS.TestCreateExtTableWithoutScheme
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SENTRY-1371) Rework Sentry start up and Hive state fetch
[
https://issues.apache.org/jira/browse/SENTRY-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433887#comment-15433887
]
Hadoop QA commented on SENTRY-1371:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12825107/SENTRY-1371.003-sentry-ha-redesign.patch
against sentry-ha-redesign.
{color:red}Overall:{color} -1 due to 2 errors
{color:red}ERROR:{color} mvn test exited 1
{color:red}ERROR:{color} Failed:
org.apache.sentry.tests.e2e.dbprovider.TestConcurrentClients
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/1974/console
This message is automatically generated.
> Rework Sentry start up and Hive state fetch
> ---
>
> Key: SENTRY-1371
> URL: https://issues.apache.org/jira/browse/SENTRY-1371
> Project: Sentry
> Issue Type: Sub-task
> Components: Service
>Reporter: Colin P. McCabe
>Assignee: Hao Hao
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1371.001-sentry-ha-redesign.patch,
> SENTRY-1371.002-sentry-ha-redesign.patch,
> SENTRY-1371.003-sentry-ha-redesign.patch
>
>
> Rework Sentry start up and Hive state fetch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SENTRY-1447) When s3 is configured as HDFS defaultFS and Hive Warehouse Dir, need to fix some e2e test failures. For example, TestDbHdfsMaxGroups.java.
[ https://issues.apache.org/jira/browse/SENTRY-1447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433866#comment-15433866 ] Anne Yu commented on SENTRY-1447: - TestColumnEndToEnd.java times out frequently after fix the test. > When s3 is configured as HDFS defaultFS and Hive Warehouse Dir, need to fix > some e2e test failures. For example, TestDbHdfsMaxGroups.java. > -- > > Key: SENTRY-1447 > URL: https://issues.apache.org/jira/browse/SENTRY-1447 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 1.8.0 >Reporter: Anne Yu >Assignee: Anne Yu > Fix For: 1.8.0 > > Attachments: SENTRY-1447.0.patch, SENTRY-1447.1.patch > > > org.apache.sentry.tests.e2e.dbprovider.TestDbExportImportPrivileges.testExportImportPrivileges > org.apache.sentry.tests.e2e.dbprovider.TestDbOperationsPart2.testExportImport > org.apache.sentry.tests.e2e.hdfs.TestDbHdfsExtMaxGroups.testIntDbTblMaxAclsWithGroups > org.apache.sentry.tests.e2e.hdfs.TestDbHdfsExtMaxGroups.testIntColMaxAclsWithGroups > org.apache.sentry.tests.e2e.hdfs.TestDbHdfsExtMaxGroups.testIntDbTblMaxAclsWithGroupsNoPar > org.apache.sentry.tests.e2e.hdfs.TestDbHdfsMaxGroups.testIntDbTblMaxAclsWithGroups > org.apache.sentry.tests.e2e.hdfs.TestDbHdfsMaxGroups.testIntDbTblMaxAclsWithGroupsNoPar > org.apache.sentry.tests.e2e.hdfs.TestDbHdfsMaxGroups.testIntColMaxAclsWithGroups > org.apache.sentry.tests.e2e.hive.fs.TestHiveWarehouseOnExtFs.TestCreateTable > org.apache.sentry.tests.e2e.hive.fs.TestTableOnExtFS.TestCreateExtTableWithoutScheme -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SENTRY-1422) JDO deadlocks while processing grant while a background thread processes Notificationlogs
[
https://issues.apache.org/jira/browse/SENTRY-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433786#comment-15433786
]
Sravya Tirukkovalur commented on SENTRY-1422:
-
Thanks for the patch Colin! Making Sentrystore a singleton and synchronizing
all write methods sounds good to me. Is that all we are doing here? Are the
test changes with respect to fencing required?
> JDO deadlocks while processing grant while a background thread processes
> Notificationlogs
> -
>
> Key: SENTRY-1422
> URL: https://issues.apache.org/jira/browse/SENTRY-1422
> Project: Sentry
> Issue Type: Sub-task
> Components: Hdfs Plugin
>Reporter: Sravya Tirukkovalur
>Assignee: Colin Ma
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1422.001-sentry-ha-redesign.patch,
> SENTRY-1422.002-sentry-ha-redesign.patch,
> SENTRY-1422.003-sentry-ha-redesign.patch,
> SENTRY-1422.004-sentry-ha-redesign.patch
>
>
> As I was working on Sentry-1321. I see that test case
> TestDbPrivilegeCleanupOnDrop#testDropObjects fails while "Granting select on
> table" with following deadlock.
> {code}
> java.sql.SQLException: Error while processing statement: FAILED: Execution
> Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Unknown error for
> request: TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2,
> requestorUserName:admin1, roleName:select_tbl1,
> privileges:[TSentryPrivilege(privilegeScope:TABLE, serverName:server1,
> dbName:db_2, tableName:tb_1, URI:, action:select, createTime:1469562185573,
> grantOption:FALSE, columnName:)]), message: Iteration request failed : SELECT
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
> FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?. Server
> Stacktrace: javax.jdo.JDODataStoreException: Iteration request failed :
> SELECT 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
> FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
> at
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitTransaction(SentryStore.java:279)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitUpdateTransaction(SentryStore.java:261)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:458)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:261)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
> at
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> NestedThrowablesStackTrace:
> java.sql.SQLTransactionRollbackException: A lock could not be obtained due to
> a deadlock, cycle of locks and waiters is:
> Lock : ROW, SENTRY_ROLE_DB_PRIVILEGE_MAP, (1,10)
> Waiting XID : {712, S} , SENTRY, SELECT
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
> FROM
[jira] [Resolved] (SENTRY-1250) Document kafka integration with sentry
[ https://issues.apache.org/jira/browse/SENTRY-1250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anne Yu resolved SENTRY-1250. - Resolution: Fixed Doc is posted in the upstream wiki. Thanks Ashish! > Document kafka integration with sentry > -- > > Key: SENTRY-1250 > URL: https://issues.apache.org/jira/browse/SENTRY-1250 > Project: Sentry > Issue Type: Bug > Components: kafka-integration, Sentry >Affects Versions: 1.7.0 >Reporter: Anne Yu >Assignee: Ashish K Singh > Labels: Docs > > [~singhashish] could it be possible to finish this > [doc|https://cwiki.apache.org/confluence/display/SENTRY/%5BWIP%5DKafka+Authorization+via+Sentry]? > Thanks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SENTRY-1250) Document kafka integration with sentry
[ https://issues.apache.org/jira/browse/SENTRY-1250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anne Yu updated SENTRY-1250: Fix Version/s: 1.8.0 > Document kafka integration with sentry > -- > > Key: SENTRY-1250 > URL: https://issues.apache.org/jira/browse/SENTRY-1250 > Project: Sentry > Issue Type: Bug > Components: kafka-integration, Sentry >Affects Versions: 1.7.0 >Reporter: Anne Yu >Assignee: Ashish K Singh > Labels: Docs > Fix For: 1.8.0 > > > [~singhashish] could it be possible to finish this > [doc|https://cwiki.apache.org/confluence/display/SENTRY/%5BWIP%5DKafka+Authorization+via+Sentry]? > Thanks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SENTRY-1250) Document kafka integration with sentry
[ https://issues.apache.org/jira/browse/SENTRY-1250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433782#comment-15433782 ] Ashish K Singh commented on SENTRY-1250: Done, https://cwiki.apache.org/confluence/display/SENTRY/Apache+Kafka+Authorization+with+Apache+Sentry. > Document kafka integration with sentry > -- > > Key: SENTRY-1250 > URL: https://issues.apache.org/jira/browse/SENTRY-1250 > Project: Sentry > Issue Type: Bug > Components: kafka-integration, Sentry >Affects Versions: 1.7.0 >Reporter: Anne Yu >Assignee: Ashish K Singh > Labels: Docs > Fix For: 1.8.0 > > > [~singhashish] could it be possible to finish this > [doc|https://cwiki.apache.org/confluence/display/SENTRY/%5BWIP%5DKafka+Authorization+via+Sentry]? > Thanks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (SENTRY-1250) Document kafka integration with sentry
[ https://issues.apache.org/jira/browse/SENTRY-1250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anne Yu reassigned SENTRY-1250: --- Assignee: Ashish K Singh > Document kafka integration with sentry > -- > > Key: SENTRY-1250 > URL: https://issues.apache.org/jira/browse/SENTRY-1250 > Project: Sentry > Issue Type: Bug > Components: kafka-integration, Sentry >Affects Versions: 1.7.0 >Reporter: Anne Yu >Assignee: Ashish K Singh > Labels: Docs > > [~singhashish] could it be possible to finish this > [doc|https://cwiki.apache.org/confluence/display/SENTRY/%5BWIP%5DKafka+Authorization+via+Sentry]? > Thanks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SENTRY-1422) JDO deadlocks while processing grant while a background thread processes Notificationlogs
[
https://issues.apache.org/jira/browse/SENTRY-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433740#comment-15433740
]
Hao Hao commented on SENTRY-1422:
-
[~colin_mjj] Thanks a lot for the patch! Would you mind sharing a review board
link for this?
> JDO deadlocks while processing grant while a background thread processes
> Notificationlogs
> -
>
> Key: SENTRY-1422
> URL: https://issues.apache.org/jira/browse/SENTRY-1422
> Project: Sentry
> Issue Type: Sub-task
> Components: Hdfs Plugin
>Reporter: Sravya Tirukkovalur
>Assignee: Colin Ma
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1422.001-sentry-ha-redesign.patch,
> SENTRY-1422.002-sentry-ha-redesign.patch,
> SENTRY-1422.003-sentry-ha-redesign.patch,
> SENTRY-1422.004-sentry-ha-redesign.patch
>
>
> As I was working on Sentry-1321. I see that test case
> TestDbPrivilegeCleanupOnDrop#testDropObjects fails while "Granting select on
> table" with following deadlock.
> {code}
> java.sql.SQLException: Error while processing statement: FAILED: Execution
> Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Unknown error for
> request: TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2,
> requestorUserName:admin1, roleName:select_tbl1,
> privileges:[TSentryPrivilege(privilegeScope:TABLE, serverName:server1,
> dbName:db_2, tableName:tb_1, URI:, action:select, createTime:1469562185573,
> grantOption:FALSE, columnName:)]), message: Iteration request failed : SELECT
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
> FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?. Server
> Stacktrace: javax.jdo.JDODataStoreException: Iteration request failed :
> SELECT 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
> FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
> at
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitTransaction(SentryStore.java:279)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitUpdateTransaction(SentryStore.java:261)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:458)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:261)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
> at
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> NestedThrowablesStackTrace:
> java.sql.SQLTransactionRollbackException: A lock could not be obtained due to
> a deadlock, cycle of locks and waiters is:
> Lock : ROW, SENTRY_ROLE_DB_PRIVILEGE_MAP, (1,10)
> Waiting XID : {712, S} , SENTRY, SELECT
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
> FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
[jira] [Commented] (SENTRY-1447) When s3 is configured as HDFS defaultFS and Hive Warehouse Dir, need to fix some e2e test failures. For example, TestDbHdfsMaxGroups.java.
[
https://issues.apache.org/jira/browse/SENTRY-1447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433569#comment-15433569
]
Hadoop QA commented on SENTRY-1447:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12825112/SENTRY-1447.0.patch
against master.
{color:green}Overall:{color} +1 all checks pass
{color:green}SUCCESS:{color} all tests passed
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/1972/console
This message is automatically generated.
> When s3 is configured as HDFS defaultFS and Hive Warehouse Dir, need to fix
> some e2e test failures. For example, TestDbHdfsMaxGroups.java.
> --
>
> Key: SENTRY-1447
> URL: https://issues.apache.org/jira/browse/SENTRY-1447
> Project: Sentry
> Issue Type: Bug
> Components: Sentry
>Affects Versions: 1.8.0
>Reporter: Anne Yu
>Assignee: Anne Yu
> Fix For: 1.8.0
>
> Attachments: SENTRY-1447.0.patch
>
>
> org.apache.sentry.tests.e2e.dbprovider.TestDbExportImportPrivileges.testExportImportPrivileges
> org.apache.sentry.tests.e2e.dbprovider.TestDbOperationsPart2.testExportImport
> org.apache.sentry.tests.e2e.hdfs.TestDbHdfsExtMaxGroups.testIntDbTblMaxAclsWithGroups
> org.apache.sentry.tests.e2e.hdfs.TestDbHdfsExtMaxGroups.testIntColMaxAclsWithGroups
> org.apache.sentry.tests.e2e.hdfs.TestDbHdfsExtMaxGroups.testIntDbTblMaxAclsWithGroupsNoPar
> org.apache.sentry.tests.e2e.hdfs.TestDbHdfsMaxGroups.testIntDbTblMaxAclsWithGroups
> org.apache.sentry.tests.e2e.hdfs.TestDbHdfsMaxGroups.testIntDbTblMaxAclsWithGroupsNoPar
> org.apache.sentry.tests.e2e.hdfs.TestDbHdfsMaxGroups.testIntColMaxAclsWithGroups
> org.apache.sentry.tests.e2e.hive.fs.TestHiveWarehouseOnExtFs.TestCreateTable
> org.apache.sentry.tests.e2e.hive.fs.TestTableOnExtFS.TestCreateExtTableWithoutScheme
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (SENTRY-1371) Rework Sentry start up and Hive state fetch
[
https://issues.apache.org/jira/browse/SENTRY-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15433399#comment-15433399
]
Hadoop QA commented on SENTRY-1371:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12825107/SENTRY-1371.003-sentry-ha-redesign.patch
against sentry-ha-redesign.
{color:red}Overall:{color} -1 due to 4 errors
{color:red}ERROR:{color} mvn test exited 1
{color:red}ERROR:{color} Failed:
org.apache.sentry.provider.db.generic.service.thrift.TestAuditLogForSentryGenericService
{color:red}ERROR:{color} Failed:
org.apache.sentry.provider.db.generic.tools.TestSentryConfigToolSolr
{color:red}ERROR:{color} Failed:
org.apache.sentry.tests.e2e.dbprovider.TestDbPrivilegesAtFunctionScope
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/1971/console
This message is automatically generated.
> Rework Sentry start up and Hive state fetch
> ---
>
> Key: SENTRY-1371
> URL: https://issues.apache.org/jira/browse/SENTRY-1371
> Project: Sentry
> Issue Type: Sub-task
> Components: Service
>Reporter: Colin P. McCabe
>Assignee: Hao Hao
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1371.001-sentry-ha-redesign.patch,
> SENTRY-1371.002-sentry-ha-redesign.patch,
> SENTRY-1371.003-sentry-ha-redesign.patch
>
>
> Rework Sentry start up and Hive state fetch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (SENTRY-1447) When s3 is configured as HDFS defaultFS and Hive Warehouse Dir, need to fix some e2e test failures. For example, TestDbHdfsMaxGroups.java.
[ https://issues.apache.org/jira/browse/SENTRY-1447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anne Yu updated SENTRY-1447: Attachment: SENTRY-1447.0.patch > When s3 is configured as HDFS defaultFS and Hive Warehouse Dir, need to fix > some e2e test failures. For example, TestDbHdfsMaxGroups.java. > -- > > Key: SENTRY-1447 > URL: https://issues.apache.org/jira/browse/SENTRY-1447 > Project: Sentry > Issue Type: Bug > Components: Sentry >Affects Versions: 1.8.0 >Reporter: Anne Yu >Assignee: Anne Yu > Fix For: 1.8.0 > > Attachments: SENTRY-1447.0.patch > > > org.apache.sentry.tests.e2e.dbprovider.TestDbExportImportPrivileges.testExportImportPrivileges > org.apache.sentry.tests.e2e.dbprovider.TestDbOperationsPart2.testExportImport > org.apache.sentry.tests.e2e.hdfs.TestDbHdfsExtMaxGroups.testIntDbTblMaxAclsWithGroups > org.apache.sentry.tests.e2e.hdfs.TestDbHdfsExtMaxGroups.testIntColMaxAclsWithGroups > org.apache.sentry.tests.e2e.hdfs.TestDbHdfsExtMaxGroups.testIntDbTblMaxAclsWithGroupsNoPar > org.apache.sentry.tests.e2e.hdfs.TestDbHdfsMaxGroups.testIntDbTblMaxAclsWithGroups > org.apache.sentry.tests.e2e.hdfs.TestDbHdfsMaxGroups.testIntDbTblMaxAclsWithGroupsNoPar > org.apache.sentry.tests.e2e.hdfs.TestDbHdfsMaxGroups.testIntColMaxAclsWithGroups > org.apache.sentry.tests.e2e.hive.fs.TestHiveWarehouseOnExtFs.TestCreateTable > org.apache.sentry.tests.e2e.hive.fs.TestTableOnExtFS.TestCreateExtTableWithoutScheme -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (SENTRY-1371) Rework Sentry start up and Hive state fetch
[ https://issues.apache.org/jira/browse/SENTRY-1371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hao Hao updated SENTRY-1371: Attachment: SENTRY-1371.003-sentry-ha-redesign.patch > Rework Sentry start up and Hive state fetch > --- > > Key: SENTRY-1371 > URL: https://issues.apache.org/jira/browse/SENTRY-1371 > Project: Sentry > Issue Type: Sub-task > Components: Service >Reporter: Colin P. McCabe >Assignee: Hao Hao > Fix For: sentry-ha-redesign > > Attachments: SENTRY-1371.001-sentry-ha-redesign.patch, > SENTRY-1371.002-sentry-ha-redesign.patch, > SENTRY-1371.003-sentry-ha-redesign.patch > > > Rework Sentry start up and Hive state fetch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (SENTRY-1422) JDO deadlocks while processing grant while a background thread processes Notificationlogs
[
https://issues.apache.org/jira/browse/SENTRY-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432625#comment-15432625
]
Hadoop QA commented on SENTRY-1422:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12825003/SENTRY-1422.004-sentry-ha-redesign.patch
against sentry-ha-redesign.
{color:red}Overall:{color} -1 due to 2 errors
{color:red}ERROR:{color} mvn test exited 1
{color:red}ERROR:{color} Failed:
org.apache.sentry.provider.db.generic.service.persistent.TestPrivilegeOperatePersistence
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/1970/console
This message is automatically generated.
> JDO deadlocks while processing grant while a background thread processes
> Notificationlogs
> -
>
> Key: SENTRY-1422
> URL: https://issues.apache.org/jira/browse/SENTRY-1422
> Project: Sentry
> Issue Type: Sub-task
> Components: Hdfs Plugin
>Reporter: Sravya Tirukkovalur
>Assignee: Colin Ma
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1422.001-sentry-ha-redesign.patch,
> SENTRY-1422.002-sentry-ha-redesign.patch,
> SENTRY-1422.003-sentry-ha-redesign.patch,
> SENTRY-1422.004-sentry-ha-redesign.patch
>
>
> As I was working on Sentry-1321. I see that test case
> TestDbPrivilegeCleanupOnDrop#testDropObjects fails while "Granting select on
> table" with following deadlock.
> {code}
> java.sql.SQLException: Error while processing statement: FAILED: Execution
> Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Unknown error for
> request: TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2,
> requestorUserName:admin1, roleName:select_tbl1,
> privileges:[TSentryPrivilege(privilegeScope:TABLE, serverName:server1,
> dbName:db_2, tableName:tb_1, URI:, action:select, createTime:1469562185573,
> grantOption:FALSE, columnName:)]), message: Iteration request failed : SELECT
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
> FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?. Server
> Stacktrace: javax.jdo.JDODataStoreException: Iteration request failed :
> SELECT 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
> FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
> at
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitTransaction(SentryStore.java:279)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitUpdateTransaction(SentryStore.java:261)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:458)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:261)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
> at
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> NestedThrowablesStackTrace:
> java.sql.SQLTransactionRollbackException: A lock could not be obtained due to
> a deadlock, cycle of locks and waiters is:
> Lock : ROW, SENTRY_ROLE_DB_PRIVILEGE_MAP,
[jira] [Commented] (SENTRY-996) Faulty loop logic in HdfsHAClientInvocationHandler + HAClientInvocationHandler
[
https://issues.apache.org/jira/browse/SENTRY-996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432476#comment-15432476
]
Hadoop QA commented on SENTRY-996:
--
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12824053/SENTRY-996-002.patch
against master.
{color:red}Overall:{color} -1 due to an error
{color:red}ERROR:{color} mvn test exited 1
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/1969/console
This message is automatically generated.
> Faulty loop logic in HdfsHAClientInvocationHandler +
> HAClientInvocationHandler
> ---
>
> Key: SENTRY-996
> URL: https://issues.apache.org/jira/browse/SENTRY-996
> Project: Sentry
> Issue Type: Bug
>Reporter: Colm O hEigeartaigh
>Assignee: Ke Jia
> Attachments: SENTRY-996-002.patch, SENTRY-996.001.patch
>
>
> As discussed on the review board here: https://reviews.apache.org/r/41349/
> Both the HdfsHAClientInvocationHandler + HAClientInvocationHandler contain a
> while(true) loop that looks like it was designed to catch certain exceptions
> + retry. However, the logic is faulty in that it will always return after the
> first iteration, not matter what (as picked up by PMD).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Issue Comment Deleted] (SENTRY-996) Faulty loop logic in HdfsHAClientInvocationHandler + HAClientInvocationHandler
[
https://issues.apache.org/jira/browse/SENTRY-996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dapeng Sun updated SENTRY-996:
--
Comment: was deleted
(was: Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12806107/SENTRY-996.001.patch
against master.
{color:red}Overall:{color} -1 due to 3 errors
{color:red}ERROR:{color} mvn test exited 1
{color:red}ERROR:{color} Failed:
org.apache.sentry.provider.db.service.thrift.TestSentryWebServerWithSSL
{color:red}ERROR:{color} Failed:
org.apache.sentry.provider.db.service.thrift.TestSentryServiceFailureCase
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/1638/console
This message is automatically generated.)
> Faulty loop logic in HdfsHAClientInvocationHandler +
> HAClientInvocationHandler
> ---
>
> Key: SENTRY-996
> URL: https://issues.apache.org/jira/browse/SENTRY-996
> Project: Sentry
> Issue Type: Bug
>Reporter: Colm O hEigeartaigh
>Assignee: Ke Jia
> Attachments: SENTRY-996-002.patch, SENTRY-996.001.patch
>
>
> As discussed on the review board here: https://reviews.apache.org/r/41349/
> Both the HdfsHAClientInvocationHandler + HAClientInvocationHandler contain a
> while(true) loop that looks like it was designed to catch certain exceptions
> + retry. However, the logic is faulty in that it will always return after the
> first iteration, not matter what (as picked up by PMD).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Issue Comment Deleted] (SENTRY-996) Faulty loop logic in HdfsHAClientInvocationHandler + HAClientInvocationHandler
[
https://issues.apache.org/jira/browse/SENTRY-996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dapeng Sun updated SENTRY-996:
--
Comment: was deleted
(was: Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12824053/SENTRY-996-002.patch
against master.
{color:red}Overall:{color} -1 due to 2 errors
{color:red}ERROR:{color} mvn test exited 1
{color:red}ERROR:{color} Failed:
org.apache.sentry.provider.db.generic.service.persistent.TestPrivilegeOperatePersistence
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/1956/console
This message is automatically generated.)
> Faulty loop logic in HdfsHAClientInvocationHandler +
> HAClientInvocationHandler
> ---
>
> Key: SENTRY-996
> URL: https://issues.apache.org/jira/browse/SENTRY-996
> Project: Sentry
> Issue Type: Bug
>Reporter: Colm O hEigeartaigh
>Assignee: Ke Jia
> Attachments: SENTRY-996-002.patch, SENTRY-996.001.patch
>
>
> As discussed on the review board here: https://reviews.apache.org/r/41349/
> Both the HdfsHAClientInvocationHandler + HAClientInvocationHandler contain a
> while(true) loop that looks like it was designed to catch certain exceptions
> + retry. However, the logic is faulty in that it will always return after the
> first iteration, not matter what (as picked up by PMD).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Issue Comment Deleted] (SENTRY-996) Faulty loop logic in HdfsHAClientInvocationHandler + HAClientInvocationHandler
[
https://issues.apache.org/jira/browse/SENTRY-996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dapeng Sun updated SENTRY-996:
--
Comment: was deleted
(was: Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12806107/SENTRY-996.001.patch
against master.
{color:red}Overall:{color} -1 due to an error
{color:red}ERROR:{color} failed to build with patch (exit code 1)
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/1639/console
This message is automatically generated.)
> Faulty loop logic in HdfsHAClientInvocationHandler +
> HAClientInvocationHandler
> ---
>
> Key: SENTRY-996
> URL: https://issues.apache.org/jira/browse/SENTRY-996
> Project: Sentry
> Issue Type: Bug
>Reporter: Colm O hEigeartaigh
>Assignee: Ke Jia
> Attachments: SENTRY-996-002.patch, SENTRY-996.001.patch
>
>
> As discussed on the review board here: https://reviews.apache.org/r/41349/
> Both the HdfsHAClientInvocationHandler + HAClientInvocationHandler contain a
> while(true) loop that looks like it was designed to catch certain exceptions
> + retry. However, the logic is faulty in that it will always return after the
> first iteration, not matter what (as picked up by PMD).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Issue Comment Deleted] (SENTRY-996) Faulty loop logic in HdfsHAClientInvocationHandler + HAClientInvocationHandler
[
https://issues.apache.org/jira/browse/SENTRY-996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dapeng Sun updated SENTRY-996:
--
Comment: was deleted
(was: Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12806107/SENTRY-996.001.patch
against master.
{color:red}Overall:{color} -1 due to an error
{color:red}ERROR:{color} mvn test exited 1
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/1649/console
This message is automatically generated.)
> Faulty loop logic in HdfsHAClientInvocationHandler +
> HAClientInvocationHandler
> ---
>
> Key: SENTRY-996
> URL: https://issues.apache.org/jira/browse/SENTRY-996
> Project: Sentry
> Issue Type: Bug
>Reporter: Colm O hEigeartaigh
>Assignee: Ke Jia
> Attachments: SENTRY-996-002.patch, SENTRY-996.001.patch
>
>
> As discussed on the review board here: https://reviews.apache.org/r/41349/
> Both the HdfsHAClientInvocationHandler + HAClientInvocationHandler contain a
> while(true) loop that looks like it was designed to catch certain exceptions
> + retry. However, the logic is faulty in that it will always return after the
> first iteration, not matter what (as picked up by PMD).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Issue Comment Deleted] (SENTRY-996) Faulty loop logic in HdfsHAClientInvocationHandler + HAClientInvocationHandler
[
https://issues.apache.org/jira/browse/SENTRY-996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dapeng Sun updated SENTRY-996:
--
Comment: was deleted
(was: Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12806107/SENTRY-996.001.patch
against master.
{color:red}Overall:{color} -1 due to an error
{color:red}ERROR:{color} mvn test exited 1
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/1646/console
This message is automatically generated.)
> Faulty loop logic in HdfsHAClientInvocationHandler +
> HAClientInvocationHandler
> ---
>
> Key: SENTRY-996
> URL: https://issues.apache.org/jira/browse/SENTRY-996
> Project: Sentry
> Issue Type: Bug
>Reporter: Colm O hEigeartaigh
>Assignee: Ke Jia
> Attachments: SENTRY-996-002.patch, SENTRY-996.001.patch
>
>
> As discussed on the review board here: https://reviews.apache.org/r/41349/
> Both the HdfsHAClientInvocationHandler + HAClientInvocationHandler contain a
> while(true) loop that looks like it was designed to catch certain exceptions
> + retry. However, the logic is faulty in that it will always return after the
> first iteration, not matter what (as picked up by PMD).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Issue Comment Deleted] (SENTRY-996) Faulty loop logic in HdfsHAClientInvocationHandler + HAClientInvocationHandler
[
https://issues.apache.org/jira/browse/SENTRY-996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dapeng Sun updated SENTRY-996:
--
Comment: was deleted
(was: Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12806107/SENTRY-996.001.patch
against master.
{color:red}Overall:{color} -1 due to an error
{color:red}ERROR:{color} mvn test exited 1
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/1630/console
This message is automatically generated.)
> Faulty loop logic in HdfsHAClientInvocationHandler +
> HAClientInvocationHandler
> ---
>
> Key: SENTRY-996
> URL: https://issues.apache.org/jira/browse/SENTRY-996
> Project: Sentry
> Issue Type: Bug
>Reporter: Colm O hEigeartaigh
>Assignee: Ke Jia
> Attachments: SENTRY-996-002.patch, SENTRY-996.001.patch
>
>
> As discussed on the review board here: https://reviews.apache.org/r/41349/
> Both the HdfsHAClientInvocationHandler + HAClientInvocationHandler contain a
> while(true) loop that looks like it was designed to catch certain exceptions
> + retry. However, the logic is faulty in that it will always return after the
> first iteration, not matter what (as picked up by PMD).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Issue Comment Deleted] (SENTRY-996) Faulty loop logic in HdfsHAClientInvocationHandler + HAClientInvocationHandler
[
https://issues.apache.org/jira/browse/SENTRY-996?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dapeng Sun updated SENTRY-996:
--
Comment: was deleted
(was: Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12806107/SENTRY-996.001.patch
against master.
{color:red}Overall:{color} -1 due to an error
{color:red}ERROR:{color} mvn test exited 1
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/1643/console
This message is automatically generated.)
> Faulty loop logic in HdfsHAClientInvocationHandler +
> HAClientInvocationHandler
> ---
>
> Key: SENTRY-996
> URL: https://issues.apache.org/jira/browse/SENTRY-996
> Project: Sentry
> Issue Type: Bug
>Reporter: Colm O hEigeartaigh
>Assignee: Ke Jia
> Attachments: SENTRY-996-002.patch, SENTRY-996.001.patch
>
>
> As discussed on the review board here: https://reviews.apache.org/r/41349/
> Both the HdfsHAClientInvocationHandler + HAClientInvocationHandler contain a
> while(true) loop that looks like it was designed to catch certain exceptions
> + retry. However, the logic is faulty in that it will always return after the
> first iteration, not matter what (as picked up by PMD).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (SENTRY-1422) JDO deadlocks while processing grant while a background thread processes Notificationlogs
[
https://issues.apache.org/jira/browse/SENTRY-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Ma updated SENTRY-1422:
-
Attachment: SENTRY-1422.004-sentry-ha-redesign.patch
> JDO deadlocks while processing grant while a background thread processes
> Notificationlogs
> -
>
> Key: SENTRY-1422
> URL: https://issues.apache.org/jira/browse/SENTRY-1422
> Project: Sentry
> Issue Type: Sub-task
> Components: Hdfs Plugin
>Reporter: Sravya Tirukkovalur
>Assignee: Colin Ma
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1422.001-sentry-ha-redesign.patch,
> SENTRY-1422.002-sentry-ha-redesign.patch,
> SENTRY-1422.003-sentry-ha-redesign.patch,
> SENTRY-1422.004-sentry-ha-redesign.patch
>
>
> As I was working on Sentry-1321. I see that test case
> TestDbPrivilegeCleanupOnDrop#testDropObjects fails while "Granting select on
> table" with following deadlock.
> {code}
> java.sql.SQLException: Error while processing statement: FAILED: Execution
> Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Unknown error for
> request: TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2,
> requestorUserName:admin1, roleName:select_tbl1,
> privileges:[TSentryPrivilege(privilegeScope:TABLE, serverName:server1,
> dbName:db_2, tableName:tb_1, URI:, action:select, createTime:1469562185573,
> grantOption:FALSE, columnName:)]), message: Iteration request failed : SELECT
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
> FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?. Server
> Stacktrace: javax.jdo.JDODataStoreException: Iteration request failed :
> SELECT 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
> FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
> at
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitTransaction(SentryStore.java:279)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitUpdateTransaction(SentryStore.java:261)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:458)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:261)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
> at
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> NestedThrowablesStackTrace:
> java.sql.SQLTransactionRollbackException: A lock could not be obtained due to
> a deadlock, cycle of locks and waiters is:
> Lock : ROW, SENTRY_ROLE_DB_PRIVILEGE_MAP, (1,10)
> Waiting XID : {712, S} , SENTRY, SELECT
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
> FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
> Granted XID : {704, X}
> Lock : ROW, SENTRY_DB_PRIVILEGE, (1,11)
>
[jira] [Commented] (SENTRY-1422) JDO deadlocks while processing grant while a background thread processes Notificationlogs
[
https://issues.apache.org/jira/browse/SENTRY-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432288#comment-15432288
]
Hadoop QA commented on SENTRY-1422:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12824968/SENTRY-1422.003-sentry-ha-redesign.patch
against sentry-ha-redesign.
{color:red}Overall:{color} -1 due to 2 errors
{color:red}ERROR:{color} mvn test exited 1
{color:red}ERROR:{color} Failed: org.apache.sentry.tests.e2e.kafka.TestAclsCrud
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/1968/console
This message is automatically generated.
> JDO deadlocks while processing grant while a background thread processes
> Notificationlogs
> -
>
> Key: SENTRY-1422
> URL: https://issues.apache.org/jira/browse/SENTRY-1422
> Project: Sentry
> Issue Type: Sub-task
> Components: Hdfs Plugin
>Reporter: Sravya Tirukkovalur
>Assignee: Colin Ma
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1422.001-sentry-ha-redesign.patch,
> SENTRY-1422.002-sentry-ha-redesign.patch,
> SENTRY-1422.003-sentry-ha-redesign.patch
>
>
> As I was working on Sentry-1321. I see that test case
> TestDbPrivilegeCleanupOnDrop#testDropObjects fails while "Granting select on
> table" with following deadlock.
> {code}
> java.sql.SQLException: Error while processing statement: FAILED: Execution
> Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Unknown error for
> request: TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2,
> requestorUserName:admin1, roleName:select_tbl1,
> privileges:[TSentryPrivilege(privilegeScope:TABLE, serverName:server1,
> dbName:db_2, tableName:tb_1, URI:, action:select, createTime:1469562185573,
> grantOption:FALSE, columnName:)]), message: Iteration request failed : SELECT
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
> FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?. Server
> Stacktrace: javax.jdo.JDODataStoreException: Iteration request failed :
> SELECT 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
> FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
> at
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitTransaction(SentryStore.java:279)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitUpdateTransaction(SentryStore.java:261)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:458)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:261)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
> at
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> NestedThrowablesStackTrace:
> java.sql.SQLTransactionRollbackException: A lock could not be obtained due to
> a deadlock, cycle of locks and waiters is:
> Lock : ROW, SENTRY_ROLE_DB_PRIVILEGE_MAP, (1,10)
> Waiting XID : {712, S} , SENTRY, SELECT
>
[jira] [Issue Comment Deleted] (SENTRY-1422) JDO deadlocks while processing grant while a background thread processes Notificationlogs
[
https://issues.apache.org/jira/browse/SENTRY-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Ma updated SENTRY-1422:
-
Comment: was deleted
(was: Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12824968/SENTRY-1422.003-sentry-ha-redesign.patch
against sentry-ha-redesign.
{color:red}Overall:{color} -1 due to 2 errors
{color:red}ERROR:{color} mvn test exited 1
{color:red}ERROR:{color} Failed:
org.apache.sentry.tests.e2e.solr.db.integration.TestSolrAdminOperations
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/1965/console
This message is automatically generated.)
> JDO deadlocks while processing grant while a background thread processes
> Notificationlogs
> -
>
> Key: SENTRY-1422
> URL: https://issues.apache.org/jira/browse/SENTRY-1422
> Project: Sentry
> Issue Type: Sub-task
> Components: Hdfs Plugin
>Reporter: Sravya Tirukkovalur
>Assignee: Colin Ma
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1422.001-sentry-ha-redesign.patch,
> SENTRY-1422.002-sentry-ha-redesign.patch,
> SENTRY-1422.003-sentry-ha-redesign.patch
>
>
> As I was working on Sentry-1321. I see that test case
> TestDbPrivilegeCleanupOnDrop#testDropObjects fails while "Granting select on
> table" with following deadlock.
> {code}
> java.sql.SQLException: Error while processing statement: FAILED: Execution
> Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Unknown error for
> request: TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2,
> requestorUserName:admin1, roleName:select_tbl1,
> privileges:[TSentryPrivilege(privilegeScope:TABLE, serverName:server1,
> dbName:db_2, tableName:tb_1, URI:, action:select, createTime:1469562185573,
> grantOption:FALSE, columnName:)]), message: Iteration request failed : SELECT
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
> FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?. Server
> Stacktrace: javax.jdo.JDODataStoreException: Iteration request failed :
> SELECT 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
> FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
> at
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitTransaction(SentryStore.java:279)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitUpdateTransaction(SentryStore.java:261)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:458)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:261)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
> at
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> NestedThrowablesStackTrace:
> java.sql.SQLTransactionRollbackException: A lock could not be obtained due to
> a deadlock, cycle of locks and waiters is:
> Lock : ROW, SENTRY_ROLE_DB_PRIVILEGE_MAP, (1,10)
> Waiting XID : {712, S} , SENTRY, SELECT
>
[jira] [Issue Comment Deleted] (SENTRY-1422) JDO deadlocks while processing grant while a background thread processes Notificationlogs
[
https://issues.apache.org/jira/browse/SENTRY-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Ma updated SENTRY-1422:
-
Comment: was deleted
(was: Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12824759/SENTRY-1422.001-sentry-ha-redesign.patch
against sentry-ha-redesign.
{color:red}Overall:{color} -1 due to 24 errors
{color:red}ERROR:{color} mvn test exited 1
{color:red}ERROR:{color} Failed:
org.apache.sentry.provider.db.service.thrift.TestSentryServiceWithInvalidMsgSize
{color:red}ERROR:{color} Failed:
org.apache.sentry.provider.db.service.thrift.TestSentryServiceClientPool
{color:red}ERROR:{color} Failed:
org.apache.sentry.provider.db.generic.service.persistent.TestPrivilegeOperatePersistence
{color:red}ERROR:{color} Failed:
org.apache.sentry.provider.db.generic.service.persistent.TestPrivilegeOperatePersistence
{color:red}ERROR:{color} Failed:
org.apache.sentry.provider.db.generic.service.persistent.TestPrivilegeOperatePersistence
{color:red}ERROR:{color} Failed:
org.apache.sentry.provider.db.generic.service.persistent.TestPrivilegeOperatePersistence
{color:red}ERROR:{color} Failed:
org.apache.sentry.provider.db.generic.service.persistent.TestPrivilegeOperatePersistence
{color:red}ERROR:{color} Failed:
org.apache.sentry.provider.db.generic.service.persistent.TestPrivilegeOperatePersistence
{color:red}ERROR:{color} Failed:
org.apache.sentry.provider.db.generic.service.persistent.TestPrivilegeOperatePersistence
{color:red}ERROR:{color} Failed:
org.apache.sentry.provider.db.generic.service.persistent.TestPrivilegeOperatePersistence
{color:red}ERROR:{color} Failed:
org.apache.sentry.provider.db.generic.service.persistent.TestPrivilegeOperatePersistence
{color:red}ERROR:{color} Failed:
org.apache.sentry.provider.db.generic.service.persistent.TestPrivilegeOperatePersistence
{color:red}ERROR:{color} Failed:
org.apache.sentry.provider.db.generic.service.persistent.TestPrivilegeOperatePersistence
{color:red}ERROR:{color} Failed:
org.apache.sentry.provider.db.generic.service.persistent.TestPrivilegeOperatePersistence
{color:red}ERROR:{color} Failed:
org.apache.sentry.provider.db.generic.service.persistent.TestPrivilegeOperatePersistence
{color:red}ERROR:{color} Failed:
org.apache.sentry.provider.db.generic.service.persistent.TestPrivilegeOperatePersistence
{color:red}ERROR:{color} Failed:
org.apache.sentry.provider.db.generic.service.persistent.TestPrivilegeOperatePersistence
{color:red}ERROR:{color} Failed:
org.apache.sentry.provider.db.generic.service.persistent.TestPrivilegeOperatePersistence
{color:red}ERROR:{color} Failed:
org.apache.sentry.provider.db.generic.service.persistent.TestPrivilegeOperatePersistence
{color:red}ERROR:{color} Failed:
org.apache.sentry.provider.db.generic.service.persistent.TestPrivilegeOperatePersistence
{color:red}ERROR:{color} Failed:
org.apache.sentry.provider.db.generic.service.persistent.TestPrivilegeOperatePersistence
{color:red}ERROR:{color} Failed:
org.apache.sentry.provider.db.generic.service.persistent.TestPrivilegeOperatePersistence
{color:red}ERROR:{color} Failed:
org.apache.sentry.provider.db.generic.service.persistent.TestPrivilegeOperatePersistence
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/1962/console
This message is automatically generated.)
> JDO deadlocks while processing grant while a background thread processes
> Notificationlogs
> -
>
> Key: SENTRY-1422
> URL: https://issues.apache.org/jira/browse/SENTRY-1422
> Project: Sentry
> Issue Type: Sub-task
> Components: Hdfs Plugin
>Reporter: Sravya Tirukkovalur
>Assignee: Colin Ma
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1422.001-sentry-ha-redesign.patch,
> SENTRY-1422.002-sentry-ha-redesign.patch,
> SENTRY-1422.003-sentry-ha-redesign.patch
>
>
> As I was working on Sentry-1321. I see that test case
> TestDbPrivilegeCleanupOnDrop#testDropObjects fails while "Granting select on
> table" with following deadlock.
> {code}
> java.sql.SQLException: Error while processing statement: FAILED: Execution
> Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Unknown error for
> request: TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2,
> requestorUserName:admin1, roleName:select_tbl1,
> privileges:[TSentryPrivilege(privilegeScope:TABLE, serverName:server1,
> dbName:db_2, tableName:tb_1, URI:, action:select, createTime:1469562185573,
> grantOption:FALSE, columnName:)]), message: Iteration request failed : SELECT
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS
>
[jira] [Issue Comment Deleted] (SENTRY-1422) JDO deadlocks while processing grant while a background thread processes Notificationlogs
[
https://issues.apache.org/jira/browse/SENTRY-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Ma updated SENTRY-1422:
-
Comment: was deleted
(was: Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12824790/SENTRY-1422.002-sentry-ha-redesign.patch
against sentry-ha-redesign.
{color:red}Overall:{color} -1 due to 6 errors
{color:red}ERROR:{color} mvn test exited 1
{color:red}ERROR:{color} Failed:
org.apache.sentry.provider.db.service.thrift.TestSentryServiceWithInvalidMsgSize
{color:red}ERROR:{color} Failed:
org.apache.sentry.service.thrift.TestHMSFollower
{color:red}ERROR:{color} Failed:
org.apache.sentry.service.thrift.TestHMSFollower
{color:red}ERROR:{color} Failed:
org.apache.sentry.provider.db.service.thrift.TestSentryServiceClientPool
{color:red}ERROR:{color} Failed:
org.apache.sentry.provider.db.service.thrift.TestSentryServiceClientPool
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/1963/console
This message is automatically generated.)
> JDO deadlocks while processing grant while a background thread processes
> Notificationlogs
> -
>
> Key: SENTRY-1422
> URL: https://issues.apache.org/jira/browse/SENTRY-1422
> Project: Sentry
> Issue Type: Sub-task
> Components: Hdfs Plugin
>Reporter: Sravya Tirukkovalur
>Assignee: Colin Ma
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1422.001-sentry-ha-redesign.patch,
> SENTRY-1422.002-sentry-ha-redesign.patch,
> SENTRY-1422.003-sentry-ha-redesign.patch
>
>
> As I was working on Sentry-1321. I see that test case
> TestDbPrivilegeCleanupOnDrop#testDropObjects fails while "Granting select on
> table" with following deadlock.
> {code}
> java.sql.SQLException: Error while processing statement: FAILED: Execution
> Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Unknown error for
> request: TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2,
> requestorUserName:admin1, roleName:select_tbl1,
> privileges:[TSentryPrivilege(privilegeScope:TABLE, serverName:server1,
> dbName:db_2, tableName:tb_1, URI:, action:select, createTime:1469562185573,
> grantOption:FALSE, columnName:)]), message: Iteration request failed : SELECT
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
> FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?. Server
> Stacktrace: javax.jdo.JDODataStoreException: Iteration request failed :
> SELECT 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
> FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
> at
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitTransaction(SentryStore.java:279)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitUpdateTransaction(SentryStore.java:261)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:458)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:261)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
> at
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
>
[jira] [Commented] (SENTRY-1422) JDO deadlocks while processing grant while a background thread processes Notificationlogs
[
https://issues.apache.org/jira/browse/SENTRY-1422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15432220#comment-15432220
]
Hadoop QA commented on SENTRY-1422:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12824968/SENTRY-1422.003-sentry-ha-redesign.patch
against sentry-ha-redesign.
{color:red}Overall:{color} -1 due to 2 errors
{color:red}ERROR:{color} mvn test exited 1
{color:red}ERROR:{color} Failed: org.apache.sentry.tests.e2e.kafka.TestAuthorize
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/1967/console
This message is automatically generated.
> JDO deadlocks while processing grant while a background thread processes
> Notificationlogs
> -
>
> Key: SENTRY-1422
> URL: https://issues.apache.org/jira/browse/SENTRY-1422
> Project: Sentry
> Issue Type: Sub-task
> Components: Hdfs Plugin
>Reporter: Sravya Tirukkovalur
>Assignee: Colin Ma
> Fix For: sentry-ha-redesign
>
> Attachments: SENTRY-1422.001-sentry-ha-redesign.patch,
> SENTRY-1422.002-sentry-ha-redesign.patch,
> SENTRY-1422.003-sentry-ha-redesign.patch
>
>
> As I was working on Sentry-1321. I see that test case
> TestDbPrivilegeCleanupOnDrop#testDropObjects fails while "Granting select on
> table" with following deadlock.
> {code}
> java.sql.SQLException: Error while processing statement: FAILED: Execution
> Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.SentryGrantRevokeTask. Unknown error for
> request: TAlterSentryRoleGrantPrivilegeRequest(protocol_version:2,
> requestorUserName:admin1, roleName:select_tbl1,
> privileges:[TSentryPrivilege(privilegeScope:TABLE, serverName:server1,
> dbName:db_2, tableName:tb_1, URI:, action:select, createTime:1469562185573,
> grantOption:FALSE, columnName:)]), message: Iteration request failed : SELECT
> 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
> FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?. Server
> Stacktrace: javax.jdo.JDODataStoreException: Iteration request failed :
> SELECT 'org.apache.sentry.provider.db.service.model.MSentryPrivilege' AS
> NUCLEUS_TYPE,A1.URI,A1."ACTION",A1."COLUMN_NAME",A1.CREATE_TIME,A1.DB_NAME,A1.WITH_GRANT_OPTION,A1.PRIVILEGE_SCOPE,A1."SERVER_NAME",A1."TABLE_NAME",A1.DB_PRIVILEGE_ID
> FROM SENTRY_ROLE_DB_PRIVILEGE_MAP A0 INNER JOIN SENTRY_DB_PRIVILEGE A1 ON
> A0.DB_PRIVILEGE_ID = A1.DB_PRIVILEGE_ID WHERE A0.ROLE_ID = ?
> at
> org.datanucleus.api.jdo.NucleusJDOHelper.getJDOExceptionForNucleusException(NucleusJDOHelper.java:451)
> at
> org.datanucleus.api.jdo.JDOTransaction.commit(JDOTransaction.java:165)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitTransaction(SentryStore.java:279)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.commitUpdateTransaction(SentryStore.java:261)
> at
> org.apache.sentry.provider.db.service.persistent.SentryStore.alterSentryRoleGrantPrivileges(SentryStore.java:458)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyStoreProcessor.alter_sentry_role_grant_privilege(SentryPolicyStoreProcessor.java:261)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1237)
> at
> org.apache.sentry.provider.db.service.thrift.SentryPolicyService$Processor$alter_sentry_role_grant_privilege.getResult(SentryPolicyService.java:1222)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> at
> org.apache.sentry.provider.db.service.thrift.SentryProcessorWrapper.process(SentryProcessorWrapper.java:35)
> at
> org.apache.thrift.TMultiplexedProcessor.process(TMultiplexedProcessor.java:123)
> at
> org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> NestedThrowablesStackTrace:
> java.sql.SQLTransactionRollbackException: A lock could not be obtained due to
> a deadlock, cycle of locks and waiters is:
> Lock : ROW, SENTRY_ROLE_DB_PRIVILEGE_MAP, (1,10)
> Waiting XID : {712, S} , SENTRY, SELECT
>
