[jira] [Commented] (SENTRY-2476) Optimize deleting specific paths for objects
[
https://issues.apache.org/jira/browse/SENTRY-2476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713452#comment-16713452
]
Hadoop QA commented on SENTRY-2476:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12951057/SENTRY-2476.01.patch
against master.
{color:green}Overall:{color} +1 all checks pass
{color:green}SUCCESS:{color} all tests passed
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/4281/console
This message is automatically generated.
> Optimize deleting specific paths for objects
>
>
> Key: SENTRY-2476
> URL: https://issues.apache.org/jira/browse/SENTRY-2476
> Project: Sentry
> Issue Type: Bug
>Reporter: Arjun Mishra
>Assignee: Arjun Mishra
>Priority: Major
> Attachments: SENTRY-2476.01.patch, SENTRY-2476.02.patch
>
>
> Right now when we process a drop partition event, we fetch each path object
> for paths_mapping object then find the one we want to delete and then delete
> it. We should avoid fetching all objects and directly delete the path that
> needs to be deleted
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SENTRY-2476) Optimize deleting specific paths for objects
[
https://issues.apache.org/jira/browse/SENTRY-2476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713420#comment-16713420
]
Hadoop QA commented on SENTRY-2476:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12951065/SENTRY-2476.02.patch
against master.
{color:red}Overall:{color} -1 due to 3 errors
{color:red}ERROR:{color} mvn test exited 1
{color:red}ERROR:{color} Failed:
org.apache.sentry.provider.db.service.persistent.TestSentryStore
{color:red}ERROR:{color} Failed:
org.apache.sentry.provider.db.service.persistent.TestSentryStore
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/4282/console
This message is automatically generated.
> Optimize deleting specific paths for objects
>
>
> Key: SENTRY-2476
> URL: https://issues.apache.org/jira/browse/SENTRY-2476
> Project: Sentry
> Issue Type: Bug
>Reporter: Arjun Mishra
>Assignee: Arjun Mishra
>Priority: Major
> Attachments: SENTRY-2476.01.patch, SENTRY-2476.02.patch
>
>
> Right now when we process a drop partition event, we fetch each path object
> for paths_mapping object then find the one we want to delete and then delete
> it. We should avoid fetching all objects and directly delete the path that
> needs to be deleted
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (SENTRY-2476) Optimize deleting specific paths for objects
[ https://issues.apache.org/jira/browse/SENTRY-2476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arjun Mishra updated SENTRY-2476: - Attachment: SENTRY-2476.02.patch > Optimize deleting specific paths for objects > > > Key: SENTRY-2476 > URL: https://issues.apache.org/jira/browse/SENTRY-2476 > Project: Sentry > Issue Type: Bug >Reporter: Arjun Mishra >Assignee: Arjun Mishra >Priority: Major > Attachments: SENTRY-2476.01.patch, SENTRY-2476.02.patch > > > Right now when we process a drop partition event, we fetch each path object > for paths_mapping object then find the one we want to delete and then delete > it. We should avoid fetching all objects and directly delete the path that > needs to be deleted -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (SENTRY-2476) Optimize deleting specific paths for objects
[
https://issues.apache.org/jira/browse/SENTRY-2476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713361#comment-16713361
]
Arjun Mishra edited comment on SENTRY-2476 at 12/7/18 11:27 PM:
Found a new solution that deletes paths and processes events in less than 10
ms. *_The optimization is aimed at not caching as many objects_*. This was
primarily triggered by the *_paths_* attribute in MAuthsPathsMapping class. See
below for breakdown of steps
# Get mapping objects for object name
{noformat}
2018-12-07 14:05:12,670 INFO
org.apache.sentry.provider.db.service.persistent.SentryStore: Start
getAllMAuthzPathsMappingCore for object dbx.tbly
2018-12-07 14:05:12,670 DEBUG DataNucleus.Persistence:
ExecutionContext.internalFlush() process started using ordered flush - 2
enlisted objects
2018-12-07 14:05:12,670 DEBUG DataNucleus.Persistence:
ExecutionContext.internalFlush() process finished
2018-12-07 14:05:12,671 DEBUG DataNucleus.Connection: Connection found in the
pool :
org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl@1ad3c7d5
[conn=com.jolbox.bonecp.ConnectionHandle@16c2b4fe, commitOnRelease=false,
closeOnRelease=false, closeOnTxnEnd=true] for
key=org.datanucleus.ExecutionContextThreadedImpl@50b469b4 in
factory=ConnectionFactory:tx[org.datanucleus.store.rdbms.ConnectionFactoryImpl@5d5d9e5]
2018-12-07 14:05:12,671 DEBUG DataNucleus.Datastore.Native: BATCH [INSERT INTO
`SENTRY_HMS_NOTIFICATION_ID` (`NOTIFICATION_ID`) VALUES (<21958>)]
2018-12-07 14:05:12,671 DEBUG DataNucleus.Datastore: Execution Time = 0 ms
(number of rows = [1]) on PreparedStatement
"org.datanucleus.store.rdbms.ParamLoggingPreparedStatement@496079d2"
2018-12-07 14:05:12,671 DEBUG DataNucleus.Datastore: Closing PreparedStatement
"com.jolbox.bonecp.PreparedStatementHandle@254a68f2"
2018-12-07 14:05:12,672 DEBUG DataNucleus.Datastore.Native: SELECT
'org.apache.sentry.provider.db.service.model.MAuthzPathsMapping' AS
NUCLEUS_TYPE,`A0`.`AUTHZ_OBJ_NAME`,`A0`.`AUTHZ_SNAPSHOT_ID`,`A0`.`CREATE_TIME_MS`,`A0`.`AUTHZ_OBJ_ID`
FROM `AUTHZ_PATHS_MAPPING` `A0` WHERE `A0`.`AUTHZ_OBJ_NAME` = <'dbx.tbly'>
2018-12-07 14:05:12,672 DEBUG DataNucleus.Datastore.Retrieve: Execution Time =
0 ms
2018-12-07 14:05:12,672 INFO
org.apache.sentry.provider.db.service.persistent.SentryStore: End
getAllMAuthzPathsMappingCore for object dbx.tbly
{noformat}
# Get MPath objects for path name
{noformat}
2018-12-07 14:05:12,672 INFO
org.apache.sentry.provider.db.service.persistent.SentryStore: Start
getMAuthzPathsCore for path [user/hive/warehouse/dbx.db/tbly/col10=1]
2018-12-07 14:05:12,672 DEBUG DataNucleus.Connection: Connection found in the
pool :
org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl@1ad3c7d5
[conn=com.jolbox.bonecp.ConnectionHandle@16c2b4fe, commitOnRelease=false,
closeOnRelease=false, closeOnTxnEnd=true] for
key=org.datanucleus.ExecutionContextThreadedImpl@50b469b4 in
factory=ConnectionFactory:tx[org.datanucleus.store.rdbms.ConnectionFactoryImpl@5d5d9e5]
2018-12-07 14:05:12,672 DEBUG DataNucleus.Datastore: Closing PreparedStatement
"com.jolbox.bonecp.PreparedStatementHandle@2be897c0"
2018-12-07 14:05:12,672 DEBUG DataNucleus.Datastore.Native: SELECT
'org.apache.sentry.provider.db.service.model.MPath' AS
NUCLEUS_TYPE,`A0`.`PATH_NAME`,`A0`.`PATH_ID` FROM `AUTHZ_PATH` `A0` WHERE
`A0`.`PATH_NAME` = <'user/hive/warehouse/dbx.db/tbly/col10=1'>
2018-12-07 14:05:12,676 DEBUG DataNucleus.Datastore.Retrieve: Execution Time =
4 ms
2018-12-07 14:05:12,676 DEBUG DataNucleus.Persistence: Retrieved object with
OID "11255[OID]org.apache.sentry.provider.db.service.model.MPath"
2018-12-07 14:05:12,676 DEBUG DataNucleus.Cache: Object with id
"11255[OID]org.apache.sentry.provider.db.service.model.MPath" not found in
Level 1 cache [cache size = 2]
2018-12-07 14:05:12,676 DEBUG DataNucleus.Cache: Object
"org.apache.sentry.provider.db.service.model.MPath@48707c3"
(id="11255[OID]org.apache.sentry.provider.db.service.model.MPath") added to
Level 1 cache (loadedFlags="[N]")
2018-12-07 14:05:12,676 INFO
org.apache.sentry.provider.db.service.persistent.SentryStore: End
getMAuthzPathsCore for path [user/hive/warehouse/dbx.db/tbly/col10=1]. Size 1
{noformat}
# Delete the path objects
{noformat}
2018-12-07 14:05:12,676 INFO
org.apache.sentry.provider.db.service.persistent.SentryStore: Start deleting
all objects for path [user/hive/warehouse/dbx.db/tbly/col10=1]
2018-12-07 14:05:12,676 DEBUG DataNucleus.Persistence: Deleting object from
persistence : "org.apache.sentry.provider.db.service.model.MPath@48707c3"
2018-12-07 14:05:12,676 DEBUG DataNucleus.Lifecycle: Object
"org.apache.sentry.provider.db.service.model.MPath@48707c3"
(id="11255[OID]org.apache.sentry.provider.db.service.model.MPath") has a
lifecycle change : "HOLLOW"->"P_DELETED"
2018-12-07 14:05:12,676 DEBUG DataNucleus.Transaction: Object
[jira] [Updated] (SENTRY-2476) Optimize deleting specific paths for objects
[ https://issues.apache.org/jira/browse/SENTRY-2476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arjun Mishra updated SENTRY-2476: - Attachment: SENTRY-2476.01.patch > Optimize deleting specific paths for objects > > > Key: SENTRY-2476 > URL: https://issues.apache.org/jira/browse/SENTRY-2476 > Project: Sentry > Issue Type: Bug >Reporter: Arjun Mishra >Assignee: Arjun Mishra >Priority: Major > Attachments: SENTRY-2476.01.patch > > > Right now when we process a drop partition event, we fetch each path object > for paths_mapping object then find the one we want to delete and then delete > it. We should avoid fetching all objects and directly delete the path that > needs to be deleted -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SENTRY-2476) Optimize deleting specific paths for objects
[ https://issues.apache.org/jira/browse/SENTRY-2476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arjun Mishra updated SENTRY-2476: - Status: Patch Available (was: Open) > Optimize deleting specific paths for objects > > > Key: SENTRY-2476 > URL: https://issues.apache.org/jira/browse/SENTRY-2476 > Project: Sentry > Issue Type: Bug >Reporter: Arjun Mishra >Assignee: Arjun Mishra >Priority: Major > Attachments: SENTRY-2476.01.patch > > > Right now when we process a drop partition event, we fetch each path object > for paths_mapping object then find the one we want to delete and then delete > it. We should avoid fetching all objects and directly delete the path that > needs to be deleted -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2476) Optimize deleting specific paths for objects
[
https://issues.apache.org/jira/browse/SENTRY-2476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713361#comment-16713361
]
Arjun Mishra commented on SENTRY-2476:
--
Found a new solution that deletes paths and processes events in less than 10
ms. The optimization is aimed at not caching objects. This was primarily
triggered by the *_paths_* attribute in MAuthsPathsMapping class. See below for
breakdown of steps
# Get mapping objects for object name
{noformat}
2018-12-07 14:05:12,670 INFO
org.apache.sentry.provider.db.service.persistent.SentryStore: Start
getAllMAuthzPathsMappingCore for object dbx.tbly
2018-12-07 14:05:12,670 DEBUG DataNucleus.Persistence:
ExecutionContext.internalFlush() process started using ordered flush - 2
enlisted objects
2018-12-07 14:05:12,670 DEBUG DataNucleus.Persistence:
ExecutionContext.internalFlush() process finished
2018-12-07 14:05:12,671 DEBUG DataNucleus.Connection: Connection found in the
pool :
org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl@1ad3c7d5
[conn=com.jolbox.bonecp.ConnectionHandle@16c2b4fe, commitOnRelease=false,
closeOnRelease=false, closeOnTxnEnd=true] for
key=org.datanucleus.ExecutionContextThreadedImpl@50b469b4 in
factory=ConnectionFactory:tx[org.datanucleus.store.rdbms.ConnectionFactoryImpl@5d5d9e5]
2018-12-07 14:05:12,671 DEBUG DataNucleus.Datastore.Native: BATCH [INSERT INTO
`SENTRY_HMS_NOTIFICATION_ID` (`NOTIFICATION_ID`) VALUES (<21958>)]
2018-12-07 14:05:12,671 DEBUG DataNucleus.Datastore: Execution Time = 0 ms
(number of rows = [1]) on PreparedStatement
"org.datanucleus.store.rdbms.ParamLoggingPreparedStatement@496079d2"
2018-12-07 14:05:12,671 DEBUG DataNucleus.Datastore: Closing PreparedStatement
"com.jolbox.bonecp.PreparedStatementHandle@254a68f2"
2018-12-07 14:05:12,672 DEBUG DataNucleus.Datastore.Native: SELECT
'org.apache.sentry.provider.db.service.model.MAuthzPathsMapping' AS
NUCLEUS_TYPE,`A0`.`AUTHZ_OBJ_NAME`,`A0`.`AUTHZ_SNAPSHOT_ID`,`A0`.`CREATE_TIME_MS`,`A0`.`AUTHZ_OBJ_ID`
FROM `AUTHZ_PATHS_MAPPING` `A0` WHERE `A0`.`AUTHZ_OBJ_NAME` = <'dbx.tbly'>
2018-12-07 14:05:12,672 DEBUG DataNucleus.Datastore.Retrieve: Execution Time =
0 ms
2018-12-07 14:05:12,672 INFO
org.apache.sentry.provider.db.service.persistent.SentryStore: End
getAllMAuthzPathsMappingCore for object dbx.tbly
{noformat}
# Get MPath objects for path name
{noformat}
2018-12-07 14:05:12,672 INFO
org.apache.sentry.provider.db.service.persistent.SentryStore: Start
getMAuthzPathsCore for path [user/hive/warehouse/dbx.db/tbly/col10=1]
2018-12-07 14:05:12,672 DEBUG DataNucleus.Connection: Connection found in the
pool :
org.datanucleus.store.rdbms.ConnectionFactoryImpl$ManagedConnectionImpl@1ad3c7d5
[conn=com.jolbox.bonecp.ConnectionHandle@16c2b4fe, commitOnRelease=false,
closeOnRelease=false, closeOnTxnEnd=true] for
key=org.datanucleus.ExecutionContextThreadedImpl@50b469b4 in
factory=ConnectionFactory:tx[org.datanucleus.store.rdbms.ConnectionFactoryImpl@5d5d9e5]
2018-12-07 14:05:12,672 DEBUG DataNucleus.Datastore: Closing PreparedStatement
"com.jolbox.bonecp.PreparedStatementHandle@2be897c0"
2018-12-07 14:05:12,672 DEBUG DataNucleus.Datastore.Native: SELECT
'org.apache.sentry.provider.db.service.model.MPath' AS
NUCLEUS_TYPE,`A0`.`PATH_NAME`,`A0`.`PATH_ID` FROM `AUTHZ_PATH` `A0` WHERE
`A0`.`PATH_NAME` = <'user/hive/warehouse/dbx.db/tbly/col10=1'>
2018-12-07 14:05:12,676 DEBUG DataNucleus.Datastore.Retrieve: Execution Time =
4 ms
2018-12-07 14:05:12,676 DEBUG DataNucleus.Persistence: Retrieved object with
OID "11255[OID]org.apache.sentry.provider.db.service.model.MPath"
2018-12-07 14:05:12,676 DEBUG DataNucleus.Cache: Object with id
"11255[OID]org.apache.sentry.provider.db.service.model.MPath" not found in
Level 1 cache [cache size = 2]
2018-12-07 14:05:12,676 DEBUG DataNucleus.Cache: Object
"org.apache.sentry.provider.db.service.model.MPath@48707c3"
(id="11255[OID]org.apache.sentry.provider.db.service.model.MPath") added to
Level 1 cache (loadedFlags="[N]")
2018-12-07 14:05:12,676 INFO
org.apache.sentry.provider.db.service.persistent.SentryStore: End
getMAuthzPathsCore for path [user/hive/warehouse/dbx.db/tbly/col10=1]. Size 1
{noformat}
# Delete the path objects
{noformat}
2018-12-07 14:05:12,676 INFO
org.apache.sentry.provider.db.service.persistent.SentryStore: Start deleting
all objects for path [user/hive/warehouse/dbx.db/tbly/col10=1]
2018-12-07 14:05:12,676 DEBUG DataNucleus.Persistence: Deleting object from
persistence : "org.apache.sentry.provider.db.service.model.MPath@48707c3"
2018-12-07 14:05:12,676 DEBUG DataNucleus.Lifecycle: Object
"org.apache.sentry.provider.db.service.model.MPath@48707c3"
(id="11255[OID]org.apache.sentry.provider.db.service.model.MPath") has a
lifecycle change : "HOLLOW"->"P_DELETED"
2018-12-07 14:05:12,676 DEBUG DataNucleus.Transaction: Object
"org.apache.sentry.provider.db.service.model.MPath@48707c3"
[jira] [Commented] (SENTRY-2454) Add new sentry store api to gather the privileges for a list of authorizables.
[
https://issues.apache.org/jira/browse/SENTRY-2454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713110#comment-16713110
]
Hadoop QA commented on SENTRY-2454:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12951002/SENTRY-2454.002.patch
against master.
{color:green}Overall:{color} +1 all checks pass
{color:green}SUCCESS:{color} all tests passed
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/4279/console
This message is automatically generated.
> Add new sentry store api to gather the privileges for a list of authorizables.
> --
>
> Key: SENTRY-2454
> URL: https://issues.apache.org/jira/browse/SENTRY-2454
> Project: Sentry
> Issue Type: Improvement
> Components: Sentry
>Affects Versions: 2.2.0
>Reporter: kalyan kumar kalvagadda
>Assignee: kalyan kumar kalvagadda
>Priority: Major
> Attachments: SENTRY-2454.001.patch, SENTRY-2454.002.patch
>
>
> New sentry API should be implemented to fetch the privileges granted to
> authorizables and it's children. authorizables include database, tables,
> columns and URI's.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (SENTRY-2452) Change the thrift interface to send the list of authorizable to sentry server
[
https://issues.apache.org/jira/browse/SENTRY-2452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713105#comment-16713105
]
Hadoop QA commented on SENTRY-2452:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12951001/SENTRY-2452.003.patch
against master.
{color:green}Overall:{color} +1 all checks pass
{color:green}SUCCESS:{color} all tests passed
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/4280/console
This message is automatically generated.
> Change the thrift interface to send the list of authorizable to sentry server
> -
>
> Key: SENTRY-2452
> URL: https://issues.apache.org/jira/browse/SENTRY-2452
> Project: Sentry
> Issue Type: Improvement
>Reporter: kalyan kumar kalvagadda
>Assignee: kalyan kumar kalvagadda
>Priority: Major
> Attachments: SENTRY-2452.001.patch, SENTRY-2452.003.patch
>
>
> TSentryExportMappingDataRequest and TSentryImportMappingDataRequest which are
> used to send export and import requests to sentry server should be changed to
> be able to send a list authorizables for which permission information has to
> be exported/imported.
> These requests should accommodate source and target cluster information as
> well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (SENTRY-2468) Replicate owner privileges.
[ https://issues.apache.org/jira/browse/SENTRY-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda resolved SENTRY-2468. - Resolution: Duplicate > Replicate owner privileges. > --- > > Key: SENTRY-2468 > URL: https://issues.apache.org/jira/browse/SENTRY-2468 > Project: Sentry > Issue Type: Sub-task > Components: Sentry >Affects Versions: 2.2.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > > In general exporting owner privileges is an unnecessary overhead for couple > of reasons > # Owner privileges are updated automatically in the target cluster when Hive > replication is performed on the target cluster. > # Number of owner privileges could be huge and the magnitude of that would > increase with increase in the number of databases and tables. > > There are still some use cases where there is a need to replicate owner > privilege as well. Here are some of them > # When sentry is not configured in the target cluster when owner privileges > feature is enabled in source cluster. > # When owner privileges feature is enabled in source cluster and not in > target cluster. > In both the above case, target cluster will not have owner privileges for the > hive objects that are replicated. > > In summary, replicating owner privilege should be configurable.It is better > to have it as an input from the administrator performing the export/import > operations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2468) Replicate owner privileges.
[ https://issues.apache.org/jira/browse/SENTRY-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713000#comment-16713000 ] kalyan kumar kalvagadda commented on SENTRY-2468: - [~spena], I agree. I will resolve this Jira as i have created # SENTRY-2479 # SENTRY-2478 > Replicate owner privileges. > --- > > Key: SENTRY-2468 > URL: https://issues.apache.org/jira/browse/SENTRY-2468 > Project: Sentry > Issue Type: Sub-task > Components: Sentry >Affects Versions: 2.2.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > > In general exporting owner privileges is an unnecessary overhead for couple > of reasons > # Owner privileges are updated automatically in the target cluster when Hive > replication is performed on the target cluster. > # Number of owner privileges could be huge and the magnitude of that would > increase with increase in the number of databases and tables. > > There are still some use cases where there is a need to replicate owner > privilege as well. Here are some of them > # When sentry is not configured in the target cluster when owner privileges > feature is enabled in source cluster. > # When owner privileges feature is enabled in source cluster and not in > target cluster. > In both the above case, target cluster will not have owner privileges for the > hive objects that are replicated. > > In summary, replicating owner privilege should be configurable.It is better > to have it as an input from the administrator performing the export/import > operations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SENTRY-2479) Avoid exporting/importing privileges that are were asked to be excluded
kalyan kumar kalvagadda created SENTRY-2479: --- Summary: Avoid exporting/importing privileges that are were asked to be excluded Key: SENTRY-2479 URL: https://issues.apache.org/jira/browse/SENTRY-2479 Project: Sentry Issue Type: Improvement Components: Sentry Affects Versions: 2.2.0 Reporter: kalyan kumar kalvagadda When sentry is exporting, privileges with excluded actions should not be exported. When sentry is importing, privileges with excluded actions should not be should be not cleaned up before importing the permissions in the target cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SENTRY-2448) Ability read/write import/export file(s) from HDFS.
[ https://issues.apache.org/jira/browse/SENTRY-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2448: Affects Version/s: (was: 2.1.0) 2.2.0 > Ability read/write import/export file(s) from HDFS. > --- > > Key: SENTRY-2448 > URL: https://issues.apache.org/jira/browse/SENTRY-2448 > Project: Sentry > Issue Type: Sub-task > Components: Sentry >Affects Versions: 2.2.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Fix For: 2.2.0 > > > Currently sentry doesn't have capability to export the backup file into HDFS. > This has to be implemented. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SENTRY-2448) Ability read/write import/export file(s) from HDFS.
[ https://issues.apache.org/jira/browse/SENTRY-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2448: Component/s: Sentry > Ability read/write import/export file(s) from HDFS. > --- > > Key: SENTRY-2448 > URL: https://issues.apache.org/jira/browse/SENTRY-2448 > Project: Sentry > Issue Type: Sub-task > Components: Sentry >Affects Versions: 2.2.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Fix For: 2.2.0 > > > Currently sentry doesn't have capability to export the backup file into HDFS. > This has to be implemented. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SENTRY-2448) Ability read/write import/export file(s) from HDFS.
[ https://issues.apache.org/jira/browse/SENTRY-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2448: Fix Version/s: 2.2.0 > Ability read/write import/export file(s) from HDFS. > --- > > Key: SENTRY-2448 > URL: https://issues.apache.org/jira/browse/SENTRY-2448 > Project: Sentry > Issue Type: Sub-task > Components: Sentry >Affects Versions: 2.2.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Fix For: 2.2.0 > > > Currently sentry doesn't have capability to export the backup file into HDFS. > This has to be implemented. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (SENTRY-2448) Ability read/write import/export file(s) from HDFS.
[ https://issues.apache.org/jira/browse/SENTRY-2448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda resolved SENTRY-2448. - Resolution: Fixed Resolving the subtask as the work items in it are resolved. > Ability read/write import/export file(s) from HDFS. > --- > > Key: SENTRY-2448 > URL: https://issues.apache.org/jira/browse/SENTRY-2448 > Project: Sentry > Issue Type: Sub-task >Affects Versions: 2.1.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > > Currently sentry doesn't have capability to export the backup file into HDFS. > This has to be implemented. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2453) Sentry client should perform translation before sending export/import requests.
[ https://issues.apache.org/jira/browse/SENTRY-2453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16712977#comment-16712977 ] kalyan kumar kalvagadda commented on SENTRY-2453: - This will be addressed with the code changes performed by SENTRY-2452 > Sentry client should perform translation before sending export/import > requests. > --- > > Key: SENTRY-2453 > URL: https://issues.apache.org/jira/browse/SENTRY-2453 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 2.2.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > > Sentry client should parse and translate the provides databases, tables and > URI into authorizable before sending the export/import request to sentry > server. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (SENTRY-2451) Extend the export-import tool to provide list of authorizables
[ https://issues.apache.org/jira/browse/SENTRY-2451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda reassigned SENTRY-2451: --- Assignee: Sergio Peña (was: kalyan kumar kalvagadda) > Extend the export-import tool to provide list of authorizables > -- > > Key: SENTRY-2451 > URL: https://issues.apache.org/jira/browse/SENTRY-2451 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 2.2.0 >Reporter: kalyan kumar kalvagadda >Assignee: Sergio Peña >Priority: Major > > CLI client should be extended to accept list of authorizable (databases and > tables) for which the permission information has to be exported or imported. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (SENTRY-2478) Extend the export-import tool to exclude permissions with specific actions
kalyan kumar kalvagadda created SENTRY-2478: --- Summary: Extend the export-import tool to exclude permissions with specific actions Key: SENTRY-2478 URL: https://issues.apache.org/jira/browse/SENTRY-2478 Project: Sentry Issue Type: Improvement Components: Sentry Affects Versions: 2.2.0 Reporter: kalyan kumar kalvagadda Assignee: Sergio Peña CLI client should be extended to be able to list of actions for which the permission information should not be exported/imported. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SENTRY-2454) Add new sentry store api to gather the privileges for a list of authorizables.
[ https://issues.apache.org/jira/browse/SENTRY-2454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2454: Attachment: SENTRY-2454.002.patch > Add new sentry store api to gather the privileges for a list of authorizables. > -- > > Key: SENTRY-2454 > URL: https://issues.apache.org/jira/browse/SENTRY-2454 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 2.2.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Attachments: SENTRY-2454.001.patch, SENTRY-2454.002.patch > > > New sentry API should be implemented to fetch the privileges granted to > authorizables and it's children. authorizables include database, tables, > columns and URI's. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SENTRY-2460) Export sentry permission information to HDFS location
[ https://issues.apache.org/jira/browse/SENTRY-2460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2460: Resolution: Fixed Status: Resolved (was: Patch Available) [~spena] Thanks for the review. > Export sentry permission information to HDFS location > - > > Key: SENTRY-2460 > URL: https://issues.apache.org/jira/browse/SENTRY-2460 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 2.2.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Fix For: 2.2.0 > > Attachments: SENTRY-2460.001.patch, SENTRY-2460.002.patch, > SENTRY-2460.003.patch > > > Sentry should be able to export permission information to a HDFS location. > This can be done using hadoop-common library. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2470) Export privileges granted to the users
[
https://issues.apache.org/jira/browse/SENTRY-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16712963#comment-16712963
]
Hadoop QA commented on SENTRY-2470:
---
Here are the results of testing the latest attachment
https://issues.apache.org/jira/secure/attachment/12950998/SENTRY-2470.002.patch
against master.
{color:red}Overall:{color} -1 due to 2 errors
{color:red}ERROR:{color} mvn test exited 1
{color:red}ERROR:{color} Failed:
org.apache.sentry.api.service.thrift.TestSentryServerLogLevel
Console output:
https://builds.apache.org/job/PreCommit-SENTRY-Build/4278/console
This message is automatically generated.
> Export privileges granted to the users
> --
>
> Key: SENTRY-2470
> URL: https://issues.apache.org/jira/browse/SENTRY-2470
> Project: Sentry
> Issue Type: Improvement
> Components: Sentry
>Affects Versions: 2.2.0
>Reporter: kalyan kumar kalvagadda
>Assignee: kalyan kumar kalvagadda
>Priority: Major
> Attachments: SENTRY-2470.001.patch, SENTRY-2470.002.patch
>
>
> Export mechanism should be able to export the privileges granted to the users
> as well.
> This required changed to sentry store, thrift processor, thirft definition
> and the export file formatter as well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (SENTRY-2454) Add new sentry store api to gather the privileges for a list of authorizables.
[ https://issues.apache.org/jira/browse/SENTRY-2454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16712961#comment-16712961 ] kalyan kumar kalvagadda edited comment on SENTRY-2454 at 12/7/18 3:15 PM: -- Looks to be a un-related failure. As the sentry code doesn't use the new API added. Will re-submit the patch. was (Author: kkalyan): Looks to be a un-related failure. As the sentry code doesn't use the new API added. > Add new sentry store api to gather the privileges for a list of authorizables. > -- > > Key: SENTRY-2454 > URL: https://issues.apache.org/jira/browse/SENTRY-2454 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 2.2.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Attachments: SENTRY-2454.001.patch, SENTRY-2454.002.patch > > > New sentry API should be implemented to fetch the privileges granted to > authorizables and it's children. authorizables include database, tables, > columns and URI's. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (SENTRY-2454) Add new sentry store api to gather the privileges for a list of authorizables.
[ https://issues.apache.org/jira/browse/SENTRY-2454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16712961#comment-16712961 ] kalyan kumar kalvagadda commented on SENTRY-2454: - Looks to be a un-related failure. As the sentry code doesn't use the new API added. > Add new sentry store api to gather the privileges for a list of authorizables. > -- > > Key: SENTRY-2454 > URL: https://issues.apache.org/jira/browse/SENTRY-2454 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 2.2.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Attachments: SENTRY-2454.001.patch, SENTRY-2454.002.patch > > > New sentry API should be implemented to fetch the privileges granted to > authorizables and it's children. authorizables include database, tables, > columns and URI's. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SENTRY-2454) Add new sentry store api to gather the privileges for a list of authorizables.
[ https://issues.apache.org/jira/browse/SENTRY-2454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2454: Attachment: (was: SENTRY-2454.002.patch) > Add new sentry store api to gather the privileges for a list of authorizables. > -- > > Key: SENTRY-2454 > URL: https://issues.apache.org/jira/browse/SENTRY-2454 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 2.2.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Attachments: SENTRY-2454.001.patch, SENTRY-2454.002.patch > > > New sentry API should be implemented to fetch the privileges granted to > authorizables and it's children. authorizables include database, tables, > columns and URI's. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SENTRY-2452) Change the thrift interface to send the list of authorizable to sentry server
[ https://issues.apache.org/jira/browse/SENTRY-2452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2452: Attachment: SENTRY-2452.003.patch > Change the thrift interface to send the list of authorizable to sentry server > - > > Key: SENTRY-2452 > URL: https://issues.apache.org/jira/browse/SENTRY-2452 > Project: Sentry > Issue Type: Improvement >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Attachments: SENTRY-2452.001.patch, SENTRY-2452.003.patch > > > TSentryExportMappingDataRequest and TSentryImportMappingDataRequest which are > used to send export and import requests to sentry server should be changed to > be able to send a list authorizables for which permission information has to > be exported/imported. > These requests should accommodate source and target cluster information as > well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SENTRY-2452) Change the thrift interface to send the list of authorizable to sentry server
[ https://issues.apache.org/jira/browse/SENTRY-2452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2452: Attachment: (was: SENTRY-2452.003.patch) > Change the thrift interface to send the list of authorizable to sentry server > - > > Key: SENTRY-2452 > URL: https://issues.apache.org/jira/browse/SENTRY-2452 > Project: Sentry > Issue Type: Improvement >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Attachments: SENTRY-2452.001.patch, SENTRY-2452.003.patch > > > TSentryExportMappingDataRequest and TSentryImportMappingDataRequest which are > used to send export and import requests to sentry server should be changed to > be able to send a list authorizables for which permission information has to > be exported/imported. > These requests should accommodate source and target cluster information as > well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SENTRY-2470) Export privileges granted to the users
[ https://issues.apache.org/jira/browse/SENTRY-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2470: Attachment: SENTRY-2470.002.patch > Export privileges granted to the users > -- > > Key: SENTRY-2470 > URL: https://issues.apache.org/jira/browse/SENTRY-2470 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 2.2.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Attachments: SENTRY-2470.001.patch, SENTRY-2470.002.patch > > > Export mechanism should be able to export the privileges granted to the users > as well. > This required changed to sentry store, thrift processor, thirft definition > and the export file formatter as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (SENTRY-2470) Export privileges granted to the users
[ https://issues.apache.org/jira/browse/SENTRY-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kalyan kumar kalvagadda updated SENTRY-2470: Attachment: (was: SENTRY-2470.002.patch) > Export privileges granted to the users > -- > > Key: SENTRY-2470 > URL: https://issues.apache.org/jira/browse/SENTRY-2470 > Project: Sentry > Issue Type: Improvement > Components: Sentry >Affects Versions: 2.2.0 >Reporter: kalyan kumar kalvagadda >Assignee: kalyan kumar kalvagadda >Priority: Major > Attachments: SENTRY-2470.001.patch, SENTRY-2470.002.patch > > > Export mechanism should be able to export the privileges granted to the users > as well. > This required changed to sentry store, thrift processor, thirft definition > and the export file formatter as well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
