Review Request 68741: De-normalize schema for storing tags and related objects
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68741/ --- Review request for ranger, Madhan Neethiraj, Ramesh Mani, and Velmurugan Periasamy. Bugs: RANGER-2219 https://issues.apache.org/jira/browse/RANGER-2219 Repository: ranger Description --- Currently, tag-definitions, tags and service-resources are stored in database using a normalized form. When constructing resource->tag mappings, this schema design may lead to a large number of database accesses, thereby causing a major performance bottleneck when the number of resource->tag associations is large. Denormalized schema will reduce the number of database accesses, and improve overall performance significantly. Diffs - agents-common/src/main/java/org/apache/ranger/plugin/model/RangerServiceResource.java 67230c6de agents-common/src/main/java/org/apache/ranger/plugin/model/RangerTag.java 9e620c32c agents-common/src/main/java/org/apache/ranger/plugin/model/RangerTagDef.java c787beca5 security-admin/db/mysql/optimized/current/ranger_core_db_mysql.sql 3f23b0082 security-admin/db/mysql/patches/040-denormalize-tag-tables.sql PRE-CREATION security-admin/db/oracle/optimized/current/ranger_core_db_oracle.sql bafdb9676 security-admin/db/oracle/patches/040-denormalize-tag-tables.sql PRE-CREATION security-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql 2bc58acfe security-admin/db/postgres/patches/040-denormalize-tag-tables.sql PRE-CREATION security-admin/db/sqlanywhere/optimized/current/ranger_core_db_sqlanywhere.sql 1b64eeac8 security-admin/db/sqlanywhere/patches/040-denormalize-tag-tables.sql PRE-CREATION security-admin/db/sqlserver/optimized/current/ranger_core_db_sqlserver.sql 4a216fed9 security-admin/db/sqlserver/patches/040-denormalize-tag-tables.sql PRE-CREATION security-admin/scripts/db_setup.py 02701c726 security-admin/src/main/java/org/apache/ranger/biz/RangerTagDBRetriever.java 7875bc2a8 security-admin/src/main/java/org/apache/ranger/biz/TagDBStore.java d29df930c security-admin/src/main/java/org/apache/ranger/db/XXServiceResourceDao.java ee0e40043 security-admin/src/main/java/org/apache/ranger/db/XXServiceResourceElementDao.java c9a1c2132 security-admin/src/main/java/org/apache/ranger/db/XXServiceResourceElementValueDao.java 364af6d0f security-admin/src/main/java/org/apache/ranger/db/XXTagAttributeDao.java 40c3a887b security-admin/src/main/java/org/apache/ranger/db/XXTagAttributeDefDao.java 129f3c13c security-admin/src/main/java/org/apache/ranger/db/XXTagDao.java e14f836e7 security-admin/src/main/java/org/apache/ranger/db/XXTagDefDao.java c0dd88371 security-admin/src/main/java/org/apache/ranger/entity/XXServiceResource.java 961627a3c security-admin/src/main/java/org/apache/ranger/entity/XXTag.java d26a0b079 security-admin/src/main/java/org/apache/ranger/entity/XXTagDef.java 818908ba8 security-admin/src/main/java/org/apache/ranger/patch/PatchForUpdatingTagsJson_J10020.java PRE-CREATION security-admin/src/main/java/org/apache/ranger/rest/ServiceTagsProcessor.java 9c19bb032 security-admin/src/main/java/org/apache/ranger/service/RangerServiceResourceService.java d7256802a security-admin/src/main/java/org/apache/ranger/service/RangerServiceResourceServiceBase.java 6af682a81 security-admin/src/main/java/org/apache/ranger/service/RangerTagDefService.java 82eb252e6 security-admin/src/main/java/org/apache/ranger/service/RangerTagService.java 28b9115fa security-admin/src/main/resources/META-INF/jpa_named_queries.xml cdf6ba655 security-admin/src/test/java/org/apache/ranger/rest/TestServiceTagProcessor.java 952463318 security-admin/src/test/java/org/apache/ranger/service/TestRangerTagDefService.java 8f19ffd78 security-admin/src/test/java/org/apache/ranger/service/TestRangerTagService.java 1175989e2 Diff: https://reviews.apache.org/r/68741/diff/1/ Testing --- Tested using a local VM. Verified that the x_tag_def, x_tag, x_service_resource tables are updated correctly (data porting) when ranger-admin starts. Verified that plugins receive correct tags by checking /etc/ranger//policycache/__tag.json file in policycache directory. Verified that tags sent by ranger-tagsync are processed correctly by ranger-admin. Thanks, Abhay Kulkarni
[jira] [Commented] (RANGER-2209) Service Definition for ABFS to support Ranger Authorization
[ https://issues.apache.org/jira/browse/RANGER-2209?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16618108#comment-16618108 ] Yuan Gao commented on RANGER-2209: -- [~rmani] The fine-grained operations will fall into READ / WRITE categories as of now. Here is the driver change on how to categorize them: [https://reviews.apache.org/r/68708.] Please take a look and let me know if you have more questions. > Service Definition for ABFS to support Ranger Authorization > --- > > Key: RANGER-2209 > URL: https://issues.apache.org/jira/browse/RANGER-2209 > Project: Ranger > Issue Type: New Feature > Components: Ranger >Reporter: Yuan Gao >Priority: Major > > This is to add Service Definition for Azure Blob File System (ABFS) in Ranger > Authorization. This will give an option to add ABFS as a Service in the > Ranger. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (RANGER-2186) Increment service-specific policy and tag versions after update transaction is committed
[ https://issues.apache.org/jira/browse/RANGER-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Velmurugan Periasamy updated RANGER-2186: - Fix Version/s: 2.0.0 > Increment service-specific policy and tag versions after update transaction > is committed > > > Key: RANGER-2186 > URL: https://issues.apache.org/jira/browse/RANGER-2186 > Project: Ranger > Issue Type: Bug > Components: Ranger >Affects Versions: master >Reporter: Abhay Kulkarni >Assignee: Abhay Kulkarni >Priority: Major > Fix For: master, 2.0.0 > > > Policy updates to different policies within a service, when successful, > update the service's policy version. If the update transactions are > concurrent, and executed on different ranger-admin servers (in HA > configuration), then it is possible that policy-version of the transaction > that commits later overwrites policy-version of earlier transaction, > effectively losing track of the first change. > If policy-version is updated after update to policy is committed, then the > window of such loss is greatly reduced. > Similar considerations apply for tag updates. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (RANGER-2186) Increment service-specific policy and tag versions after update transaction is committed
[ https://issues.apache.org/jira/browse/RANGER-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581927#comment-16581927 ] Abhay Kulkarni edited comment on RANGER-2186 at 9/17/18 6:44 PM: - Commit details: master: [https://git-wip-us.apache.org/repos/asf?p=ranger.git;a=commit;h=3aa94596be73371a64f190ef6e01ab93d6ef8e78] Additional fix: Review: https://reviews.apache.org/r/68738/ master: [https://git-wip-us.apache.org/repos/asf?p=ranger.git;a=commit;h=9f639f3c5876c7134f43df33467691a2b01735a7] was (Author: abhayk): Commit details: master: [https://git-wip-us.apache.org/repos/asf?p=ranger.git;a=commit;h=3aa94596be73371a64f190ef6e01ab93d6ef8e78] master: https://git-wip-us.apache.org/repos/asf?p=ranger.git;a=commit;h=9f639f3c5876c7134f43df33467691a2b01735a7 > Increment service-specific policy and tag versions after update transaction > is committed > > > Key: RANGER-2186 > URL: https://issues.apache.org/jira/browse/RANGER-2186 > Project: Ranger > Issue Type: Bug > Components: Ranger >Affects Versions: master >Reporter: Abhay Kulkarni >Assignee: Abhay Kulkarni >Priority: Major > Fix For: master > > > Policy updates to different policies within a service, when successful, > update the service's policy version. If the update transactions are > concurrent, and executed on different ranger-admin servers (in HA > configuration), then it is possible that policy-version of the transaction > that commits later overwrites policy-version of earlier transaction, > effectively losing track of the first change. > If policy-version is updated after update to policy is committed, then the > window of such loss is greatly reduced. > Similar considerations apply for tag updates. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (RANGER-2186) Increment service-specific policy and tag versions after update transaction is committed
[ https://issues.apache.org/jira/browse/RANGER-2186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581927#comment-16581927 ] Abhay Kulkarni edited comment on RANGER-2186 at 9/17/18 6:43 PM: - Commit details: master: [https://git-wip-us.apache.org/repos/asf?p=ranger.git;a=commit;h=3aa94596be73371a64f190ef6e01ab93d6ef8e78] master: https://git-wip-us.apache.org/repos/asf?p=ranger.git;a=commit;h=9f639f3c5876c7134f43df33467691a2b01735a7 was (Author: abhayk): Commit details: master: https://git-wip-us.apache.org/repos/asf?p=ranger.git;a=commit;h=3aa94596be73371a64f190ef6e01ab93d6ef8e78 > Increment service-specific policy and tag versions after update transaction > is committed > > > Key: RANGER-2186 > URL: https://issues.apache.org/jira/browse/RANGER-2186 > Project: Ranger > Issue Type: Bug > Components: Ranger >Affects Versions: master >Reporter: Abhay Kulkarni >Assignee: Abhay Kulkarni >Priority: Major > Fix For: master > > > Policy updates to different policies within a service, when successful, > update the service's policy version. If the update transactions are > concurrent, and executed on different ranger-admin servers (in HA > configuration), then it is possible that policy-version of the transaction > that commits later overwrites policy-version of earlier transaction, > effectively losing track of the first change. > If policy-version is updated after update to policy is committed, then the > window of such loss is greatly reduced. > Similar considerations apply for tag updates. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Review Request 68738: RANGER-2186: Part II - check for NPE when incrementing service-specific policy and tag versions after update transaction is committed
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68738/#review208686 --- Ship it! Ship It! - Velmurugan Periasamy On Sept. 17, 2018, 6:28 p.m., Abhay Kulkarni wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/68738/ > --- > > (Updated Sept. 17, 2018, 6:28 p.m.) > > > Review request for ranger, Ramesh Mani and Velmurugan Periasamy. > > > Bugs: RANGER-2186 > https://issues.apache.org/jira/browse/RANGER-2186 > > > Repository: ranger > > > Description > --- > > When a service is deleted, there is a NPE thrown while creating/updating > policy-version value in x_service_version_info table in the row corresponding > to the deleted service. There needs to be a check if to see if the service > exists before this operation. > > > Diffs > - > > security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java > 32af4cd2d > > > Diff: https://reviews.apache.org/r/68738/diff/1/ > > > Testing > --- > > Created and deleted a service. Ensured that the service is deleted and there > is no Null-Pointer Exception thrown in xa_portal.log. > > > Thanks, > > Abhay Kulkarni > >
Re: Review Request 68738: RANGER-2186: Part II - check for NPE when incrementing service-specific policy and tag versions after update transaction is committed
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68738/#review208685 --- Ship it! Ship It! - Ramesh Mani On Sept. 17, 2018, 6:28 p.m., Abhay Kulkarni wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/68738/ > --- > > (Updated Sept. 17, 2018, 6:28 p.m.) > > > Review request for ranger, Ramesh Mani and Velmurugan Periasamy. > > > Bugs: RANGER-2186 > https://issues.apache.org/jira/browse/RANGER-2186 > > > Repository: ranger > > > Description > --- > > When a service is deleted, there is a NPE thrown while creating/updating > policy-version value in x_service_version_info table in the row corresponding > to the deleted service. There needs to be a check if to see if the service > exists before this operation. > > > Diffs > - > > security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java > 32af4cd2d > > > Diff: https://reviews.apache.org/r/68738/diff/1/ > > > Testing > --- > > Created and deleted a service. Ensured that the service is deleted and there > is no Null-Pointer Exception thrown in xa_portal.log. > > > Thanks, > > Abhay Kulkarni > >
Review Request 68738: RANGER-2186: Part II - check for NPE when incrementing service-specific policy and tag versions after update transaction is committed
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68738/ --- Review request for ranger, Ramesh Mani and Velmurugan Periasamy. Bugs: RANGER-2186 https://issues.apache.org/jira/browse/RANGER-2186 Repository: ranger Description --- When a service is deleted, there is a NPE thrown while creating/updating policy-version value in x_service_version_info table in the row corresponding to the deleted service. There needs to be a check if to see if the service exists before this operation. Diffs - security-admin/src/main/java/org/apache/ranger/biz/ServiceDBStore.java 32af4cd2d Diff: https://reviews.apache.org/r/68738/diff/1/ Testing --- Created and deleted a service. Ensured that the service is deleted and there is no Null-Pointer Exception thrown in xa_portal.log. Thanks, Abhay Kulkarni
[jira] [Updated] (RANGER-707) Redesign of Apache Ranger Website
[ https://issues.apache.org/jira/browse/RANGER-707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Selvamohan Neethiraj updated RANGER-707: Affects Version/s: (was: 0.5.0) 1.1.0 > Redesign of Apache Ranger Website > - > > Key: RANGER-707 > URL: https://issues.apache.org/jira/browse/RANGER-707 > Project: Ranger > Issue Type: Improvement > Components: documentation, Ranger >Affects Versions: 1.1.0 >Reporter: Selvamohan Neethiraj >Assignee: Gautam Borad >Priority: Critical > Attachments: Logo01.jpg, Logo02.jpg, Logo03.jpg, Logo04.jpg, > Logo06.jpg > > > The Apache Ranger website (http://ranger.incubator.apache.org/) needs to > re-organized to have all of the information available to developers and > end-users in an easier manner. > Example Site: > http://cordova.apache.org/ > http://hawq.incubator.apache.org/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (RANGER-2226) Define explicit (test) dependency on json-smart in the Knox agent
[ https://issues.apache.org/jira/browse/RANGER-2226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colm O hEigeartaigh updated RANGER-2226: Fix Version/s: (was: 1.1.0) > Define explicit (test) dependency on json-smart in the Knox agent > - > > Key: RANGER-2226 > URL: https://issues.apache.org/jira/browse/RANGER-2226 > Project: Ranger > Issue Type: Task > Components: plugins >Reporter: Colm O hEigeartaigh >Assignee: Colm O hEigeartaigh >Priority: Major > Fix For: 2.0.0 > > Attachments: > 0001-RANGER-2226-Define-explicit-test-dependency-on-json-.patch > > > The build is currently failing for me with: > Failed to read artifact descriptor for > net.minidev:json-smart:jar:2.3-SNAPSHOT: Could not transfer artifact > net.minidev:json-smart:pom:2.3-SNAPSHOT from/to dynamodb-local-oregon > (https://s3-us-west-2.amazonaws.com/dynamodb-local/release): Access denied > to: > https://s3-us-west-2.amazonaws.com/dynamodb-local/release/net/minidev/json-smart/2.3-SNAPSHOT/json-smart-2.3-SNAPSHOT.pom > , ReasonPhrase:Forbidden. -> [Help 1] > The fix is to declare an explicit (test) dependency on json-smart in the Knox > module. > See: https://issues.apache.org/jira/browse/KNOX-1275 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
Re: Review Request 68715: RANGER-2226 - Define explicit (test) dependency on json-smart in the Knox agent
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/68715/ --- (Updated Sept. 17, 2018, 9:56 a.m.) Review request for ranger. Bugs: RANGER-2226 https://issues.apache.org/jira/browse/RANGER-2226 Repository: ranger Description --- The build is currently failing for me with: Failed to read artifact descriptor for net.minidev:json-smart:jar:2.3-SNAPSHOT: Could not transfer artifact net.minidev:json-smart:pom:2.3-SNAPSHOT from/to dynamodb-local-oregon (https://s3-us-west-2.amazonaws.com/dynamodb-local/release): Access denied to: https://s3-us-west-2.amazonaws.com/dynamodb-local/release/net/minidev/json-smart/2.3-SNAPSHOT/json-smart-2.3-SNAPSHOT.pom , ReasonPhrase:Forbidden. -> [Help 1] The fix is to declare an explicit (test) dependency on json-smart in the Knox module. See: https://issues.apache.org/jira/browse/KNOX-1275 Diffs - knox-agent/pom.xml 1d8fdae62 pom.xml ae3f4be4c Diff: https://reviews.apache.org/r/68715/diff/1/ Testing --- Thanks, Colm O hEigeartaigh
