[
https://issues.apache.org/jira/browse/RANGER-3899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ramachandran updated RANGER-3899:
---------------------------------
Description:
*{color:#0000ff}Policy Creation Steps in Apache Ranger:{color}*{color:#0000ff}
{color}
||Heading 1||
|
||{color:#0000ff}1. Get the service by Name:{color}
{color:#800080}RangerService service =
getServiceByName(policy.getService());\{*}→ 1DB Read call{color}{*}
2. Get XXServiceDef by Name
{color:#800080}XXServiceDef xServiceDef =
daoMgr.getXXServiceDef().findByName(service.getType());{color}
{color:#800080}*→ 1DB Read call{color}*
3. Get the existing XXPolicy by Name
{color:#800080}XXPolicy existing =
daoMgr.getXXPolicy().findByNameAndServiceIdAndZoneId(policy.getName(),
service.getId(), zoneId);{color} {color:#800080}*→ 1DB Read call{color}*
4. Create a policy
{color:#800080}policy = policyService.create(policy, true); *→ 1DB Write
call{color}*
5. Get policy by Id
{color:#800000}XXPolicy xCreatedPolicy =
daoMgr.getXXPolicy().getById(policy.getId());{color} {color:#800080}*→ 1DB Read
call{color}*
6. createObjectDataHistory for the newly created policy
{color:#800000}dataHistService.createObjectDataHistory(createdPolicy,
RangerDataHistService.ACTION_CREATE);{color} {color:#800080}*→ 1DB Write
call{color}*
7. createTrxLog for the newly created policy
{color:#993300}bizUtil.createTrxLog(trxLogList);{color} {color:#800080}*→ 1DB
Write call{color}*
8. Create Policy Label for the newly created policy
createOrMapLabels(xCreatedPolicy, uniquePolicyLabels); {color:#800080}*→ 1DB
Write call{color}*
9. Create Policy Mapping For Ref Table
policyRefUpdater.createNewPolMappingForRefTable(policy, xCreatedPolicy,
xServiceDef);
{color:#0000ff}1. Getting
roleNames,groupNames,userNames,accessTypes,dataMaskTypes,conditionTypes from
each policy items of the policy{color}
{color:#0000ff}2. Getting resourceNames from the policy{color}
3.{color:#0000ff}For each resource{color}
{color:#0000ff}Getting the XXResourceDef by resource and policy.getId()
{color}
{color:#993300}XXResourceDef xResDef =
daoMgr.getXXResourceDef().findByNameAndPolicyId(resource,
policy.getId());{color} {color:#800080}*→ 1DB Read call for each
resource{color}*
populate XXPolicyRefResource
{color:#0000ff}4. Create the batch of XXPolicyRefResource in DB{color}
{color:#993300}daoMgr.getXXPolicyRefResource().batchCreate(xPolResources);{color}
{color:#800080}*→ 1DB Batch Write call{color}*
5.For each roleNames
{color:#0000ff} Getting XXRole by roleName{color}
{color:#993300}XXRole xRole =
daoMgr.getXXRole().findByRoleName(role);{color} {color:#800080}*→ 1DB Read call
for each role{color}*
populate XXPolicyRefRole
{color:#0000ff}6. Create the batch of XXPolicyRefRole in DB{color}
{color:#993300}daoMgr.getXXPolicyRefResource().batchCreate(xPolRoles);
{color}{color:#800080}*→ 1DB Batch Write call{color}*
7.For each groupNames
{color:#0000ff}Getting the XXGroup by groupName{color}
{color:#993300}XXGroup xGroup =
daoMgr.getXXGroup().findByGroupName(context.group.getName());{color}{color:#800080}*→
1DB Read call for each group{color}*
populate XXPolicyRefGroup
{color:#0000ff}Insert into DB{color}
{color:#993300}daoMgr.getXXPolicyRefGroup().create(xPolGroup);*→ 1DB write call
for each group{color}*
{color:#0000ff}8.For each userNames {color}
{color:#0000ff}Getting XXUser by userName{color}
{color:#993300}XXUser xUser = daoMgr.getXXUser().findByUserName(user);{color}
{color:#800080}*→ 1DB Read call for each user{color}*
populate XXPolicyRefUser
{color:#993300}daoMgr.getXXPolicyRefUser().create(xPolUser); *→ 1DB write call
for each user{color}*
{color:#0000ff}9.For each accessTypes{color}
{color:#0000ff}Getting the XXAccessTypeDef by accessType and
xPolicy.getService() {color}
{color:#800080}XXAccessTypeDef xAccTypeDef =
daoMgr.getXXAccessTypeDef().findByNameAndServiceId(accessType,
xPolicy.getService()); *→ 1DB Read call for each accesType{color}*
populate XXAccessTypeDef
{color:#0000ff}10. Create the batch of xPolAccesses in DB{color}
{color:#993300}daoMgr.getXXPolicyRefAccessType().batchCreate(xPolAccesses);{color}
{color:#800080}*→ 1DB Batch Write call{color}*
11.For each conditionTypes
{color:#0000ff}Getting the XXPolicyConditionDef by condition and
xServiceDef.getId(){color}
{color:#993300}XXPolicyConditionDef xPolCondDef =
daoMgr.getXXPolicyConditionDef().findByServiceDefIdAndName(xServiceDef.getId(),
condition){color}; {color:#800080}*→ 1DB Read call for each
conditionType{color}*
populate XXPolicyConditionDef
{color:#0000ff}12. Create the batch of xPolConds in DB{color}
{color:#993300}daoMgr.getXXPolicyRefCondition().batchCreate(xPolConds);{color}
{color:#800080}→ 1DB Batch Write call{color}||
|
||*
*Total number of DB calls involved for the below Policy creation in Apache
Ranger:*||
1.Policy contains 500 users,5 access Types (permissions),12 resources
(1DB,1Table,10 columns)
||DB Write count||DB Read count||DB Batch Write count||Time taken to create the
policy||
|504|526|3|{color:#ff0000}4~8 seconds{color}|
2.Policy contains 50 roles,5 access Types (permissions),12 resources
(1DB,1Table,10 columns)
||DB Write count||DB Read count||DB Batch Write count||Time taken to create the
policy||
|4|76|4|{color:#ff0000}~1 second{color}|
{color:#ff0000}Note :{color}
When we reduce the number of users added into policy directly -→ Total number
of DB calls will be reduced
We can use {color:#008000}roles or groups {color} into the policy instead of
adding users directly
{color:#ff0000}Proposal :{color}
{color:#ff0000}We can try *DB batch write call instead of 1 DB write call for
every user{color}*
was:
*{color:#0000ff}Policy Creation Steps in Apache Ranger:{color}*{color:#0000ff}
{color}
||{color:#0000ff}1. Get the service by Name:{color}
{color:#800080}RangerService service =
getServiceByName(policy.getService());{*}→ {color:#ff0000}1DB Read
call{color}{*}{color}
{color:#0000ff}2. Get XXServiceDef by Name{color}
{color:#800080}XXServiceDef xServiceDef =
daoMgr.getXXServiceDef().findByName(service.getType());{color}
{color:#800080}*→ {color:#ff0000}1DB Read call{color}*{color}
{color:#0000ff}3. Get the existing XXPolicy by Name{color}
{color:#800080}XXPolicy existing =
daoMgr.getXXPolicy().findByNameAndServiceIdAndZoneId(policy.getName(),
service.getId(), zoneId);{color} {color:#800080}*→ {color:#ff0000}1DB Read
call{color}*{color}
{color:#0000ff}4. Create a policy{color}
{color:#800080}policy = policyService.create(policy, true); *→
{color:#ff0000}1DB Write call{color}*{color}
{color:#0000ff}5. Get policy by Id{color}
{color:#800000}XXPolicy xCreatedPolicy =
daoMgr.getXXPolicy().getById(policy.getId());{color} {color:#800080}*→
{color:#ff0000}1DB Read call{color}*{color}
{color:#0000ff}6. createObjectDataHistory for the newly created policy{color}
{color:#800000}dataHistService.createObjectDataHistory(createdPolicy,
RangerDataHistService.ACTION_CREATE);{color} {color:#800080}*→
{color:#ff0000}1DB Write call{color}*{color}
{color:#0000ff}7. createTrxLog for the newly created policy{color}
{color:#993300}bizUtil.createTrxLog(trxLogList);{color} {color:#800080}*→
{color:#ff0000}1DB Write call{color}*{color}
{color:#0000ff}8. Create Policy Label for the newly created policy{color}
createOrMapLabels(xCreatedPolicy, uniquePolicyLabels); {color:#800080}*→
{color:#ff0000}1DB Write call{color}*{color}
{color:#0000ff}9. Create Policy Mapping For Ref Table{color}
policyRefUpdater.createNewPolMappingForRefTable(policy, xCreatedPolicy,
xServiceDef);
{color:#0000ff}1. Getting
roleNames,groupNames,userNames,accessTypes,dataMaskTypes,conditionTypes from
each policy items of the policy{color}
{color:#0000ff}2. Getting resourceNames from the policy{color}
3.{color:#0000ff}For each resource{color}
{color:#0000ff}Getting the XXResourceDef by resource and policy.getId()
{color}
{color:#993300}XXResourceDef xResDef =
daoMgr.getXXResourceDef().findByNameAndPolicyId(resource,
policy.getId());{color} {color:#800080}*→ {color:#ff0000}1DB Read call for each
resource{color}*{color}
{color:#0000ff}populate XXPolicyRefResource{color}
{color:#0000ff}4. Create the batch of XXPolicyRefResource in DB{color}
{color:#993300}daoMgr.getXXPolicyRefResource().batchCreate(xPolResources);{color}
{color:#800080}*→ {color:#ff0000}1DB Batch Write call{color}*{color}
{color:#0000ff}5.For each roleNames {color}
{color:#0000ff} Getting XXRole by roleName{color}
{color:#993300}XXRole xRole =
daoMgr.getXXRole().findByRoleName(role);{color} {color:#800080}*→
{color:#ff0000}1DB Read call for each role{color}*{color}
{color:#0000ff} populate XXPolicyRefRole{color}
{color:#0000ff}6. Create the batch of XXPolicyRefRole in DB{color}
{color:#993300}daoMgr.getXXPolicyRefResource().batchCreate(xPolRoles);
{color}{color:#800080}*→ {color:#ff0000}1DB Batch Write call{color}*{color}
{color:#0000ff}7.For each groupNames{color}
{color:#0000ff}Getting the XXGroup by groupName{color}
{color:#993300}XXGroup xGroup =
daoMgr.getXXGroup().findByGroupName(context.group.getName());{color}{color:#800080}*→
{color:#ff0000}1DB Read call for each group{color}*{color}
{color:#0000ff}populate XXPolicyRefGroup{color}
{color:#0000ff}Insert into DB{color}
{color:#993300}daoMgr.getXXPolicyRefGroup().create(xPolGroup);{color:#800080}*→
{color:#ff0000}1DB write call for each group{color}*{color}{color}
{color:#0000ff}8.For each userNames {color}
{color:#0000ff}Getting XXUser by userName{color}
{color:#993300}XXUser xUser = daoMgr.getXXUser().findByUserName(user);{color}
{color:#800080}*→ {color:#ff0000}1DB Read call for each user{color}*{color}
{color:#0000ff}populate XXPolicyRefUser{color}
{color:#993300}daoMgr.getXXPolicyRefUser().create(xPolUser); {color:#800080}*→
{color:#ff0000}1DB write call for each user{color}*{color}{color}
{color:#0000ff}9.For each accessTypes{color}
{color:#0000ff}Getting the XXAccessTypeDef by accessType and
xPolicy.getService() {color}
{color:#800080}XXAccessTypeDef xAccTypeDef =
daoMgr.getXXAccessTypeDef().findByNameAndServiceId(accessType,
xPolicy.getService()); *→ {color:#ff0000}1DB Read call for each
accesType{color}*{color}
{color:#0000ff}populate XXAccessTypeDef{color}
{color:#0000ff}10. Create the batch of xPolAccesses in DB{color}
{color:#993300}daoMgr.getXXPolicyRefAccessType().batchCreate(xPolAccesses);{color}
{color:#800080}*→ {color:#ff0000}1DB Batch Write call{color}*{color}
{color:#0000ff}11.For each conditionTypes{color}
{color:#0000ff}Getting the XXPolicyConditionDef by condition and
xServiceDef.getId(){color}
{color:#993300}XXPolicyConditionDef xPolCondDef =
daoMgr.getXXPolicyConditionDef().findByServiceDefIdAndName(xServiceDef.getId(),
condition){color}; {color:#800080}*→ {color:#ff0000}1DB Read call for each
conditionType{color}*{color}
{color:#0000ff}populate XXPolicyConditionDef{color}
{color:#0000ff}12. Create the batch of xPolConds in DB{color}
{color:#993300}daoMgr.getXXPolicyRefCondition().batchCreate(xPolConds);{color}
{color:#800080}*→ {color:#ff0000}1DB Batch Write call{color}*{color}||
*{color:#0000ff}Total number of DB calls involved for the below Policy creation
in Apache Ranger:{color}*
1.Policy contains 500 users,5 access Types (permissions),12 resources
(1DB,1Table,10 columns)
||DB Write count||DB Read count||DB Batch Write count||Time taken to create the
policy||
|504|526|3|{color:#ff0000}4~8 seconds{color}|
2.Policy contains 50 roles,5 access Types (permissions),12 resources
(1DB,1Table,10 columns)
||DB Write count||DB Read count||DB Batch Write count||Time taken to create the
policy||
|4|76|4|{color:#ff0000}~1 second{color}|
{color:#ff0000}Note :{color}
When we reduce the number of users added into policy directly -→ Total number
of DB calls will be reduced
We can use {color:#008000}roles or groups {color} into the policy instead of
adding users directly
{color:#ff0000}Proposal :{color}
{color:#ff0000}We can try {color:#993300}{color:#800080}*{color:#ff0000}DB
batch write call instead of 1 DB write call for every
user{color}*{color}{color}{color}
> Policy creation takes more when policy contains more users
> -----------------------------------------------------------
>
> Key: RANGER-3899
> URL: https://issues.apache.org/jira/browse/RANGER-3899
> Project: Ranger
> Issue Type: Improvement
> Components: Ranger
> Affects Versions: 3.0.0
> Reporter: Ramachandran
> Priority: Major
>
> *{color:#0000ff}Policy Creation Steps in Apache
> Ranger:{color}*{color:#0000ff} {color}
>
> ||Heading 1||
> |
> ||{color:#0000ff}1. Get the service by Name:{color}
> {color:#800080}RangerService service =
> getServiceByName(policy.getService());\{*}→ 1DB Read call{color}{*}
> 2. Get XXServiceDef by Name
> {color:#800080}XXServiceDef xServiceDef =
> daoMgr.getXXServiceDef().findByName(service.getType());{color}
> {color:#800080}*→ 1DB Read call{color}*
> 3. Get the existing XXPolicy by Name
> {color:#800080}XXPolicy existing =
> daoMgr.getXXPolicy().findByNameAndServiceIdAndZoneId(policy.getName(),
> service.getId(), zoneId);{color} {color:#800080}*→ 1DB Read call{color}*
> 4. Create a policy
> {color:#800080}policy = policyService.create(policy, true); *→ 1DB Write
> call{color}*
> 5. Get policy by Id
> {color:#800000}XXPolicy xCreatedPolicy =
> daoMgr.getXXPolicy().getById(policy.getId());{color} {color:#800080}*→ 1DB
> Read call{color}*
> 6. createObjectDataHistory for the newly created policy
> {color:#800000}dataHistService.createObjectDataHistory(createdPolicy,
> RangerDataHistService.ACTION_CREATE);{color} {color:#800080}*→ 1DB Write
> call{color}*
> 7. createTrxLog for the newly created policy
> {color:#993300}bizUtil.createTrxLog(trxLogList);{color} {color:#800080}*→ 1DB
> Write call{color}*
> 8. Create Policy Label for the newly created policy
> createOrMapLabels(xCreatedPolicy, uniquePolicyLabels); {color:#800080}*→ 1DB
> Write call{color}*
> 9. Create Policy Mapping For Ref Table
> policyRefUpdater.createNewPolMappingForRefTable(policy, xCreatedPolicy,
> xServiceDef);
> {color:#0000ff}1. Getting
> roleNames,groupNames,userNames,accessTypes,dataMaskTypes,conditionTypes from
> each policy items of the policy{color}
> {color:#0000ff}2. Getting resourceNames from the policy{color}
> 3.{color:#0000ff}For each resource{color}
> {color:#0000ff}Getting the XXResourceDef by resource and policy.getId()
> {color}
> {color:#993300}XXResourceDef xResDef =
> daoMgr.getXXResourceDef().findByNameAndPolicyId(resource,
> policy.getId());{color} {color:#800080}*→ 1DB Read call for each
> resource{color}*
> populate XXPolicyRefResource
> {color:#0000ff}4. Create the batch of XXPolicyRefResource in DB{color}
>
> {color:#993300}daoMgr.getXXPolicyRefResource().batchCreate(xPolResources);{color}
> {color:#800080}*→ 1DB Batch Write call{color}*
> 5.For each roleNames
> {color:#0000ff} Getting XXRole by roleName{color}
> {color:#993300}XXRole xRole =
> daoMgr.getXXRole().findByRoleName(role);{color} {color:#800080}*→ 1DB Read
> call for each role{color}*
> populate XXPolicyRefRole
> {color:#0000ff}6. Create the batch of XXPolicyRefRole in DB{color}
> {color:#993300}daoMgr.getXXPolicyRefResource().batchCreate(xPolRoles);
> {color}{color:#800080}*→ 1DB Batch Write call{color}*
> 7.For each groupNames
> {color:#0000ff}Getting the XXGroup by groupName{color}
> {color:#993300}XXGroup xGroup =
> daoMgr.getXXGroup().findByGroupName(context.group.getName());{color}{color:#800080}*→
> 1DB Read call for each group{color}*
> populate XXPolicyRefGroup
> {color:#0000ff}Insert into DB{color}
> {color:#993300}daoMgr.getXXPolicyRefGroup().create(xPolGroup);*→ 1DB write
> call for each group{color}*
> {color:#0000ff}8.For each userNames {color}
> {color:#0000ff}Getting XXUser by userName{color}
> {color:#993300}XXUser xUser = daoMgr.getXXUser().findByUserName(user);{color}
> {color:#800080}*→ 1DB Read call for each user{color}*
> populate XXPolicyRefUser
> {color:#993300}daoMgr.getXXPolicyRefUser().create(xPolUser); *→ 1DB write
> call for each user{color}*
> {color:#0000ff}9.For each accessTypes{color}
> {color:#0000ff}Getting the XXAccessTypeDef by accessType and
> xPolicy.getService() {color}
> {color:#800080}XXAccessTypeDef xAccTypeDef =
> daoMgr.getXXAccessTypeDef().findByNameAndServiceId(accessType,
> xPolicy.getService()); *→ 1DB Read call for each accesType{color}*
> populate XXAccessTypeDef
> {color:#0000ff}10. Create the batch of xPolAccesses in DB{color}
> {color:#993300}daoMgr.getXXPolicyRefAccessType().batchCreate(xPolAccesses);{color}
> {color:#800080}*→ 1DB Batch Write call{color}*
> 11.For each conditionTypes
> {color:#0000ff}Getting the XXPolicyConditionDef by condition and
> xServiceDef.getId(){color}
> {color:#993300}XXPolicyConditionDef xPolCondDef =
> daoMgr.getXXPolicyConditionDef().findByServiceDefIdAndName(xServiceDef.getId(),
> condition){color}; {color:#800080}*→ 1DB Read call for each
> conditionType{color}*
> populate XXPolicyConditionDef
> {color:#0000ff}12. Create the batch of xPolConds in DB{color}
> {color:#993300}daoMgr.getXXPolicyRefCondition().batchCreate(xPolConds);{color}
> {color:#800080}→ 1DB Batch Write call{color}||
> |
> ||*
> *Total number of DB calls involved for the below Policy creation in Apache
> Ranger:*||
> 1.Policy contains 500 users,5 access Types (permissions),12 resources
> (1DB,1Table,10 columns)
>
> ||DB Write count||DB Read count||DB Batch Write count||Time taken to create
> the policy||
> |504|526|3|{color:#ff0000}4~8 seconds{color}|
>
> 2.Policy contains 50 roles,5 access Types (permissions),12 resources
> (1DB,1Table,10 columns)
>
> ||DB Write count||DB Read count||DB Batch Write count||Time taken to create
> the policy||
> |4|76|4|{color:#ff0000}~1 second{color}|
>
> {color:#ff0000}Note :{color}
> When we reduce the number of users added into policy directly -→ Total number
> of DB calls will be reduced
> We can use {color:#008000}roles or groups {color} into the policy instead of
> adding users directly
> {color:#ff0000}Proposal :{color}
> {color:#ff0000}We can try *DB batch write call instead of 1 DB write call for
> every user{color}*
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
