ramackri opened a new pull request, #962:
URL: https://github.com/apache/ranger/pull/962
## What changes were proposed in this pull request?
Optimized `PolicyRefUpdater` and related DAOs so policy create/update
performs fewer database round-trips when a policy references many users,
groups, and roles.
### Problem
When creating or updating a policy with a large number of principals (users,
groups, roles), Ranger Admin was slow because:
1. **N+1 lookups** — Each user, group, role, resource, access type,
condition, and data mask type was resolved with a separate DB query
(`findByUserName`, `findByGroupName`, `findByRoleName`, etc.).
2. **Full ref-table rebuild on update** — On policy update,
`cleanupRefTables()` deleted all rows from `x_policy_ref_user`,
`x_policy_ref_group`, and `x_policy_ref_role` (and other ref tables), then
re-inserted every principal even when nothing changed.
### Changes
1. **Batch lookups via named queries** — Added `IN (:names)` JPA named
queries and DAO helpers to resolve principals and definitions in bulk:
- `XXUser.getIdsByUserNames`
- `XXGroup.getIdsByGroupNames`
- `XXRole.getIdsByRoleNames`
- `XXResourceDef.findByNamesAndPolicyId`
- `XXAccessTypeDef.findByNamesAndServiceId`
- `XXPolicyConditionDef.findByServiceDefIdAndNames`
- `XXDataMaskTypeDef.findByNamesAndServiceId`
2. **Pre-resolved principal IDs** — `PolicyPrincipalAssociator` now accepts
a pre-fetched principal ID from the bulk map, avoiding per-name lookups when
the principal already exists.
3. **Selective cleanup on policy update** — Introduced
`cleanupRefTablesForUpdate()` that:
- Loads existing user/role/group ref rows for the policy
- Deletes only principals removed from the policy
- Skips re-insert for principals that are unchanged
- Still performs full delete/rebuild for resources, access types,
conditions, and data mask types (typically far fewer entries)
4. **Unified create/update flow** — `createNewPolMappingForRefTable()` takes
a new `isCleanupRefTablesNeeded` flag:
- **Create:** `false` (no prior ref rows)
- **Update:** `true` (runs selective cleanup before insert)
- `ServiceDBStore` no longer calls `cleanupRefTables()` separately before
mapping
5. **Batch insert helper** — Added `batchInsert()` with bulk-mode handling
and debug timing logs around `dao.batchCreate()`.
### Files changed (15)
| Area | Files |
|------|-------|
| Core logic | `PolicyRefUpdater.java`, `ServiceDBStore.java` |
| DAOs | `XXUserDao`, `XXGroupDao`, `XXRoleDao`, `XXResourceDefDao`,
`XXAccessTypeDefDao`, `XXPolicyConditionDefDao`, `XXDataMaskTypeDefDao`,
`XXPolicyRefUserDao`, `XXPolicyRefGroupDao`, `XXPolicyRefRoleDao` |
| Queries | `jpa_named_queries.xml` |
| Tests | `TestPolicyRefUpdater.java`, `TestServiceDBStore.java` |
---
## How was this patch tested?
### Unit tests
- [x] Updated `TestPolicyRefUpdater` for the new bulk-lookup APIs and
`isCleanupRefTablesNeeded` parameter
- [x] Added `testCleanupRefTablesForUpdate_SelectivePrincipalCleanup` to
verify selective delete/insert for users, roles, and groups on update
- [x] Updated `TestServiceDBStore` for the new
`createNewPolMappingForRefTable` signature
```bash
mvn -pl security-admin test -Dtest=TestPolicyRefUpdater,TestServiceDBStore
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
