Siyao Meng created HDDS-16854:
---------------------------------
Summary: Ranger background sync never retries a failed role delete
of a removed tenant, so the tenant can never be created again
Key: HDDS-16854
URL: https://issues.apache.org/jira/browse/HDDS-16854
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: CR-3-retry-failed-ranger-role-deletes-in-sync.patch,
TestBugCR3OrphanRoleCleanup.java
h3. Mechanism
{{OMRangerBGSyncService}} removes the Ranger state of a tenant that is not in
the OM DB by deleting its labeled policies in {{processAllPoliciesFromOMDB}}
and then its roles in {{processAllRolesFromOMDB}}. A failed role delete is only
logged ("Failed to delete role", without the exception). The sync learns about
roles only from the role ACLs of the policies labeled {{OzoneTenant}}
({{loadAllPoliciesAndRoleNamesFromRanger}}), and those policies are already
gone. So no later run sees the role again, and the run records the Ranger
service version as in sync. {{createRole}} fails when a role with the same name
exists (HDDS-15982 kept creation fail fast), so every later create of that
tenant fails.
HDDS-16188 describes the same end state reached through a partially failed
tenant create. This report is about the cleanup path of the sync, which needs
its own change.
h3. Trigger
# A tenant create leaves Ranger roles and policies without a tenant in OM. One
way: {{OMTenantCreateRequest.preExecute}} creates them, then the apply fails,
for example with {{VOLUME_ALREADY_EXISTS}} because the volume was created
concurrently. {{preExecute}} expects the sync to clean this up.
# The sync deletes both policies, then the delete of {{tenant1-UserRole}} fails
once (HTTP 5xx, timeout, Ranger restart).
# Later sync runs, also runs that execute because the Ranger service version
changed, never delete the role.
# Every {{ozone tenant create tenant1}} fails with {{TENANT_AUTHORIZER_ERROR}},
cause "Role already exists". {{ozone tenant delete tenant1}} cannot help
because OM has no such tenant (argued, not run).
h3. Impact
The tenant id cannot be created again until an operator deletes the leftover
roles in Ranger by hand. It needs two independent events: a tenant create that
fails after its Ranger objects were created, and a failed Ranger call during
the cleanup. No data is lost. Creating a tenant requires cluster admin
privilege.
h3. Reproduction
PASS, deterministic, unmodified source. [^TestBugCR3OrphanRoleCleanup.java]
({{hadoop-ozone/ozone-manager}}) runs the real {{OMTenantCreateRequest}}
({{preExecute}}, then {{validateAndUpdateCache}} after a volume of the same
name was added to the OM DB), the real {{AuthorizerOp}}, the real
{{OMRangerBGSyncService.triggerRangerSyncOnce}} and a real OM DB. Ranger is the
in-memory controller that OM uses with {{ozone.om.tenant.dev.skip.ranger}},
wrapped in a Mockito spy that fails one delete of the user role. Before the
second and third sync run the test changes the Ranger service version with an
unrelated labeled policy, so that the sync executes. Output on the unmodified
source:
{noformat}
create #1: VOLUME_ALREADY_EXISTS; Ranger roles [tenant1-UserRole,
tenant1-AdminRole], policies [tenant1-BucketAccess, tenant1-VolumeAccess],
tenant in OM DB: false
sync run 1: completed true; Ranger roles [tenant1-UserRole], policies []
sync run 2: completed true; Ranger roles [tenant1-UserRole], policies []
sync run 3: completed true; Ranger roles [tenant1-UserRole], policies []
retry 1: TENANT_AUTHORIZER_ERROR java.io.IOException: Role already exists.
retry 2: TENANT_AUTHORIZER_ERROR java.io.IOException: Role already exists.
{noformat}
With real Ranger, the admin role delete is then refused too, because the user
role still names it as delegated admin ({{RoleDBStore.ensureRoleDeleteAllowed}}
refuses to delete a role referenced by a policy, another role or a security
zone), so both roles are left. The in-memory controller does not check this, so
only the user role is left here. A passing test means the defect is present.
Not run against a real Ranger.
h3. Suggested fix
Remember the roles whose delete failed and retry them in the following runs,
skipping any role that belongs to a tenant in OM again. Finding tenant roles in
Ranger by name or description instead was rejected, because Ranger roles are
not scoped to a service, so the sync of one cluster could delete the tenant
roles of another cluster using the same Ranger. Persisting the pending deletes
in the OM DB would survive a leader change, but needs a new DB entry.
h3. Patch
[^CR-3-retry-failed-ranger-role-deletes-in-sync.patch], against
a6b7bdb937109ba2893688b41a89470bc95c88cf. It also applies to master at
b0aa6475b78.
The sync keeps the names of roles whose delete failed in memory. At the start
of each run, before the version check, it retries each of them once, under the
authorizer write lock and against the tenant cache read inside that lock
(tenant requests hold the lock until their cache update), so a role that
belongs to a tenant again, for example one re-created after a manual cleanup,
is dropped instead of deleted. A successful retry changes the Ranger service
version, so the normal sync follows as today. The first failure of a delete is
now logged with its exception, a failed retry with one line. Behaviour changes
only while a role delete keeps failing, which includes a delete that Ranger
refuses permanently (a leftover role that is referenced from another policy,
role or security zone): each sync run (default every 600s) then makes one
Ranger delete call per such role and logs one line, with no reload and no
{{SetRangerServiceVersion}} request. The set lives on the leader only and is
cleared when the sync task runs on an OM that is not the ready leader, so a
leader change or restart before the retry succeeds loses it.
Covered by two tests in the existing {{TestOMMultiTenantManagerImpl}}.
{{testRangerSyncRetriesFailedRoleDelete}}: the user role delete fails in the
first run and in the retry of the second run, which then submits no
{{SetRangerServiceVersion}} request; the third run deletes the role and the
fourth issues no further delete. Without the change it fails with
"tenant2-UserRole left in Ranger".
{{testRangerSyncKeepsRoleOfRecreatedTenant}}: after a failed delete, the role
is deleted by hand and the tenant is created again; the next run keeps the new
tenant's role. With the change the multi-tenancy unit suites in
{{hadoop-ozone/ozone-manager}} (35 tests) and {{TestOzoneTenantShell}} and
{{TestMultiTenantVolume}} on a mini cluster (11 tests) pass, checkstyle is
clean. With the patch the reproduction fails after the second sync run, because
no role is left.
Found by code review of the OM multi-tenancy and Ranger sync paths, as part of
the TLA+ verification effort under HDDS-15926, on commit
a6b7bdb937109ba2893688b41a89470bc95c88cf. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
