Siyao Meng created HDDS-17198:
---------------------------------
Summary: Force creating a tenant on a volume already used by
another tenant terminates every OM, also on restart
Key: HDDS-17198
URL: https://issues.apache.org/jira/browse/HDDS-17198
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
MC-3-reject-tenant-create-on-volume-used-by-another-tenant.patch
h3. Mechanism
With {{forceCreationWhenVolumeExists}} set,
{{OMTenantCreateRequest.preExecute}} skips the volume check and never looks at
the reference count of the existing volume, so the request is committed. In
{{OMTenantCreateRequest.validateAndUpdateCache}} the force path increments the
{{refCount}} of the existing volume and then runs
{{Preconditions.checkState(omVolumeArgs.getRefCount() == 1L, "refCount should
have been set to 1")}}. If the volume already belongs to a tenant, the count
becomes 2 and an {{IllegalStateException}} is thrown. The request only catches
{{IOException}} and {{InvalidPathException}}, so the exception reaches
{{OzoneManagerStateMachine.runCommand}}, which calls {{ExitUtils.terminate}}.
The entry is in the Raft log, so every OM terminates when it applies it, and an
OM that is restarted replays it and terminates again.
In the review of HDDS-8994 (apache/ozone#5045, which added the hidden
{{--force}} option) this case was discussed and the precondition was described
as what makes several tenants on one volume impossible, so a rejection was
intended rather than a crash.
h3. Trigger
# A cluster admin creates tenant {{tenanta}} (volume {{tenanta}}, {{refCount}}
1).
# The admin calls {{ObjectStore.createTenant("tenantb",
TenantArgs.newBuilder().setVolumeName("tenanta").setForceCreationWhenVolumeExists(true).build())}}.
{{RpcClient.createTenant}} passes both fields unchanged. The shell cannot do it
on its own, because {{ozone tenant create --force}} always uses the tenant name
as the volume name. Two admins force creating two different tenants on the same
plain volume at the same time also reach it, because both pass the volume
checks in {{preExecute}}, which run before the authorizer lock is taken, before
either is applied (covered by the unit test below, not run on a cluster).
h3. Impact
* Every OM terminates on the committed entry, and the client gets
{{INTERNAL_ERROR}}.
* A restarted OM terminates again when it replays the entry, so the OM service
stays down until the binary is fixed or the entry is removed from the log.
* The request requires cluster admin rights.
h3. Reproduction
The issue has no standalone reproduction. The regression test in the attached
patch covers it (see below).
h3. Patch
[^MC-3-reject-tenant-create-on-volume-used-by-another-tenant.patch], against
a6b7bdb937109ba2893688b41a89470bc95c88cf. It also applies to master at
1753288f333.
{{validateAndUpdateCache}} throws {{OMException}} with the existing result code
{{VOLUME_IS_REFERENCED}} instead of failing the precondition, so the request
fails and the OM keeps running. This also makes replay of an entry that is
already in the log harmless once the OMs run the fixed code. {{preExecute}}
rejects the request with the same code when the existing volume has a nonzero
{{refCount}}, so in the common case nothing is created in Ranger for the
rejected tenant. No proto or client change. Requests that succeeded before
behave the same.
Covered by {{testForceCreationWhenVolumeIsUsedByAnotherTenant}} in the existing
{{TestOMTenantCreateRequest}}: two force requests on one plain volume are both
pre executed, the second apply must return {{VOLUME_IS_REFERENCED}} and leave
the volume and tenant tables unchanged, and a later request must be rejected in
{{preExecute}}. Without the change it fails with {{IllegalStateException:
refCount should have been set to 1}}. With it the tenant and volume request
suites and {{TestOMMultiTenantManagerImpl}} (50 tests) and the
{{TestMultiTenantVolume}} and {{TestOzoneTenantShell}} mini cluster suites (11
tests) pass, checkstyle is clean. Not run: a restart of a fixed OM over a log
written by an unfixed one.
Found by TLA+ model checking and code review of the OM multi-tenancy and Ranger
sync paths under HDDS-15926, on commit
a6b7bdb937109ba2893688b41a89470bc95c88cf. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
