[
https://issues.apache.org/jira/browse/HDDS-17237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian-Xuan Yang reassigned HDDS-17237:
-------------------------------------
Assignee: Dian-Xuan Yang
> Tenant create on a Kerberos cluster overwrites the creating admin's volume
> list, so their existing volumes disappear from listVolumes
> -------------------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17237
> URL: https://issues.apache.org/jira/browse/HDDS-17237
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 1.3.0, 1.4.0, 2.0.0, 1.4.1, 2.1.0, 2.2.0, 2.1.1, 2.1.2,
> 2.2.1
> Reporter: Siyao Meng
> Assignee: Dian-Xuan Yang
> Priority: Major
> Attachments:
> CR-3-use-volume-owner-as-user-key-on-tenant-create.patch,
> TestBugCR3TenantCreateOwnerVolumeList.java
>
>
> h3. Mechanism
> {{OMTenantCreateRequest.preExecute}} sets the owner of the tenant volume to
> the caller's short name ({{ugi.getShortUserName()}}).
> {{OMTenantCreateRequest.validateAndUpdateCache}} instead uses
> {{UserInfo.userName}} as the owner, which {{OMClientRequest.getUserInfo}}
> fills with {{ugi.getUserName()}}, the full principal under Kerberos (for
> example {{[email protected]}}). It takes the user lock, reads the owner's
> volume list and adds the updated list to the user table cache under that
> principal. {{OMTenantCreateResponse.addToDBBatch}} then writes the list under
> {{getUserKey(omVolumeArgs.getOwnerName())}}, the short name.
> Every other request that writes the user table ({{OMVolumeCreateRequest}},
> {{OMVolumeDeleteRequest}}, {{OMVolumeSetOwnerRequest}}) keys it by the volume
> owner, which {{RpcClient.createVolume}} sets to the short name. So the list
> read under the full principal is normally empty, the computed list is just
> the tenant volume, and the flush replaces the owner's real list with it.
> h3. Trigger
> # On a Kerberos cluster, {{[email protected]}} (mapped to {{alice}} by
> auth_to_local) owns at least one volume, for example {{vol1}} or the volume
> of a tenant she created earlier.
> # alice, an OM admin, runs {{ozone tenant create tenant1}}.
> S3 multi-tenancy can only be enabled with Kerberos, so every tenant create in
> a real deployment runs with a full principal. Without Kerberos (user name
> equals short name) both keys match and the result is correct.
> h3. Impact
> * After the flush the user table row {{alice}} is {{[tenant1]}}.
> {{listVolumes}} for alice, which serves {{OzoneManager.listVolumeByUser}} and
> {{ozone sh volume list}} for the user, no longer returns {{vol1}}. The volume
> still exists, is still owned by alice and its data is untouched.
> * The per-user volume count ({{ozone.om.user.max.volume}}) reads the same
> row, so it undercounts from then on, and it already undercounts in the check
> during tenant create itself.
> * The row is permanent. No request rebuilds the user table from the volume
> table, later volume creates by alice append to the truncated list, and each
> further tenant create by alice truncates it again, so an admin who created
> several tenants sees only the last tenant volume.
> * If the owner's list later becomes empty, for example after the listed
> tenant volume is deleted, the row is removed. Deleting a volume that was
> dropped from the list, or changing its owner, then fails with
> {{USER_NOT_FOUND}} ({{OMVolumeRequest.delVolumeFromOwnerList}}), and the
> volume stays. Creating any volume for the owner first makes both work again.
> h3. Reproduction
> PASS on unmodified source when the defect is present.
> [^TestBugCR3TenantCreateOwnerVolumeList.java]
> ({{hadoop-ozone/ozone-manager}}; target path and Maven command in the class
> comment) sets the RPC caller to a remote UGI {{[email protected]}} with an
> auth_to_local rule mapping it to {{alice}}, as the RPC server does under
> Kerberos. Each request runs through the real {{preExecute}},
> {{validateAndUpdateCache}}, response {{checkAndUpdateDB}}, a batch commit and
> a cache cleanup, and the test waits until the user table cache is evicted, as
> it is on an OM by the next command. alice's row is read from RocksDB.
> * alice owns {{vol1}} and creates {{tenant1}}: her row is {{[tenant1]}}.
> After {{tenant2}} it is {{[tenant2]}}. Setting the owner of {{vol1}} to
> {{bob}} and back to {{alice}} gives {{[tenant2, vol1]}}.
> * alice owns {{vol1}}, creates {{tenant1}}, deletes the tenant and its volume
> (both OK): her row is gone. Deleting {{vol1}} and setting its owner to
> {{bob}} both fail with {{USER_NOT_FOUND User not found: alice}}, and {{vol1}}
> is still in the volume table. After creating {{vol2}} for alice, deleting
> {{vol1}} is OK.
> * Control with caller {{alice}}: the same steps keep {{[vol1, tenant1,
> tenant2]}} and every delete is OK.
> With the patch the two Kerberos tests fail at their first check (alice's row
> is {{[vol1, tenant1]}}, and {{[vol1]}} after the tenant volume is deleted)
> and the control passes. Not run on a Kerberos cluster.
> h3. Patch
> [^CR-3-use-volume-owner-as-user-key-on-tenant-create.patch], against
> 20d95ed37fe35013b5320713665db03c06ee36fa. It also applies to master at
> 41d95dd3286.
> {{validateAndUpdateCache}} takes the owner from the {{VolumeInfo}} built in
> {{preExecute}} instead of {{UserInfo.userName}}, so the user lock, the read,
> the cache entry and the DB write use the same key as the response and the
> volume requests. Audit logging still uses {{UserInfo}}. Lists that were
> already overwritten are not repaired. A dropped volume can be put back by
> setting its owner to another user and back, or, if the owner's row is gone,
> after creating any volume for that owner. During a rolling upgrade an unfixed
> OM still writes the truncated list for the same entry.
> Covered by {{testValidateAndUpdateCacheAddsVolumeToOwnerVolumeList}} in the
> existing {{TestOMTenantCreateRequest}}, parameterized with a short name
> (control, passes before and after) and a full principal in {{UserInfo}},
> checking that the cache holds the owner's list only under the owner key, and
> the list in the cache and in RocksDB after the response is written. Without
> the change the principal case fails because the cache entry is under
> {{[email protected]}}. With it the tenant request suites,
> {{TestOMMultiTenantManager}}, {{TestOMMultiTenantManagerImpl}} and the volume
> request suites, including volume delete and set owner (59 tests), and the
> mini cluster suites {{TestMultiTenantVolume}} and {{TestOzoneTenantShell}}
> (11 tests, without Kerberos, run before the test was extended) pass,
> checkstyle is clean.
> Found by code review of the OM volume, bucket and snapshot lifecycle paths,
> as part of the TLA+ verification effort under HDDS-15926, on commit
> 20d95ed37fe35013b5320713665db03c06ee36fa. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
