Siyao Meng created HDDS-17240:
---------------------------------
Summary: Quota repair rolls back unflushed volume changes and
writes converted volumes under the bare volume name
Key: HDDS-17240
URL: https://issues.apache.org/jira/browse/HDDS-17240
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments: MC-3-quota-repair-keeps-unflushed-volume-changes.patch,
TestBugMC3QuotaRepairStaleVolumeRow.java,
TestBugMC4QuotaRepairTenantRefCount.java, TestBugMC5QuotaRepairVolumeKey.java
h3. Mechanism
{{OMQuotaRepairRequest.updateOldVolumeQuotaSupport}} finds volumes with the old
quota value ({{OLD_QUOTA_DEFAULT}}, -2) through
{{getVolumeTable().iterator()}}, which reads RocksDB only. For each one it
builds the new value from that RocksDB row and puts it into the volume table
cache. The volume table is fully cached on the OM, so that value is what every
later request reads and what any later write of the volume persists.
{{OzoneManagerStateMachine}} does not wait for the double buffer before
applying the repair, so a volume change applied just before it in the log may
not be in RocksDB yet: a volume create or delete, the {{usedNamespace}} change
of a bucket create or delete, the {{refCount}} change of a tenant create or
delete, or a quota change. The repair replaces it with the older row, or skips
a volume whose create is not flushed. Whether an OM sees the older row depends
on how far its own double buffer has flushed, so OMs that apply the same log
can end in different states.
{{OMQuotaRepairResponse.addToDBBatch}} then writes each converted volume with
{{putWithBatch(batchOp, volArgs.getVolume(), volArgs)}}, so under {{legacyvol}}
instead of the volume table key {{/legacyvol}} ({{getVolumeKey}}) that the
cache entry and every other volume writer use. The real row keeps -2 and a
second row is added under the bare name; both are loaded into the cache on the
next restart.
The bucket half of the same request reads each bucket through
{{getBucketTable().get()}} and writes it under {{getBucketKey}}; the volume
half does neither.
h3. Trigger
* A volume with namespace quota -2. Ozone 1.0.0 and earlier had no namespace
quota field, so every volume created there reads as -2. Ozone 1.4.x converted
them at its first start ({{ON_FIRST_UPGRADE_START}}, written straight to
RocksDB), so a cluster that ever ran 1.4.x has none left from that time. They
remain on clusters upgraded from 1.3.x or earlier directly to 2.x, and a pre
1.1 client creates new ones on a current cluster, because the OM does not
normalize the field.
* A full quota repair: {{ozone repair om quota start}} without {{--buckets}},
or the upgrade action when the {{QUOTA}} layout feature is finalized. Both are
admin only.
* For the rollback, a change to that volume applied but not yet flushed when
the repair is applied. For the lost conversion, an OM restart.
h3. Impact
Reproduced at request level on unmodified source with real requests in log
order; the tests only decide when the double buffer flushes and when the OM
restarts.
* *Deleted volume comes back.* After an unflushed DeleteVolume and the repair,
the volume is visible again, CreateBucket in it returns OK and writes the
volume back to RocksDB. On an OM that flushed the delete first, the same
CreateBucket returns {{VOLUME_NOT_FOUND}}.
[^TestBugMC3QuotaRepairStaleVolumeRow.java]
* *Bucket count lost.* After an unflushed CreateBucket and the repair, the
volume has {{usedNamespace}} 0 with one bucket. The repair also clears the
namespace quota, so nothing is enforced at first, but a namespace quota set
later is checked against a count that is one too low, and deleting that bucket
takes {{usedNamespace}} to -1. [^TestBugMC3QuotaRepairStaleVolumeRow.java]
* *Tenant volume reference lost.* After an unflushed {{tenant create --force}}
on such a volume and the repair, the volume has {{refCount}} 0. DeleteVolume
then deletes the tenant's volume if it is empty, the tenant row stays and
TenantDelete returns {{VOLUME_NOT_FOUND}}. A TenantDelete on the same state
throws {{IllegalStateException: refCount should not be negative}} out of
{{validateAndUpdateCache}}; {{OzoneManagerStateMachine.runCommand}} terminates
the OM on such an exception (read from the code, the test stops at the
exception). On an OM that flushed the tenant create first, DeleteVolume returns
{{VOLUME_IS_REFERENCED}}. [^TestBugMC4QuotaRepairTenantRefCount.java]
* By the same code, not run: an unflushed TenantDelete is rolled back to
{{refCount}} 1 and the volume can no longer be deleted, and a volume created
with -2 but not yet flushed is converted on one OM and left at -2 on another.
* *Conversion lost on restart, volume listed twice.* After the repair, a flush
and a restart, the volume reads -2 again, unless another request wrote the
volume in between (a bucket create in it does), and {{ozone sh volume
setquota}} keeps printing the "created before version 1.1.0" warning. The bare
name row is loaded as a second volume with the same name: a listing of all
volumes ({{listVolumes}} without a user) returns it twice,
{{VolumeUtilizationMetrics}} reports it twice, and Recon, which reads the OM
DB, lists and counts it. By the same code, the bare name row stays listed after
the volume is deleted (not run). [^TestBugMC5QuotaRepairVolumeKey.java]
The bare name write hides part of the rollback on restart, but fixing only the
key makes it worse: the repair's own write then makes the rolled back value
durable, and in the tenant case the replayed TenantDelete terminates the OM
again after every restart. Both changes are in one patch for that reason.
Not run on a three OM cluster; the difference between OMs is shown by applying
the same log with and without the flush.
h3. Reproduction
The three attached classes go into {{hadoop-ozone/ozone-manager}} (target
package and Maven command in the class comment) and pass on unmodified source
while the bug is present. They apply real CreateVolume, DeleteVolume,
CreateBucket, CreateTenant, DeleteTenant and QuotaRepair requests through
{{validateAndUpdateCache}} and flush the responses as the double buffer does;
the first two have a control that flushes before the repair, the third reopens
the OM metadata manager on the same RocksDB directory, twice. The legacy volume
is created by a CreateVolume request without the namespace quota field, or
written as Ozone 1.0.0 left it. The multi tenant manager and S3 secret manager
are mocks.
h3. Patch
[^MC-3-quota-repair-keeps-unflushed-volume-changes.patch], against
20d95ed37fe35013b5320713665db03c06ee36fa. It also applies to master at
41d95dd3286.
The repair now finds candidates through {{getVolumeTable().cacheIterator()}},
the view every request reads and the one {{listAllVolumes}} already uses, so
deleted volumes are skipped and unflushed creates are included. Inside the
existing volume lock it rereads the volume and resets the quota on that current
value. The response writes the volume under {{getVolumeKey}}. Nothing changes
when no volume write is pending, apart from the key.
{{testOldVolumeQuotaRepairKeepsUnflushedVolumeChanges}} in the existing
{{TestOMQuotaRepairRequest}} covers an unflushed delete, bucket count and
refCount change, quota change and create; it fails without the change
("expected: null").
{{TestQuotaRepairTask.testQuotaRepairForOldVersionVolumeBucket}} now also reads
the row from RocksDB and asserts that no bare name row exists; it fails without
the change ("expected: -1L"). With the patch both pass, the quota repair,
volume, tenant and bucket request suites (86 tests) pass, as does
{{TestOzoneRepairShell}} on a mini cluster, and checkstyle is clean. With the
patch the five bug tests of the three reproductions fail and the two controls
pass.
Volumes already damaged are not repaired, and bare name rows written by earlier
repairs (2.0.0 to 2.2.1) stay. Those rows are never read as a volume, because
every lookup uses the {{/}} prefixed key; they show up only where the whole
table is iterated, as listed above. No Ozone tool removes a single volume table
row ({{ozone debug ldb}} is read only, and {{ozone sh volume delete}} addresses
the {{/}} prefixed key), so removing them needs a follow-up.
Found by TLA+ model checking and code review of the OM volume, bucket and
snapshot lifecycle paths under HDDS-15926, on commit
20d95ed37fe35013b5320713665db03c06ee36fa. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
