[
https://issues.apache.org/jira/browse/HDDS-17240?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17240:
------------------------------
Attachment: TestBugMC5QuotaRepairVolumeKey.java
> Quota repair rolls back unflushed volume changes and writes converted volumes
> under the bare volume name
> --------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17240
> URL: https://issues.apache.org/jira/browse/HDDS-17240
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 2.0.0, 2.1.0, 2.2.0, 2.1.1, 2.1.2, 2.2.1
> Reporter: Siyao Meng
> Priority: Major
> Attachments: MC-3-quota-repair-keeps-unflushed-volume-changes.patch,
> TestBugMC3QuotaRepairStaleVolumeRow.java,
> TestBugMC4QuotaRepairTenantRefCount.java, TestBugMC5QuotaRepairVolumeKey.java
>
>
> h3. Mechanism
> {{OMQuotaRepairRequest.updateOldVolumeQuotaSupport}} finds volumes with the
> old quota value ({{OLD_QUOTA_DEFAULT}}, -2) through
> {{getVolumeTable().iterator()}}, which reads RocksDB only. For each one it
> builds the new value from that RocksDB row and puts it into the volume table
> cache. The volume table is fully cached on the OM, so that value is what
> every later request reads and what any later write of the volume persists.
> {{OzoneManagerStateMachine}} does not wait for the double buffer before
> applying the repair, so a volume change applied just before it in the log may
> not be in RocksDB yet: a volume create or delete, the {{usedNamespace}}
> change of a bucket create or delete, the {{refCount}} change of a tenant
> create or delete, or a quota change. The repair replaces it with the older
> row, or skips a volume whose create is not flushed. Whether an OM sees the
> older row depends on how far its own double buffer has flushed, so OMs that
> apply the same log can end in different states.
> {{OMQuotaRepairResponse.addToDBBatch}} then writes each converted volume with
> {{putWithBatch(batchOp, volArgs.getVolume(), volArgs)}}, so under
> {{legacyvol}} instead of the volume table key {{/legacyvol}}
> ({{getVolumeKey}}) that the cache entry and every other volume writer use.
> The real row keeps -2 and a second row is added under the bare name; both are
> loaded into the cache on the next restart.
> The bucket half of the same request reads each bucket through
> {{getBucketTable().get()}} and writes it under {{getBucketKey}}; the volume
> half does neither.
> h3. Trigger
> * A volume with namespace quota -2. Ozone 1.0.0 and earlier had no namespace
> quota field, so every volume created there reads as -2. Ozone 1.4.x converted
> them at its first start ({{ON_FIRST_UPGRADE_START}}, written straight to
> RocksDB), so a cluster that ever ran 1.4.x has none left from that time. They
> remain on clusters upgraded from 1.3.x or earlier directly to 2.x, and a pre
> 1.1 client creates new ones on a current cluster, because the OM does not
> normalize the field.
> * A full quota repair: {{ozone repair om quota start}} without {{--buckets}},
> or the upgrade action when the {{QUOTA}} layout feature is finalized. Both
> are admin only.
> * For the rollback, a change to that volume applied but not yet flushed when
> the repair is applied. For the lost conversion, an OM restart.
> h3. Impact
> Reproduced at request level on unmodified source with real requests in log
> order; the tests only decide when the double buffer flushes and when the OM
> restarts.
> * *Deleted volume comes back.* After an unflushed DeleteVolume and the
> repair, the volume is visible again, CreateBucket in it returns OK and writes
> the volume back to RocksDB. On an OM that flushed the delete first, the same
> CreateBucket returns {{VOLUME_NOT_FOUND}}.
> [^TestBugMC3QuotaRepairStaleVolumeRow.java]
> * *Bucket count lost.* After an unflushed CreateBucket and the repair, the
> volume has {{usedNamespace}} 0 with one bucket. The repair also clears the
> namespace quota, so nothing is enforced at first, but a namespace quota set
> later is checked against a count that is one too low, and deleting that
> bucket takes {{usedNamespace}} to -1.
> [^TestBugMC3QuotaRepairStaleVolumeRow.java]
> * *Tenant volume reference lost.* After an unflushed {{tenant create
> --force}} on such a volume and the repair, the volume has {{refCount}} 0.
> DeleteVolume then deletes the tenant's volume if it is empty, the tenant row
> stays and TenantDelete returns {{VOLUME_NOT_FOUND}}. A TenantDelete on the
> same state throws {{IllegalStateException: refCount should not be negative}}
> out of {{validateAndUpdateCache}}; {{OzoneManagerStateMachine.runCommand}}
> terminates the OM on such an exception (read from the code, the test stops at
> the exception). On an OM that flushed the tenant create first, DeleteVolume
> returns {{VOLUME_IS_REFERENCED}}. [^TestBugMC4QuotaRepairTenantRefCount.java]
> * By the same code, not run: an unflushed TenantDelete is rolled back to
> {{refCount}} 1 and the volume can no longer be deleted, and a volume created
> with -2 but not yet flushed is converted on one OM and left at -2 on another.
> * *Conversion lost on restart, volume listed twice.* After the repair, a
> flush and a restart, the volume reads -2 again, unless another request wrote
> the volume in between (a bucket create in it does), and {{ozone sh volume
> setquota}} keeps printing the "created before version 1.1.0" warning. The
> bare name row is loaded as a second volume with the same name: a listing of
> all volumes ({{listVolumes}} without a user) returns it twice,
> {{VolumeUtilizationMetrics}} reports it twice, and Recon, which reads the OM
> DB, lists and counts it. By the same code, the bare name row stays listed
> after the volume is deleted (not run). [^TestBugMC5QuotaRepairVolumeKey.java]
> The bare name write hides part of the rollback on restart, but fixing only
> the key makes it worse: the repair's own write then makes the rolled back
> value durable, and in the tenant case the replayed TenantDelete terminates
> the OM again after every restart. Both changes are in one patch for that
> reason.
> Not run on a three OM cluster; the difference between OMs is shown by
> applying the same log with and without the flush.
> h3. Reproduction
> The three attached classes go into {{hadoop-ozone/ozone-manager}} (target
> package and Maven command in the class comment) and pass on unmodified source
> while the bug is present. They apply real CreateVolume, DeleteVolume,
> CreateBucket, CreateTenant, DeleteTenant and QuotaRepair requests through
> {{validateAndUpdateCache}} and flush the responses as the double buffer does;
> the first two have a control that flushes before the repair, the third
> reopens the OM metadata manager on the same RocksDB directory, twice. The
> legacy volume is created by a CreateVolume request without the namespace
> quota field, or written as Ozone 1.0.0 left it. The multi tenant manager and
> S3 secret manager are mocks.
> h3. Patch
> [^MC-3-quota-repair-keeps-unflushed-volume-changes.patch], against
> 20d95ed37fe35013b5320713665db03c06ee36fa. It also applies to master at
> 41d95dd3286.
> The repair now finds candidates through {{getVolumeTable().cacheIterator()}},
> the view every request reads and the one {{listAllVolumes}} already uses, so
> deleted volumes are skipped and unflushed creates are included. Inside the
> existing volume lock it rereads the volume and resets the quota on that
> current value. The response writes the volume under {{getVolumeKey}}. Nothing
> changes when no volume write is pending, apart from the key.
> {{testOldVolumeQuotaRepairKeepsUnflushedVolumeChanges}} in the existing
> {{TestOMQuotaRepairRequest}} covers an unflushed delete, bucket count and
> refCount change, quota change and create; it fails without the change
> ("expected: null").
> {{TestQuotaRepairTask.testQuotaRepairForOldVersionVolumeBucket}} now also
> reads the row from RocksDB and asserts that no bare name row exists; it fails
> without the change ("expected: -1L"). With the patch both pass, the quota
> repair, volume, tenant and bucket request suites (86 tests) pass, as does
> {{TestOzoneRepairShell}} on a mini cluster, and checkstyle is clean. With the
> patch the five bug tests of the three reproductions fail and the two controls
> pass.
> Volumes already damaged are not repaired, and bare name rows written by
> earlier repairs (2.0.0 to 2.2.1) stay. Those rows are never read as a volume,
> because every lookup uses the {{/}} prefixed key; they show up only where the
> whole table is iterated, as listed above. No Ozone tool removes a single
> volume table row ({{ozone debug ldb}} is read only, and {{ozone sh volume
> delete}} addresses the {{/}} prefixed key), so removing them needs a
> follow-up.
> Found by TLA+ model checking and code review of the OM volume, bucket and
> snapshot lifecycle paths under HDDS-15926, on commit
> 20d95ed37fe35013b5320713665db03c06ee36fa. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. The attached patch
> is a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
