[
https://issues.apache.org/jira/browse/HDDS-17295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17295:
------------------------------
Attachment: MC-8-release-om-read-locks-on-the-acquired-lock.patch
> OzoneManager read paths release their volume or bucket read lock on a lock
> that a checkpoint install may have replaced
> ----------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17295
> URL: https://issues.apache.org/jira/browse/HDDS-17295
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 2.2.0, 2.2.1
> Reporter: Siyao Meng
> Priority: Minor
> Attachments: MC-8-release-om-read-locks-on-the-acquired-lock.patch,
> TestBugMC8VolumeOwnerLockAcrossInstall.java
>
>
> h3. Mechanism
> {{OzoneManager.getVolumeOwner(String)}} takes the VOLUME read lock on
> {{metadataManager.getLock()}} and releases it in its {{finally}} block on a
> second read of the {{metadataManager}} field.
> {{OzoneManager.installCheckpoint}} replaces that field, and with it the lock,
> in {{reloadOMState}}. If the replacement happens between the acquire and the
> release, the release goes to the new lock and throws
> {{IllegalMonitorStateException}}, and the read hold on the replaced lock is
> never released. {{getBucketOwner}}, {{getLifecycleConfiguration}},
> {{listUsersInTenant}}, {{getS3VolumeContext}} and
> {{SstFilteringService.markSSTFilteredFlagForSnapshot}} use the same pattern.
> h3. Trigger
> A read with an ACL check (it calls {{getVolumeOwner}} or {{getBucketOwner}}
> through {{OmMetadataReader.checkAcls}}), or one of the other methods above,
> is in progress on an OM while that OM installs a checkpoint from the leader,
> and it reaches the lock acquire before the swap and the release after it. A
> follower installs a checkpoint when it lags behind the purged Raft log. On a
> follower such reads are served only with follower reads (off by default on
> client and server). {{installCheckpoint}} stops the Hadoop RPC server before
> the swap, but not the OM gRPC server ({{ozone.om.s3.grpc.server_enabled}}, on
> by default), so gRPC requests keep being served during the install. A Hadoop
> RPC call that is already running when the server stops loses its connection
> anyway. Any authenticated client can send such a read; it cannot make the
> install happen. For {{SstFilteringService}} it needs a filtering task that
> outlives the shutdown wait of {{KeyManager.stop}}.
> h3. Impact
> * A read over the OM gRPC server fails with {{IllegalMonitorStateException}}
> (returned as {{Status.INTERNAL}}) although the volume exists. The failed call
> is shown in the reproduction, in process. Argued from code, not run over a
> real connection: the gRPC follower read client treats this error as a reason
> to try another OM.
> * The read hold on the replaced lock is never released. Nothing uses the
> replaced lock after the install, so this holds no current request up.
> * {{OzoneManagerLock}} updates its per thread lock tracker after the unlock,
> so the failed release also leaves a stale bit on that thread. A stale VOLUME
> bit only refuses {{S3_BUCKET_LOCK}}, which no OM code acquires. A stale
> BUCKET bit (from {{getBucketOwner}} or {{getLifecycleConfiguration}}) also
> refuses {{VOLUME_LOCK}} on that thread until the thread next releases a
> BUCKET lock. Hadoop RPC handler threads exit with the stopped server; a gRPC
> handler thread keeps the bit (code reading).
> h3. Reproduction
> PASS, deterministic, unmodified source.
> [^TestBugMC8VolumeOwnerLockAcrossInstall.java]
> ({{hadoop-ozone/ozone-manager}}) starts one OM through {{OmTestManagers}} and
> creates a volume. It holds the VOLUME write lock of the current lock, starts
> {{getVolumeOwner}} on another thread so that it waits in its read lock
> acquire, installs a checkpoint of the OM's own DB with {{installCheckpoint}}
> (presented with a newer transaction index so that it proceeds), and then
> releases the write lock. The lookup fails with
> {{IllegalMonitorStateException}}. A passing test means the defect is present.
> h3. Patch
> [^MC-8-release-om-read-locks-on-the-acquired-lock.patch], against
> e0de543cbe9d73f3bf5473a45e1c59cd6eae7173. It also applies to master at
> 669912118cf.
> The six methods read the lock once into a local variable and release on that
> same lock. Nothing else changes: the same lock is taken as before, and the
> data is read as before.
> Covered by {{testOwnerLookupsAcrossInstallReleaseTheirLocks}} in the existing
> {{TestOMRatisSnapshots}}: on the active follower of the suite's HA cluster it
> puts a {{getVolumeOwner}} and a {{getBucketOwner}} call across an install in
> the same way (the install fails at its first backup move with the suite's
> existing fault injector and reloads the old DB, which also replaces the
> metadata manager and the lock) and checks that both calls return the owner.
> Without the change the test fails with {{IllegalMonitorStateException}}. With
> it the test, {{TestOMRatisSnapshots}}, {{TestSstFilteringService}},
> {{TestMultiTenantVolume}}, {{TestOMMetadataReader}} and
> {{TestOzoneIntegrationNonHA}} pass, and checkstyle is clean. With the patch
> the reproduction fails at its assertion, because the lookup returns the owner.
> Found by TLA+ model checking and code review of the Ozone Manager lock
> ordering paths under HDDS-15926, on commit
> e0de543cbe9d73f3bf5473a45e1c59cd6eae7173. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing. Related but not
> duplicates: HDDS-4088 (added the owner lookup for the authorizer), HDDS-9279
> (follower reads, which make this reachable), HDDS-6833 (caching to reduce
> lock contention in {{getVolumeOwner}}), HDDS-17120 and HDDS-16878 (checkpoint
> servlets that keep a DB store an install replaced, the same pattern for
> another object). The attached patch is a proposal for review. Generated with
> Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
