[
https://issues.apache.org/jira/browse/HDDS-17459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-17459:
------------------------------
Attachment: TestBugMC3DeletedS3VolumeRecreatedOnRestart.java
> OM restart re-creates a deleted S3 volume outside Ratis and replaces the OM
> user's volume list, so OMs diverge
> --------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-17459
> URL: https://issues.apache.org/jira/browse/HDDS-17459
> Project: Apache Ozone
> Issue Type: Bug
> Affects Versions: 1.0.0, 1.1.0, 1.2.0, 1.3.0, 1.2.1, 1.4.0, 2.0.0, 1.4.1,
> 2.1.0, 2.2.0, 2.1.1, 2.1.2, 2.2.1
> Reporter: Siyao Meng
> Priority: Major
> Attachments:
> MC-3-do-not-recreate-s3-volume-outside-ratis-on-restart.patch,
> TestBugMC3DeletedS3VolumeRecreatedOnRestart.java
>
>
> h3. Mechanism
> The {{OzoneManager}} constructor calls {{addS3GVolumeToDB}} before the Ratis
> server is set up. If the volume table has no entry for the S3 volume
> ({{ozone.s3g.volume.name}}, default {{s3v}}), it commits the volume and a new
> {{userTable}} entry for the OM service user, holding only the S3 volume,
> directly to RocksDB and the table caches. No log entry is involved, and the
> user entry replaces the user's existing volume list instead of adding to it.
> {{OMVolumeDeleteRequest}} has no special case for the S3 volume, so an empty
> {{s3v}} can be deleted through Ratis like any other volume. After that, every
> OM process restart runs the constructor again and writes the deleted volume
> and the one entry user list on that OM only.
> h3. Trigger
> # The empty {{s3v}} is deleted through the client ({{ozone sh volume delete
> /s3v}}), for example on a cluster that does not use S3.
> # Any OM process is restarted.
> The volume list part needs the OM service user to own another volume, for
> example one created on a cluster where clients run as that user.
> No fault is needed. The mini cluster's {{restartOzoneManager}} reuses the
> {{OzoneManager}} instance and skips the constructor, so the existing restart
> tests do not show it.
> h3. Impact
> Run on a three OM mini cluster on unmodified source, restarting one follower
> with {{OzoneManager.createOm(conf)}} plus {{start()}}:
> * After the restart the restarted OM has {{s3v}} and
> {{userTable[omUser]=[s3v]}}, while its peers have no {{s3v}} and {{[vol1]}}.
> All three are at the same applied index.
> * With the restarted OM as leader, the client sees the deleted volume again,
> and {{listVolumesByUser}} returns {{[s3v]}} without {{vol1}}.
> * {{createBucket}} in {{s3v}} through that leader succeeds, and the bucket
> exists on that OM only. Keys written to such a bucket would exist on one
> replica (argued, not run).
> * A replicated {{CreateVolume(s3v)}} is rejected with
> {{VOLUME_ALREADY_EXISTS}} on the restarted OM and applied on the others, so
> it does not repair the divergence.
> Nothing reconciles state written outside the log; only a checkpoint install
> on that OM replaces it. A single OM deployment is affected too: the OM user's
> other volumes drop out of its list and {{s3v}} comes back with no log entry.
> h3. Reproduction
> [^TestBugMC3DeletedS3VolumeRecreatedOnRestart.java]
> ({{hadoop-ozone/integration-test}}, target path and Maven command in the
> class comment) runs three OMs without datanodes, with volume and bucket
> operations through the client API and leadership moved through the OM admin
> client: create {{vol1}}, delete {{s3v}}, restart one follower by building a
> new {{OzoneManager}} on its configuration, move leadership to it, then create
> a bucket in {{s3v}} and create {{s3v}} again. It prints each OM's state after
> waiting for all OMs to reach the same applied index and flushing their double
> buffers. A passing test means the defect is present. With the patch it fails
> at the check right after the restart, because the restarted OM keeps
> {{s3v=false userTable=[vol1]}} like its peers.
> h3. Suggested fix
> Create the S3 volume at startup only on a new DB. An existing DB without it
> lost it through the log (or the configured name was changed), and recreating
> it belongs in the log as well ({{ozone sh volume create}}). Moving the
> creation itself into a Ratis request submitted by the leader would also
> remove the remaining per OM differences of the startup volume (creation time,
> group ACLs), but needs new request handling.
> h3. Patch
> [^MC-3-do-not-recreate-s3-volume-outside-ratis-on-restart.patch], against
> a5814528cdc8d3c1b27f259e0db2256b585cb825. It also applies to master at
> b6d0181075e.
> {{addS3GVolumeToDB}} now returns early, with one WARN line, when the volume
> is missing and the DB already has a {{TransactionInfo}} row, that is, when
> transactions have been applied to it. A new DB still gets the volume on every
> OM as before. Because the user table of a new DB is empty, the list overwrite
> cannot happen any more either. Operator visible change: after changing
> {{ozone.s3g.volume.name}} on an existing cluster, the new volume has to be
> created through the client instead of being created by each OM at restart.
> Covered by {{testDeletedS3VolumeNotRecreatedByNewOMInstance}} in the existing
> {{TestOzoneManagerRestart}}: it deletes {{s3v}}, starts a new
> {{OzoneManager}} instance on the same DB, and checks through the client that
> {{s3v}} stays deleted, that the OM user's other volume is still listed, and
> that {{s3v}} can be created again. It fails without the change because
> {{s3v}} is back. With the change {{TestOzoneManagerRestart}},
> {{TestOmMetrics}}, {{TestAddRemoveOzoneManager}},
> {{TestOzoneManagerHAWithStoppedNodes}} and the {{ozone-manager}} suites that
> start an OM on a new DB ({{TestKeyManagerUnit}}, {{TestBucketManagerImpl}},
> {{TestVolumeOwner}}, {{TestOzoneManagerStarter}}) pass (76 tests), checkstyle
> is clean. An OM that already diverged is not repaired.
> Found by TLA+ model checking and code review of the OM table cache, double
> buffer flush and startup paths under HDDS-15926, on commit
> a5814528cdc8d3c1b27f259e0db2256b585cb825. Checked against HDDS issues and
> apache/ozone pull requests for duplicates before filing (nearest: HDDS-3993,
> which added the startup creation, and HDDS-13823, about the cache entry of
> the same write). The attached patch is a proposal for review. Generated with
> Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
