[
https://issues.apache.org/jira/browse/HDDS-16849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16849:
------------------------------
Attachment:
MC-3-reject-rename-of-hsynced-key-in-obs-and-legacy-buckets.patch
> OBS/LEGACY key rename does not reject an hsync'ed key as FSO does
> -----------------------------------------------------------------
>
> Key: HDDS-16849
> URL: https://issues.apache.org/jira/browse/HDDS-16849
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Siyao Meng
> Priority: Major
> Attachments:
> MC-3-reject-rename-of-hsynced-key-in-obs-and-legacy-buckets.patch
>
>
> h3. Mechanism
> A key that a writer has hsync'ed has two rows: its row in the key table,
> which carries the hsync state ({{OmKeyInfo.isHsync}}), and the writer's open
> key row, both under the key's name. On FILE_SYSTEM_OPTIMIZED buckets
> {{OMKeyRenameRequestWithFSO}} rejects renaming such a key with
> {{RENAME_OPEN_FILE}} (HDDS-8545). On OBJECT_STORE and LEGACY buckets
> {{OMKeyRenameRequest}} and {{OMKeysRenameRequest}} have no such check, only
> {{// TODO: Add support for renaming open key}}. They move the key table row,
> hsync state included, to the new name and leave the writer's open key row
> under the old name.
> {{OmMetadataManagerImpl.getExpiredOpenKeys}} relies on the two rows matching.
> For an open key row past {{ozone.om.lease.hard.limit}} that is hsync'ed, it
> reads the key table row under the same name, and after such a rename there is
> none, so the call throws. {{OpenKeyCleanupService.OpenKeyCleanupTask.call}}
> only catches {{IOException}}, so the DEFAULT layout run fails before it
> submits anything.
> h3. Trigger
> Hsync enabled on the OM ({{ozone.hbase.enhancements.allowed=true}} and
> {{ozone.fs.hsync.enabled=true}}, both false by default) and an OBJECT_STORE
> or LEGACY bucket:
> # A writer opens a key, writes and calls hsync.
> # The key is renamed while the writer still has it open
> ({{OzoneBucket.renameKey}} or {{renameKeys}}, {{ozone sh key rename}}, or an
> ofs/o3fs rename).
> # The writer stops (dies, or stays idle without another hsync or close) until
> its open key is past the lease hard limit (default 7d).
> h3. Impact
> * Open key cleanup for OBS and LEGACY buckets fails on that OM. Each DEFAULT
> layout run that reaches the leftover open key row fails, once the expired
> keys sorted in front of it are fewer than
> {{ozone.om.open.key.cleanup.limit.per.task}} (default 1000). Expired open
> keys sorted after the row are never cleaned, and expired hsync'ed keys are
> not auto committed. FSO buckets are not affected.
> * It lasts until the writer resumes: an hsync or close under the old name
> brings the two rows back in line. A writer that has died never does, and
> lease recovery only handles FSO.
> h3. Reproduction
> There is no standalone reproduction. The regression tests in the attached
> patch set the hsync state of the source key internally, as the existing FSO
> test does, and fail on unmodified source:
> * {{TestOMKeyRenameRequest.testRenameOpenFile}}
> ({{hadoop-ozone/ozone-manager}}): renaming an hsync'ed key must fail with
> {{RENAME_OPEN_FILE}} and leave the source in place. Without the change it
> fails with {{expected: <RENAME_OPEN_FILE> but was: <OK>}}.
> * {{TestOMKeysRenameRequest.testKeysRenameRequestWithOpenFile}}: in a
> {{RenameKeys}} request with ten plain keys and one hsync'ed key, the hsync'ed
> key must be the only pair not renamed. Without the change it fails with
> {{expected: <PARTIAL_RENAME> but was: <OK>}}.
> h3. Patch
> [^MC-3-reject-rename-of-hsynced-key-in-obs-and-legacy-buckets.patch], against
> a6b7bdb937109ba2893688b41a89470bc95c88cf. It also applies to master at
> b0aa6475b78.
> {{OMKeyRenameRequest}} now fails with {{RENAME_OPEN_FILE}} when the source
> key is hsync'ed, with the message used by {{OMKeyRenameRequestWithFSO}}.
> {{OMKeysRenameRequest}} reports such a pair in {{unRenamedKeys}} and returns
> {{PARTIAL_RENAME}}, as it does for other pairs it cannot rename. Keys that
> are not hsync'ed, closed keys and FSO buckets are unaffected.
> The check changes what the apply writes, so it is enabled by a new
> {{OzoneManagerVersion.RENAME_REJECTS_HSYNC_KEY}} (101) through
> {{OMVersionManager.isAllowed}}. Until the cluster is finalized to that
> version every OM applies the rename as before, so OMs on different software
> versions never apply the same log entry differently, and the defect remains
> until finalization. The version number is a placeholder to be agreed at
> merge, since other open patches also add an OM version 101.
> {{testRenameOpenFile}} and
> {{TestOMKeysRenameRequest.testKeysRenameRequestWithOpenFileBeforeFinalization}}
> check that an OM that is not finalized still applies the rename.
> Visible change with hsync enabled, after finalization: renaming a key that is
> still being written with hsync fails instead of succeeding. An o3fs/ofs
> directory rename on a LEGACY bucket renames key by key and stops at such a
> file, so the files renamed before it stay at the destination. HBase on Ozone
> needs FSO buckets, because lease recovery is FSO only, and there this rename
> is already rejected (HDDS-8545).
> The patch does not repair keys renamed before it. For such a leftover open
> key row, the renamed key still references the row's blocks, so any change
> that handles the row in the cleanup scan should skip it rather than delete it.
> With the patch the key rename request and response suites,
> {{TestOpenKeyCleanupService}}, {{TestOzoneManagerRequestHandler}},
> {{TestBucketLayoutAwareOMKeyFactory}}, the OM version and finalization unit
> suites and the OM upgrade integration tests, including
> {{TestOMUpgradeFinalization}}, pass, and checkstyle on {{hadoop-hdds/common}}
> and {{hadoop-ozone/ozone-manager}} is clean.
> Found by TLA+ model checking and code review of the OM open key cleanup and
> hsync lease recovery paths, as part of the TLA+ verification effort under
> HDDS-15926, on commit a6b7bdb937109ba2893688b41a89470bc95c88cf. Checked
> against HDDS issues and apache/ozone pull requests for duplicates before
> filing. Related but not duplicates: HDDS-8545 (added the rejection for FSO),
> HDDS-10631 (proposal to block every open key rename, no OBS/LEGACY or cleanup
> symptom). The attached patch is a proposal for review. Generated with Specula
> (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
