Siyao Meng created HDDS-16788:
---------------------------------
Summary: CompleteMultipartUpload terminates the OM when the open
key of an OBJECT_STORE or LEGACY upload is missing
Key: HDDS-16788
URL: https://issues.apache.org/jira/browse/HDDS-16788
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
CR-4-fail-multipart-complete-when-open-key-is-missing.patch,
TestBugCR4CompleteMissingOpenKey.java
h3. Mechanism
{{S3MultipartUploadCompleteRequest.getOmKeyInfo}} reads the open key of the
upload and uses the result without a null check in both of its branches:
{{dbOpenKeyInfo.toBuilder()}} when the key does not exist yet,
{{dbOpenKeyInfo.getFileEncryptionInfo()}} when an existing key is overwritten.
If the upload still has its {{multipartInfoTable}} entry but no open key, the
{{NullPointerException}} passes the {{IOException | InvalidPathException}}
handler in {{validateAndUpdateCache}}, reaches
{{OzoneManagerStateMachine.runCommand}}, and the OM calls
{{ExitUtils.terminate}}.
Other requests already accept this state: {{S3MultipartUploadAbortRequest}}
(since HDDS-14601) and {{S3ExpiredMultipartUploadsAbortRequest}} both continue
when the open key is missing. FSO buckets are not affected, because the FSO
complete request adds a missing open key back before it is read (checked by
unit test).
h3. Trigger
# An upload on an OBJECT_STORE or LEGACY bucket has its {{multipartInfoTable}}
entry but no open key. {{OpenKeyCleanupService}} in 1.3.x deleted the open key
of uploads older than {{ozone.om.open.key.expire.threshold}} (default 7d) and
left the {{multipartInfoTable}} entry. HDDS-9098 stopped that in 1.4.0, but
entries created before survive the upgrade. The HDDS-9095 comments of
2026-02-09 report such entries from a running cluster, which led to HDDS-14601.
# A client sends CompleteMultipartUpload for that upload.
I found no way to create the state with the current code through the client API
or the background services. Since 1.4.0 {{MultipartUploadCleanupService}}
aborts such uploads once they are older than
{{ozone.om.open.mpu.expire.threshold}} (default 30d), so with default settings
the entries disappear about a month after the upgrade. The report above shows
that they can outlive that in practice (cause not investigated).
h3. Impact
The OM terminates while applying the request. The request is a committed Ratis
log entry, so a restarted OM applies it again and terminates again (run on a
single node OM). In an HA ring every OM applies the same entry (argued from the
state machine code, not run). Nothing skips a failing log entry, so the OMs
stay down until an operator intervenes (argued, not run).
h3. Reproduction
PASS, deterministic, unmodified source.
[^TestBugCR4CompleteMissingOpenKey.java] ({{hadoop-ozone/ozone-manager}})
drives one real OM with single node Ratis through {{OzoneManagerProtocol}}. The
only step that is not a client call is the {{DeleteOpenKeys}} request that
removes the open key, which is the request {{OpenKeyCleanupService}} submitted
for such keys before HDDS-9098. The test shows the terminate for OBJECT_STORE
and for LEGACY with a {{NullPointerException}} in
{{S3MultipartUploadCompleteRequest.getOmKeyInfo}}, the second terminate on log
replay after a restart, and the client abort succeeding on the same state.
{{System.exit}} is disabled in the test JVM, so the terminate is observed as
the recorded {{ExitUtils.terminate(1, ...)}} call. A passing test means the
defect is present.
h3. Patch
[^CR-4-fail-multipart-complete-when-open-key-is-missing.patch], against
d067ba38e7aea917db7efd290b4af53987ec985a. It also applies to master at
a6332fb5987.
{{getOmKeyInfo}} reads the open key once, before the two branches, and fails
with {{NO_SUCH_MULTIPART_UPLOAD_ERROR}} when it is missing. The existing
handler returns that as an error response, the OM keeps running, and the
{{multipartInfoTable}} entry stays so the upload can still be aborted. The key
is not rebuilt from the parts as it is for FSO, because for these layouts the
ACLs, metadata, tags and encryption info of the upload are stored only in the
open key. A patched OM that replays an already logged complete for such an
upload returns the error instead of terminating (argued, not run).
Covered by {{testValidateAndUpdateCacheWithOrphanMultipartInfo}} in the
existing {{TestS3MultipartUploadCompleteRequest}}, without and with an already
committed key, inherited by the FSO subclass. Without the change both
OBJECT_STORE cases fail with the {{NullPointerException}}. The multipart
request and response unit suites and the open key and multipart cleanup service
suites (149 tests) pass, checkstyle is clean. No mini cluster suite was run.
With the patch the three complete tests of the reproduction fail, because the
client receives the error and the OM does not terminate.
Found by TLA+ model checking and code review of the OM expired multipart upload
cleanup paths under HDDS-15926, on commit
d067ba38e7aea917db7efd290b4af53987ec985a. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
