Siyao Meng created HDDS-17251:
---------------------------------
Summary: ozone admin om list-open-files fails with
NumberFormatException when an open multipart upload is in the listed range
Key: HDDS-17251
URL: https://issues.apache.org/jira/browse/HDDS-17251
Project: Apache Ozone
Issue Type: Bug
Reporter: Siyao Meng
Attachments:
CR-15-list-open-multipart-uploads-in-list-open-files.patch,
TestBugCR15ListOpenFilesMultipartUpload.java
h3. Mechanism
{{OmMetadataManagerImpl.listOpenFiles}} takes the client ID of every listed row
from its open key table key with
{{OMMetadataManager.getClientIDFromOpenKeyDBKey}}, which runs
{{Long.parseLong}} on the last path segment. The open key that
{{S3InitiateMultipartUploadRequest}} writes for an upload ends with the upload
ID instead ({{UUIDv7 + "-" + UniqueId}}, see
{{OMMultipartUploadUtils.getMultipartUploadId}}), so the parse throws
{{NumberFormatException}}. The exception is not an {{IOException}}, so
{{OzoneManagerRequestHandler.handleReadRequest}} does not turn it into an error
response. It reaches the client as
{{RemoteException(java.lang.NumberFormatException)}}, and the whole listing
fails, including the open keys before the upload.
h3. Trigger
# InitiateMultipartUpload in any bucket (FSO, OBJECT_STORE or LEGACY).
# While the upload is open, list the open files of that bucket ({{ozone admin
om lof -p /vol/bucket}}), or of {{/}} for an FSO bucket. Any page whose range
contains the open key of the upload fails.
No fault or race is needed. Every open S3 multipart upload has this row until
it is completed or aborted.
h3. Impact
* The ListOpenFiles request, and with it {{ozone admin om list-open-files}},
cannot list a bucket that has an open multipart upload, so an operator cannot
see its open or hsync'ed files.
* The client retries this error, so the command does not fail promptly. With
default client settings on a single OM it had not returned after 5 minutes, and
the OM logs a stack trace for every attempt.
* No state is changed. The request is read only.
h3. Reproduction
PASS, deterministic, unmodified source.
[^TestBugCR15ListOpenFilesMultipartUpload.java]
({{hadoop-ozone/ozone-manager}}) starts one OM without datanodes and, for each
bucket layout, opens one plain key and initiates one upload through
{{OzoneManagerProtocol}}, then calls {{listOpenFiles}} on the bucket. The
client retry settings are lowered so that the error reaches the caller. All
three layouts fail with {{RemoteException(java.lang.NumberFormatException): For
input string: "<upload ID>"}}, and the OM rejected both attempts. A passing
test means the defect is present.
h3. Patch
[^CR-15-list-open-multipart-uploads-in-list-open-files.patch], against
20d95ed37fe35013b5320713665db03c06ee36fa. It also applies to master at
64d4a791502.
{{listOpenFiles}} now recognises the open key of an upload by its key, with the
existing {{OMMultipartUploadUtils.getUploadIdFromDbKey}}, and lists it with
client ID 0 instead of parsing the upload ID, so {{ozone admin om lof}} shows
it with client ID 0, which no real client has (client IDs come from
{{UniqueId.next()}}). The key based check is used rather than the
{{isMultipartKey}} flag because the open keys of parts being written also carry
the flag and do end with a real client ID, and because upload rows written
before HDDS-9017 do not carry the flag. Listing the row, rather than skipping
it, keeps the listing consistent with {{totalOpenKeyCount}}, which already
counts it, and leaves the pagination unchanged. Other rows are listed as before.
Covered by {{testListOpenFilesWithMultipartUpload}} in the existing
{{TestOmMetadataManager}}, for every bucket layout, with and without a
continuation token. It lists a plain open key, an upload row without the
{{isMultipartKey}} flag (as written before HDDS-9017) and the open key of a
part being written, which has the flag and a real client ID. Without the change
it fails with {{NumberFormatException}} for all layouts, and so would a check
based on the flag. With it {{TestOmMetadataManager}} (33 tests) and the list
open files tests of {{TestOzoneShellHA}} (2 tests, mini cluster) pass, and
checkstyle is clean. With the patch the reproduction fails at {{assertThrows}},
because the listing succeeds.
Found during review of a fix for the OM multipart upload complete path, as part
of the TLA+ verification effort under HDDS-15926, on commit
20d95ed37fe35013b5320713665db03c06ee36fa. Checked against HDDS issues and
apache/ozone pull requests for duplicates before filing. The attached patch is
a proposal for review. Generated with Specula (Claude Opus 5.5).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
