Meng-Shuan Tsai created HDDS-16604:
--------------------------------------
Summary: S3 multi-delete returns HTTP 200 when OM rejects the
whole DeleteKeys request
Key: HDDS-16604
URL: https://issues.apache.org/jira/browse/HDDS-16604
Project: Apache Ozone
Issue Type: Bug
Reporter: Meng-Shuan Tsai
Assignee: Meng-Shuan Tsai
This is a bug. S3 {{DeleteObjects}} ({{{}POST /\{bucket}?delete{}}}) can return
HTTP 200 with every key listed under {{<Deleted>}} although Ozone Manager
rejected the whole request and deleted nothing. Only OM's audit log and metrics
record the failure.
{{BucketEndpoint.multiDelete}} is the only production caller of
{{{}OzoneBucket.deleteKeys(keys, quiet=true){}}}. In
{{OzoneManagerProtocolClientSideTranslatorPB.deleteKeys(OmDeleteKeys, boolean
quiet)}} the {{quiet}} branch copies {{DeleteKeysResponse.errors}} into the
returned map and never looks at {{{}OMResponse.status{}}}. The non-quiet branch
calls {{handleError(omResponse)}} like every other client method.
h5. *Root cause*
{{quiet}} exists so that a partial failure ({{{}PARTIAL_DELETE{}}}) does not
throw and the gateway can emit per-key {{<Error>}} elements. But when OM fails
the whole request ({{{}BUCKET_NOT_FOUND{}}}, {{{}VOLUME_NOT_FOUND{}}},
{{{}PERMISSION_DENIED{}}}, {{{}INTERNAL_ERROR{}}}, ...), the catch block in
{{OMKeysDeleteRequest.validateAndUpdateCache}} builds a {{DeleteKeysResponse}}
with {{status=false}} and {{unDeletedKeys}} but no {{{}errors{}}}. If the
failure happens in {{{}preExecute{}}}, the response has no
{{DeleteKeysResponse}} at all. Either way the client sees an empty map, which
the gateway treats as "all keys deleted": HTTP 200, {{<Deleted>}} for every key
(or an empty body with {{{}Quiet=true{}}}), success metrics and a success audit
entry. The whole-request status is silently dropped.
OzoneFS, shell and Freon use the non-quiet overload and are not affected. No
key is deleted by mistake; only the response is wrong.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
