hengfeiyang opened a new issue, #731: URL: https://github.com/apache/arrow-rs-object-store/issues/731
### Describe the bug [#549](https://github.com/apache/arrow-rs-object-store/pull/549) refactored `AmazonS3::delete()` so that it now delegates to `delete_stream()`, which issues a `POST /?delete` (S3 `DeleteObjects` / bulk delete) request instead of the previous single-object `DELETE /key`. While the S3 *core* protocol is widely implemented by third-party providers, the `DeleteObjects` multi-object API is **not** universally supported. As a result, any user of `object_store` 0.x against a non-AWS S3-compatible endpoint that doesn't implement `POST /?delete` can no longer delete objects at all — even single-object deletes — because the only public API (`delete`) now routes through bulk delete. Concretely, this breaks: - **Alibaba Cloud OSS** — its S3-compatible endpoint does not implement `DeleteObjects`. Calls fail with an XML error (`NotImplemented` / `MethodNotAllowed`, depending on region). - Likely affected: older MinIO versions, Cloudflare R2 in some configurations, Tencent COS legacy endpoints, and various on-prem S3 gateways. (Can collect more reports if helpful.) Before #549, `delete()` issued a plain `DELETE /key`, which is part of the core S3 API and is supported by essentially every S3-compatible provider. After #549, there is no way for downstream users to opt out. ### To Reproduce Using `object_store` configured with an Alibaba Cloud OSS S3-compatible endpoint: ```rust let store = AmazonS3Builder::new() .with_endpoint("https://oss-<region>.aliyuncs.com") .with_bucket_name("my-bucket") .with_access_key_id(...) .with_secret_access_key(...) .build()?; store.put(&path, bytes).await?; // OK store.delete(&path).await?; // FAILS: NotImplemented / MethodNotAllowed ``` The same call succeeded on the previous release that used single-object `DELETE`. ### Expected behavior `AmazonS3::delete(path)` should issue a single-object `DELETE /key` request, which is part of the core S3 API and works against every S3-compatible provider. Bulk delete (`POST /?delete`, the `DeleteObjects` API) is an *optimization* and should only be used by `delete_stream` / batched APIs — not by the per-object `delete()`. This also mirrors the AWS SDKs, which expose `DeleteObject` and `DeleteObjects` as distinct operations precisely because the latter is optional / extended. ### Suggested fix Two options, not mutually exclusive: 1. **Revert `delete()` to single-object DELETE** (preferred). `delete_stream` continues to use `DeleteObjects` for throughput; `delete()` stays compatible with every S3 provider. This is the lowest-risk change and matches the contract users had pre-#549. 2. **Add an opt-out config**, e.g. `S3ConfigKey::DisableBulkDelete` (or a positive `with_bulk_delete(false)` builder method), so users targeting incompatible providers can force single-object DELETEs throughout. Useful even with option 1, for `delete_stream` callers on providers that don't support `DeleteObjects`. Happy to send a PR implementing either or both — would appreciate guidance on which direction maintainers prefer. ### Environment - `object_store` version: 0.13.x - Provider: Alibaba Cloud OSS (S3-compatible endpoint) ### Additional context - Original PR: [#549](https://github.com/apache/arrow-rs-object-store/pull/549) - AWS API reference: [[DeleteObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObject.html)](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObject.html) vs [[DeleteObjects](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html)](https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html) — distinct operations in the upstream API. --- The error like this: ``` 2026-05-22T02:00:03.274139257+00:00 ERROR infra::storage: Failed to delete object: Generic S3 error: Error performing POST http://openobserve.oss-cn-beijing-internal.aliyuncs.com/openobserve?delete in 9.033372ms - Server returned non-2xx status code: 405 Method Not Allowed: <?xml version="1.0" encoding="UTF-8"?> <Error> <Code>MethodNotAllowed</Code> <Message>The specified method is not allowed against this resource.</Message> <RequestId>6A0FB8A330259633318A4958</RequestId> <HostId>openobserve.oss-cn-beijing-internal.aliyuncs.com</HostId> <Method>POST</Method> <ResourceType>Object</ResourceType> <EC>0017-00000001</EC> <RecommendDoc>https://api.aliyun.com/troubleshoot?q=0017-00000001</RecommendDoc> </Error> ``` @crepererum @alamb -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
