kylebarron opened a new issue, #6716:
URL: https://github.com/apache/arrow-rs/issues/6716
**Is your feature request related to a problem or challenge? Please describe
what you are trying to do.**
I'm trying to work with AWS requester pays buckets, but getting an error:
```
GenericError: Generic S3 error: Error performing list request: Client error
with status 403 Forbidden:
```
```xml
<?xml version="1.0" encoding="UTF-8"?>
<Error>
<Code>AccessDenied</Code>
<Message>There were headers present in the request which were not
signed</Message>
<HeadersNotSigned>x-amz-request-payer</HeadersNotSigned>
<RequestId>QEA9RD5BETE6BN7E</RequestId>
<HostId>tfgEYvVBmWEx5EZljTRXuR6S6ocChjas5qF8eXThc0yt6bRyGEJCYjCFJ+/YOvA2Q2DT/NSqBTw=</HostId>
</Error>
```
The following is a Python example from
[`obstore`](https://github.com/developmentseed/obstore). It should be easy to
interpret what the underlying Rust code is, but I can provide a pure-Rust
example if necessary
```py
import obstore
from obstore.store import ClientOptions, S3Store
import boto3
# boto3.Session reads credentials from disk, which then passes them to
AmazonS3Builder
session = boto3.Session()
# Set x-amz-request-payer as a default header for every request
options = ClientOptions(default_headers={b"x-amz-request-payer":
"requester"})
# This is a bucket exposed through the AWS open data registry
# https://registry.opendata.aws/naip/
# It's public with requester-pays turned on
store = S3Store.from_session(session, bucket="naip-visualization",
client_options=options)
# Make the request
result = obstore.list(store, "ny/2022/60cm/rgb/40073/").collect()
```
**Describe the solution you'd like**
Include AWS header in signature calculation.
**Describe alternatives you've considered**
<!--
A clear and concise description of any alternative solutions or features
you've considered.
-->
**Additional context**
[This AWS docs
page](https://docs.aws.amazon.com/AmazonS3/latest/userguide/ObjectsinRequesterPaysBuckets.html)
has a reference on requester pays and request signing
<img width="776" alt="image"
src="https://github.com/user-attachments/assets/813515bd-5bc5-415e-8ec1-cd8c798838c1";>
Presumably there would need to be some way to tell `object_store` which
default headers to include in its signing mechanism?
The only prior discussion I can find is [on
discord](https://discord.com/channels/885562378132000778/885562378132000781/1262478057692729404):
> Hi everyone, if anyone is familiar with the object_store implementation in
arrow-rs I had a question; is there a way to pass in arbitrary AWS
(specifically S3) options via HTTP headers? For example, I don't see implicit
support for requester pays buckets, which is hinted at by passing in this
header value pair: `x-amz-request-payer : requester` so I'm wondering if
there's a way to add in our own options like that.
> Not on a per-request basis, as it sort of breaks the abstraction of being
any object store, but you could look at overriding the headers using
ClientOptions when you build the AmazonS3 client
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
