lloyd-EA opened a new issue, #38492:
URL: https://github.com/apache/arrow/issues/38492
### Describe the usage question you have. Please include as many useful
details as possible.
I'm trying to use Pyarrow's native S3FileSystem to read parquet files'
metadata that are stored in a Requester Pays enabled bucket. e.g.
```
import pyarrow.fs as fs
import pyarrow.parquet as pq
uri = "s3://requester-pays-bucket/folder-1/file1.parquet"
s3_fs, s3_path = fs.S3FileSystem.from_uri(uri=object_uri)
pq_file_metadata = pq.ParquetFile(s3_path, filesystem=s3_fs).metadata
print(pq_file_metadata)
````
This above gives an error:
OSError: When reading information for key 'folder-1/file1.parquet' in bucket
'requester-pays-bucket': AWS Error ACCESS_DENIED during HeadObject operation:
No response body.
Are there ways to intervene or add s3 request header options like below, or
other ways to achieve this using pyarrow?
`s3_fs, s3_path = fs.S3FileSystem.from_uri(uri=object_uri,
header_options={'x-amz-request-payer': 'requester'})`
I would like to avoid installing other third party libraries like `s3fs` due
to constraints on my application size.
Thanks!
### Component(s)
Parquet, Python
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
