mtofano commented on issue #45305:
URL: https://github.com/apache/arrow/issues/45305#issuecomment-2675298618
Hey all,
Thanks for looking in to this issue. I am still running in to a similar
issue when using pyarrow datasets:
```py
ds.write_dataset(
data=in_dataset.scanner(),
base_dir=out_fs_path.path,
filesystem=out_fs, # This is an instance of pyarrow.fs.S3FileSystem
format="parquet"
)
```
The error I get is as follows:
```sh
OSError: When uploading part for key
'security_info/staging/trade_date=2025-02-19/part-0.parquet' in bucket
't-qe-pigs': AWS Error UNKNOWN (HTTP status 501) during UploadPart operation:
Unable to parse ExceptionName: NotImplemented Message: A header you provided
implies functionality that is not implemented.
```
I had a similar issue using `s3fs` directly and the solution was to max
version `botocore<1.36` as this change does.
Would be much appreciated if someone can look into this please!
Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
