Kotwic4 opened a new issue, #36770: URL: https://github.com/apache/arrow/issues/36770
### Describe the enhancement requested AWS_ENDPOINT_URL is now supported by the AWS for custom url (for example localhost). More info about it [docs](https://docs.aws.amazon.com/sdkref/latest/guide/feature-ss-endpoints.html) and [original github issue](https://github.com/aws/aws-cli/issues/1270). It was merged into botocore in [this pr](https://github.com/boto/botocore/pull/2973). What I can do with boto: ```python import os import boto3 os.environ["AWS_ACCESS_KEY_ID"] = "ACCESS_KEY" os.environ["AWS_SECRET_ACCESS_KEY"] = "SECRET_KEY" os.environ["AWS_ENDPOINT_URL"] = "http://localhost:9000"; session = boto3.session.Session() s3_client = session.client( service_name="s3", ) print(s3_client.list_buckets()["Buckets"]) ``` What I have to do in pyarrow: ```python import os from pyarrow import fs os.environ["AWS_ACCESS_KEY_ID"] = "ACCESS_KEY" os.environ["AWS_SECRET_ACCESS_KEY"] = "SECRET_KEY" os.environ["AWS_ENDPOINT_URL"] = "http://localhost:9000"; s3 = fs.S3FileSystem(endpoint_override=os.environ["AWS_ENDPOINT_URL"]) print(s3.get_file_info(fs.FileSelector("", recursive=False))) ``` What I would like to do in pyarrow: ```python import os from pyarrow import fs os.environ["AWS_ACCESS_KEY_ID"] = "ACCESS_KEY" os.environ["AWS_SECRET_ACCESS_KEY"] = "SECRET_KEY" os.environ["AWS_ENDPOINT_URL"] = "http://localhost:9000"; s3 = fs.S3FileSystem() print(s3.get_file_info(fs.FileSelector("", recursive=False))) ``` This will allow me to use `s3://` instead of creating file system ```python #current way s3 = fs.S3FileSystem(endpoint_override=endpoint_url) file = pq.ParquetFile('mybucket/my_file.parquet', filesystem=s3) #possible future file = pq.ParquetFile('s3://mybucket/my_file.parquet') ``` ### Component(s) Python -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
