vyasr opened a new issue, #38364: URL: https://github.com/apache/arrow/issues/38364
### Describe the enhancement requested Currently [pyarrow initializes the s3 filesystem when pyarrow.fs is imported](https://github.com/apache/arrow/blob/d68f8e21643e8aec9ed253094edfd15c0a08f1c1/python/pyarrow/fs.py#L53-L63). This leads to AWS consuming resources on startup that may never be used if the user is not actually taking advantage of that support. Ideally the s3fs would instead be delayed to first use to avoid AWS spinning up unnecessary threads/doing work on pyarrow import. Making this change would also allow sidestepping [a bug present in newer versions of the aws-sdk-cpp that occasionally leads to segfaults simply by using the AWS APIs](https://github.com/aws/aws-sdk-cpp/issues/2681), at least for the majority of users who are not using the s3fs by default. ### Component(s) C++, Python -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
