furqan-khan07 opened a new pull request, #55643:
URL: https://github.com/apache/arrow/pull/55643
### Rationale for this change
`AWS_ENDPOINT_URL_S3` and `AWS_ENDPOINT_URL` are only read in
`S3Options::FromUriAndOptions`. Options created with `S3Options::Defaults()`,
`Anonymous()`, `FromAccessKey()` or `FromAssumeRole()` ignore them.
`pyarrow.fs.S3FileSystem()` uses those factories, so it connects to AWS even
when the variables point to MinIO or another S3-compatible endpoint, while
`FileSystem.from_uri("s3://...")` in the same process uses the endpoint.
### What changes are included in this PR?
- Move the existing environment lookup from `S3Options::FromUriAndOptions`
to the `S3Options` constructor, so all ways of creating `S3Options` get the
same default `endpoint_override`. An explicitly set `endpoint_override` (C++
field, URI query parameter, Python keyword) still takes precedence.
- The default region resolver clears `endpoint_override`, so
`ResolveS3BucketRegion` keeps querying AWS when the variables are set.
- Mention the environment variables in the `endpoint_override` docs (C++
header, Python docstring, `env_vars.rst`).
### Are these changes tested?
Yes.
- C++: `S3OptionsTest.EndpointFromEnvVar` and
`S3RegionResolutionTest.EndpointEnvVarIgnored`.
- Python: `test_s3fs_endpoint_from_env`, which runs against the MinIO test
server and checks `AWS_ENDPOINT_URL`, the precedence of `AWS_ENDPOINT_URL_S3`,
and the precedence of an explicit `endpoint_override`.
Locally (macOS arm64, debug build, aws-sdk-cpp 1.11.905): `arrow-s3fs-test`,
`arrow-s3fs-module-test`, `arrow-filesystem-test` and `pyarrow/tests/test_fs.py
--enable-s3` pass. The new `EndpointFromEnvVar` and Python tests fail without
the change.
### Are there any user-facing changes?
Yes. When `AWS_ENDPOINT_URL_S3` or `AWS_ENDPOINT_URL` is set,
`pyarrow.fs.S3FileSystem(...)` and C++ `S3Options::Defaults()` / `Anonymous()`
/ `FromAccessKey()` / `FromAssumeRole()` now use that endpoint unless
`endpoint_override` is set explicitly. Previously only filesystems created from
a URI did. Passing `endpoint_override=""` in Python keeps AWS as default.
### Was AI used for this PR?
**PR code and description written by:**
- [x] Human
- [x] AI
**Reviewed before submission by:**
- [x] Human
- [x] AI
- [ ] Not reviewed
Claude (Fable 5.2) was used to investigate the issue and write the fix,
tests and this description, and Claude (Opus 5.5) re-checked it. I reviewed the
changes, and the tests and the MinIO reproduction were built and run locally.
* GitHub Issue: #49643
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]