kou commented on issue #36770:
URL: https://github.com/apache/arrow/issues/36770#issuecomment-1642755528
> That being said, it should be possible for us to provide support for this,
even if the SDK does not, in case someone wanted to create a PR.
This may work:
```diff
diff --git a/cpp/src/arrow/filesystem/s3fs.cc
b/cpp/src/arrow/filesystem/s3fs.cc
index c57fc7f29..b0c2d973e 100644
--- a/cpp/src/arrow/filesystem/s3fs.cc
+++ b/cpp/src/arrow/filesystem/s3fs.cc
@@ -339,6 +339,7 @@ Result<S3Options> S3Options::FromUri(const Uri& uri,
std::string* out_path) {
}
bool region_set = false;
+ bool endpoint_override_set = false;
for (const auto& kv : options_map) {
if (kv.first == "region") {
options.region = kv.second;
@@ -347,6 +348,7 @@ Result<S3Options> S3Options::FromUri(const Uri& uri,
std::string* out_path) {
options.scheme = kv.second;
} else if (kv.first == "endpoint_override") {
options.endpoint_override = kv.second;
+ endpoint_override_set = true;
} else if (kv.first == "allow_bucket_creation") {
ARROW_ASSIGN_OR_RAISE(options.allow_bucket_creation,
::arrow::internal::ParseBoolean(kv.second));
@@ -357,6 +359,12 @@ Result<S3Options> S3Options::FromUri(const Uri& uri,
std::string* out_path) {
return Status::Invalid("Unexpected query parameter in S3 URI: '",
kv.first, "'");
}
}
+ if (!endpoint_override_set) {
+ auto endpoint = std::getenv("AWS_ENDPOINT_URL");
+ if (!endpoint) {
+ options.endpoint_override = endpoint;
+ }
+ }
if (!region_set && !bucket.empty() && options.endpoint_override.empty()) {
// XXX Should we use a dedicated resolver with the given credentials?
```
BTW, the following will work with the current implementation:
```python
file =
pq.ParquetFile(f's3://mybucket/my_file.parquet?endpoint_override={os.environ["AWS_ENDPOINT_URL"]}')
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
