[
https://issues.apache.org/jira/browse/HADOOP-18565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17705314#comment-17705314
]
ASF GitHub Bot commented on HADOOP-18565:
-----------------------------------------
steveloughran commented on PR #5421:
URL: https://github.com/apache/hadoop/pull/5421#issuecomment-1484983979
`Endpoint is not set || Endpoint is set && ends in amazonaws.com || ARN is
set` is roughly what storediag does:
https://github.com/steveloughran/cloudstore/blob/trunk/src/main/java/org/apache/hadoop/fs/store/diag/S3ADiagnosticsInfo.java#L672
it also looks for amazonaws.cn; not sure about where else it should probe.
having a flag "is.aws" would be good as a single switch to move to all 3rd
party stuff, but the problem there is that features there may change over time
too; fs.s3a.store.vendor would let you have a table of providers (aws, ozone,
hitachi, amplidata, minio, netapp) and choose the right settings w.r.t path vs
hostname, checksums, ....
we'd have the vendor settings from properties too, e.g
fs.s3a.vendor.ozone.change.detection.mode = none
so they could be overridden in core-site/per-bucket
that could actually simplify a lot of our internal "doesn't work with vendor
XYZ" where problems 1 and 2 are
1. you forgot to set the endpoint and aws are rejecting you/your bucket
2. use path resolution
> AWS SDK V2 - Complete outstanding items
> ---------------------------------------
>
> Key: HADOOP-18565
> URL: https://issues.apache.org/jira/browse/HADOOP-18565
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.4.0
> Reporter: Ahmar Suhail
> Priority: Major
> Labels: pull-request-available
>
> The following work remains to complete the SDK upgrade work:
> * S3A allows users configure to custom signers, add in support for this.
> * Remove SDK V1 bundle dependency
> * Update `getRegion()` logic to use retries.
> * Add in progress listeners for `S3ABlockOutputStream`
> * Fix any failing tests.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
