[
https://issues.apache.org/jira/browse/HADOOP-18683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17913731#comment-17913731
]
Steve Loughran edited comment on HADOOP-18683 at 1/16/25 1:23 PM:
------------------------------------------------------------------
thinking more. this is actually fairly straightforward as this is all static
config (well, ignoring env. and sysprops) -no need to resolve across buckets,
wait for one to come up first, etc etc.And there's no support for actually
removing an option.
Every bucket just needs an "extends" attribute. This can nominate another
bucket whose options it extends. vendor and region settings just become buckets
in the chain.
{code}
# region settings
fs.s3a.bucket.aws-london.region=us-west-2
# buckets under the region
fs.s3a.bucket.stevel-london.extends=aws.london
fs.s3a.bucket.stevel-london-2.extends=aws.london
# extending another of my buckets
fs.s3a.bucket.stevel-london-3.extends=tevel-london-2
# new region extension with s3express options
fs.s3a.bucket.aws-london-s3express.extends aws-london
fs.s3a.bucket.aws-london-s3express.performance.flags=create, mkdir
fs.s3a.bucket.stevel--eu2-az2--x-s3.extends=aws-london-s3express
# google gcs
fs.s3a.bucket.gcs.region=google
fs.s3a.bucket.gcs.path.style.access=true
fs.s3a.bucket.gcs.endpoint=s3.googlegcs.com
fs.s3a.bucket.stevel-gcs.extends = gcs
{code}
How options for a bucket are resolved
* entire chain is built up from target bucket backwards
* any loops trigger a failure
* then from lowest element in chain, the configuration is patched (excluding
".extends")
* this becomes the final resolved config
propagateOptions() can do this, just should be moved out of S3AUtils, given its
own tests etc. The S3AUtils entry point MUST be retained...some apps
(storediag) use it.
harder: test removeBucketAndBaseOptions in tests, it'd need to walk the chain
too. Proposed, the propagateOptions() method to build that list of resolved
options is exported so that remove() can go through the entire chain unsetting
it.
Not hard at all, and be really useful for common options like fips, s3express,
other vendors.
thoughts?
was (Author: [email protected]):
thinking more. this is actually fairly straightforward as this is all static
config (well, ignoring env. and sysprops) -no need to resolve across buckets,
wait for one to come up first, etc etc.And there's no support for actually
removing an option.
Every bucket just needs an "extends" attribute. This can nominate another
bucket whose options it extends. vendor and region settings just become buckets
in the chain.
# region settings
fs.s3a.bucket.aws-london.region=us-west-2
# buckets under the region
fs.s3a.bucket.stevel-london.extends=aws.london
fs.s3a.bucket.stevel-london-2.extends=aws.london
# extending another of my buckets
fs.s3a.bucket.stevel-london-3.extends=tevel-london-2
# new region extension with s3express options
fs.s3a.bucket.aws-london-s3express.extends aws-london
fs.s3a.bucket.aws-london-s3express.performance.flags=create, mkdir
fs.s3a.bucket.stevel--eu2-az2--x-s3.extends=aws-london-s3express
# google gcs
fs.s3a.bucket.gcs.region=google
fs.s3a.bucket.gcs.path.style.access=true
fs.s3a.bucket.gcs.endpoint=s3.googlegcs.com
fs.s3a.bucket.stevel-gcs.extends = gcs
How options for a bucket are resolved
* entire chain is built up from target bucket backwards
* any loops trigger a failure
* then from lowest element in chain, the configuration is patched (excluding
".extends")
* this becomes the final resolved config
propagateOptions() can do this, just should be moved out of S3AUtils, given its
own tests etc. The S3AUtils entry point MUST be retained...some apps
(storediag) use it.
harder: test removeBucketAndBaseOptions in tests, it'd need to walk the chain
too. Proposed, the propagateOptions() method to build that list of resolved
options is exported so that remove() can go through the entire chain unsetting
it.
Not hard at all, and be really useful for common options like fips, s3express,
other vendors.
thoughts?
> S3A Add new store vendor config option
> --------------------------------------
>
> Key: HADOOP-18683
> URL: https://issues.apache.org/jira/browse/HADOOP-18683
> Project: Hadoop Common
> Issue Type: Sub-task
> Components: fs/s3
> Affects Versions: 3.3.5
> Reporter: Ahmar Suhail
> Priority: Minor
>
> Add in a new fs.s3a.store.vendor config, where users can specify the storage
> vendor they are using (eg: aws, netapp, minio).
> This will allow us to configure S3A correctly per vendor. For example, if the
> vendor is not AWS, you probably want to use ListObjectsV1.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
