CalvinKirs opened a new pull request, #65834:
URL: https://github.com/apache/doris/pull/65834
### What problem does this PR solve?
Issue Number: close #xxx
Problem Summary:
GCS, MinIO and Ozone currently fall through to the generic S3 filesystem
provider. This PR routes each to its own dedicated filesystem plugin, so their
property spelling, defaults and credential surface are owned and enforced per
dialect instead of leaking into the generic S3 path.
**Approach**
- **Extract `fe-filesystem-s3-base` first**, following the existing
`fe-filesystem-hdfs-base` precedent: a plain library with no provider and no
`META-INF/services`, holding the S3 protocol client (`S3FileSystem`,
`S3ObjStorage`, `S3FileSystemProperties`, credentials-provider factory) that
the dialects reuse. Alongside it live `AbstractDelegatingS3Properties` — the
base for dialects that speak the S3 protocol but differ in property spelling,
defaults and credential surface — and `S3CompatSignals`, which centralizes
dialect detection.
- **One plugin module per dialect** (`fe-filesystem-gcs` / `-minio` /
`-ozone`), matching the module-per-dialect layout already used by `cos`, `obs`
and `oss`: one package, one provider, one `META-INF/services` entry and one
plugin zip each, packaged by `build.sh`. Each owns a typed properties class, so
properties and filesystem stay bound together and the credential surface is
enforced at parse time: all three accept static HMAC `access_key`/`secret_key`
only and reject the AWS-specific mechanisms (role ARN, instance profile) they
cannot serve. GCS keeps HMAC interoperability access; the Google native SDK is
not used because it is OAuth2-only and incompatible with the existing HMAC
parameters.
- **Aliases, defaults and validation follow the legacy fe-core properties
classes**, including the anonymous-credentials fallback that every
S3-compatible dialect applies when no access/secret key is configured. The
dialects differ in supported URI schemes (GCS `gs`/`s3`/`s3a`, MinIO
`s3`/`s3a`, Ozone `s3`/`s3a`/`s3n`), so `S3ObjStorage` gains a constructor
taking an explicit scheme set — delegating through `S3FileSystemProperties`
would otherwise replace the dialect scheme set with the S3 defaults and reject
the dialect's own URIs.
- **The generic S3 provider yields, decided from the property map alone.**
S3 claims any map carrying credentials plus an endpoint or region — which now
includes every dialect map — and selection cannot fall back on
`META-INF/services` order because the dialects live in separate jars where
`ServiceLoader` order is not guaranteed. So S3 yields when the map either names
a dedicated dialect (`provider=...`, `fs.<dialect>.support=true`) or is
recognized by that dialect's guess predicate, unless the user explicitly asked
for plain S3 (the escape hatch for serving a dialect endpoint through the
generic client). The predicates are ports of the legacy fe-core `guessIsMe`
methods, so detection matches what the kernel already does when it picks a
`StorageProperties` subclass. Ozone has no legacy guess and is therefore
reachable only by explicit request; an unflagged Ozone gateway map is served by
the generic S3 provider as before.
**fe-core is untouched** — the kernel needs no knowledge of the individual
dialects (net diff against fe-core is zero).
### Release note
Add GCS, MinIO and Ozone as dedicated filesystem plugins built on a shared
S3-protocol base module.
### Check List (For Author)
- Test
- [x] Unit Test
- [ ] Regression test
- [ ] Manual test (add detailed scripts or steps below)
- [ ] No need to test or manual test.
- Behavior changed:
- [x] No. <!-- GCS/MinIO/Ozone routing and property semantics follow the
existing legacy behavior; the change is where that behavior lives, not what it
does. -->
- Does this need documentation?
- [x] No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
