nzw921rx opened a new pull request, #12885:
URL: https://github.com/apache/seatunnel/pull/12885
## Purpose
Object storage credentials currently often need to be written into
`seatunnel.yaml` as plain text. This change documents and verifies the
Flink-style approach of selecting the existing cloud SDK environment-variable
credential providers for checkpoint and IMap storage.
No production Java change is required because SeaTunnel already passes
arbitrary `fs.*` properties through to Hadoop. This PR formalizes and tests the
provider contract, aligns storage-module dependencies with the distribution,
and documents Kubernetes Secret injection.
## Credential contract
| Storage | Provider configuration | Provider class | Environment variables |
| --- | --- | --- | --- |
| S3 | `fs.s3a.aws.credentials.provider` |
`com.amazonaws.auth.EnvironmentVariableCredentialsProvider` |
`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, optional `AWS_SESSION_TOKEN` |
| OSS | `fs.oss.credentials.provider` |
`com.aliyun.oss.common.auth.EnvironmentVariableCredentialsProvider` |
`OSS_ACCESS_KEY_ID`, `OSS_ACCESS_KEY_SECRET`, optional `OSS_SESSION_TOKEN` |
| COS | `fs.cosn.credentials.provider` |
`org.apache.hadoop.fs.cosn.auth.EnvironmentVariableCredentialsProvider` |
`COSN_SECRET_ID`, `COSN_SECRET_KEY` |
| GCS | Application Default Credentials | N/A | Optional
`GOOGLE_APPLICATION_CREDENTIALS` |
Existing static credential and cloud-native identity configurations remain
supported.
## Changes
- Add checkpoint-storage unit coverage for S3, OSS, and COS environment
credential providers.
- Add IMap-storage unit coverage for S3 and OSS environment credential
providers.
- Set temporary process environment variables and verify that the actual SDK
providers receive the expected access key, secret key, and session token values.
- Document environment-variable credentials for checkpoint and IMap storage
in English and Chinese.
- Document Kubernetes Secret injection through `envFrom.secretRef`.
- Document explicit `secretKeyRef` mapping when Secret keys use different
names.
- Document that `envFrom.prefix` must not be used because the SDK expects
the exact variable names.
- Document that credentials must be injected into every Master and Worker
process that accesses the storage.
- Document that Pods must be restarted or rolled out after Secret rotation.
- Align `hadoop-aliyun` to `3.1.4` and `aliyun-sdk-oss` to `3.4.1`, matching
`seatunnel-dist`.
- Keep `hadoop-aws` on the inherited `3.1.4` property, `aws-java-sdk-bundle`
on `1.11.271`, and `hadoop-cos` on `3.4.1`.
- Declare Hadoop and cloud-client dependencies with `provided` scope in the
storage modules.
The storage plugin jars therefore do not bundle AWS, Alibaba Cloud, Tencent
Cloud, or Hadoop filesystem implementations. The distribution or an explicitly
installed `lib` dependency remains responsible for providing them at runtime.
## Configuration examples
### S3
```yaml
storage.type: s3
s3.bucket: s3a://your-bucket
fs.s3a.aws.credentials.provider:
com.amazonaws.auth.EnvironmentVariableCredentialsProvider
```
### OSS
```yaml
storage.type: oss
oss.bucket: oss://your-bucket
fs.oss.endpoint: oss-cn-hangzhou.aliyuncs.com
fs.oss.credentials.provider:
com.aliyun.oss.common.auth.EnvironmentVariableCredentialsProvider
```
### COS
```yaml
storage.type: cos
cos.bucket: cosn://your-bucket
fs.cosn.bucket.region: your-region
fs.cosn.credentials.provider:
org.apache.hadoop.fs.cosn.auth.EnvironmentVariableCredentialsProvider
```
## Kubernetes injection
When the Secret keys already use the fixed provider variable names:
```yaml
envFrom:
- secretRef:
name: seatunnel-oss-credentials
```
When the Secret uses different key names:
```yaml
env:
- name: OSS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: seatunnel-oss-credentials
key: access-key
- name: OSS_ACCESS_KEY_SECRET
valueFrom:
secretKeyRef:
name: seatunnel-oss-credentials
key: secret-key
```
Do not configure `envFrom.prefix`.
## Compatibility
- No configuration key, default, public API, or `Option` is removed or
renamed.
- Environment-variable credentials are opt-in through the existing `fs.*`
provider property.
- Existing static credentials and cloud-native identity configurations
continue to work.
- No parent POM, core, client, server, or distribution POM change is
introduced.
- No production connector or engine Java source is changed.
- Dependency versions and scopes are aligned with the existing distribution
rather than introducing a broader Hadoop or AWS SDK upgrade.
## Verification
Storage tests:
```bash
./mvnw -pl \
seatunnel-engine/seatunnel-engine-storage/checkpoint-storage-plugins/checkpoint-storage-hdfs,\
seatunnel-engine/seatunnel-engine-storage/imap-storage-plugins/imap-storage-file
\
-DskipIT=true test
```
Results:
- Checkpoint storage: 18 tests, 0 failures, 0 errors, 3 skipped real-cloud
tests.
- IMap storage: 25 tests, 0 failures, 0 errors, 1 skipped real-OSS test.
- `BUILD SUCCESS`.
OSS/COS dependency-chain packaging:
```bash
./mvnw -pl \
seatunnel-connectors-v2/connector-file/connector-file-cos,\
seatunnel-connectors-v2/connector-file/connector-file-oss \
-am -DskipTests -DskipIT=true -Dskip.spotless=true package
```
- `BUILD SUCCESS`.
- Both OSS and COS connector modules packaged successfully.
Storage plugin packaging and jar inspection:
- Both storage modules packaged successfully.
- Checkpoint and IMap jars contain no bundled `com/amazonaws`, `com/aliyun`,
`com/qcloud`, `org/apache/hadoop/fs/s3a`, `org/apache/hadoop/fs/aliyun`, or
`org/apache/hadoop/fs/cosn` classes.
- This confirms the `provided` dependency isolation.
Distribution effective POM:
- `org.apache.hadoop:hadoop-aliyun:3.1.4:provided`
- `com.amazonaws:aws-java-sdk-bundle:1.11.271:provided`
- `com.aliyun.oss:aliyun-sdk-oss:3.4.1:provided`
Additional checks:
- `git diff --check`: passed.
- `git diff --cached --check`: passed.
- Maven `verify` was not run because the local execution policy explicitly
forbids Maven goals containing `verify`.
- Real S3, OSS, and COS network E2E tests were not run because external
credentials and buckets were unavailable. The unit tests use real process
environment variables and resolve the actual SDK provider objects.
- COS runtime still requires `hadoop-cos` and `cos_api-bundle` under the
SeaTunnel runtime `lib` directory.
## Duplicate search
- `#10858` covers S3 IRSA, not fixed environment-variable credentials for
S3, OSS, and COS.
- `#10309` proposes generic external key-vault configuration resolution;
this PR is a narrower filesystem-native alternative.
- `#11648` was a broader unmerged Hadoop/AWS SDK upgrade, not an isolated
environment-credential change.
## Checklist
- [x] Added unit tests for the changed behavior.
- [x] Updated English and Chinese documentation.
- [x] Verified provider pass-through and actual SDK environment-value
resolution.
- [x] Verified `provided` dependencies are not packaged into storage plugin
jars.
- [x] Verified storage module tests and OSS/COS dependency-chain packaging.
- [x] Kept the change scoped to two storage modules and documentation.
- [x] No incompatible configuration change is expected.
- [x] No new runtime jar is added to the distribution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
