fatmanverse opened a new pull request, #11436:
URL: https://github.com/apache/seatunnel/pull/11436
## Purpose of this pull request
Closes #9233
The S3File connector previously hard-coded the
`fs.s3a.aws.credentials.provider` option as an enum
(`S3aAwsCredentialsProvider`) that accepted **only**
`SimpleAWSCredentialsProvider` and `InstanceProfileCredentialsProvider`. Any
other value threw at config-parse time, and the value could not even be
smuggled in via `hadoop_s3_properties` because the connector force-overwrote
the key. This blocked users who need container/role-based providers — e.g.
running SeaTunnel in K8s and authenticating to S3 via
`WebIdentityTokenFileCredentialsProvider` (webIdentityTokenFile + role), which
is exactly the scenario reported in #9233.
This PR replaces the enum with a free-form string option that passes any
fully-qualified S3A credentials provider class through to Hadoop, while keeping
the two well-known values and the existing default.
## Changes
- **`S3FileBaseOptions`**: `S3A_AWS_CREDENTIALS_PROVIDER` changed from
`enumType(S3aAwsCredentialsProvider.class)` to `stringType()`. Default is
unchanged (`com.amazonaws.auth.InstanceProfileCredentialsProvider`). The
`S3aAwsCredentialsProvider` enum is removed; the two well-known class names are
kept as `public static final String` constants
(`SIMPLE_AWS_CREDENTIALS_PROVIDER`, `INSTANCE_PROFILE_CREDENTIALS_PROVIDER`) so
factories and downstream modules keep compiling.
- **`S3HadoopConf`**: passes the raw provider string through to
`fs.s3a.aws.credentials.provider`, and eagerly verifies the class is on the
classpath (`Class.forName`, init=false). The check tries the thread context
classloader first, then falls back to the connector's own classloader, so a
provider bundled with the connector (including the default) is never rejected
by mistake under the Zeta plugin classloader. A missing class produces an
actionable error pointing at the config key and `${SEATUNNEL_HOME}/lib`.
- **Factories** (`S3FileSourceFactory`, `S3FileSinkFactory`,
`S3RedshiftSinkFactory`): reference `SIMPLE_AWS_CREDENTIALS_PROVIDER` in the
`.conditional(...)` rule that requires `access_key`/`secret_key`, instead of
the removed enum value. Behavior is unchanged.
- **Docs** (`docs/{en,zh}/connectors/{source,sink}/S3File.md`): the
`fs.s3a.aws.credentials.provider` row now documents that any classpath-present
S3A provider is accepted, with `ContainerCredentialsProvider` as a container
example.
- **Tests**: `S3HadoopConfTest` adds default / custom-passthrough /
unknown-class-fails cases; `S3FileFactoryTest` asserts the conditional
access_key/secret_key requirement still fires for both source and sink;
`S3StorageTest` (hive) updated to the new constant.
## Backward compatibility
Fully backward compatible. The old enum's `toString()` returned the
fully-qualified class name, so existing configs already used exactly the
strings the new `stringType()` accepts. The default value and the
`access_key`/`secret_key` conditional requirement are unchanged.
## Does this PR introduce any user-facing change?
Yes — `fs.s3a.aws.credentials.provider` now accepts any S3A credentials
provider class available on the classpath (previously only two values). This is
additive; existing configurations continue to work unchanged. Docs (en/zh) are
updated.
## How was this patch tested?
- `./mvnw spotless:apply` (clean)
- Unit tests for `connector-file-s3` (`S3HadoopConfTest`,
`S3FileFactoryTest`) and hive `S3StorageTest` — all passing.
## Check list
- [x] Code changed are covered with tests, or it does not need tests for
reason
- [x] If any new Jar binary package adding in your PR, please add License
Notice — N/A
- [x] If necessary, please update the documentation to describe the new
feature — updated en/zh
- [ ] If you are contributing the connector code, please check that the
following files are updated: N/A (no new connector)
- [x] Update the `release-note`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
