shunping opened a new pull request, #40529:
URL: https://github.com/apache/beam/pull/40529
### Motivation
`S3Options` holds settings for creating an AWS service client: credentials,
region, endpoint and SSL settings. None of them are specific to S3. Upcoming
AWS integrations, such as AWS Secrets Manager support, need the same settings,
so this PR adds a general `AwsOptions` and deprecates `S3Options`.
### Changes
- **New `AwsOptions`** in `apache_beam.options.pipeline_options`, with the
same fields as `S3Options` under an `aws_` prefix:
| New flag | Deprecated flag |
|---|---|
| `--aws_access_key_id` | `--s3_access_key_id` |
| `--aws_secret_access_key` | `--s3_secret_access_key` |
| `--aws_session_token` | `--s3_session_token` |
| `--aws_endpoint_url` | `--s3_endpoint_url` |
| `--aws_region_name` | `--s3_region_name` |
| `--aws_api_version` | `--s3_api_version` |
| `--aws_verify` | `--s3_verify` |
| `--aws_disable_ssl` | `--s3_disable_ssl` |
- **Backward compatibility**
- The `--s3_*` flags are still accepted. Each `aws_*` option falls back to
its `s3_*` counterpart when it isn't set, and logs a one-time deprecation
warning. If both are set, `--aws_*` wins.
- The fallback also covers keyword arguments (e.g.
`PipelineOptions(s3_endpoint_url=...)`) and the dict form accepted by the S3
client.
- `S3Options` is now a deprecated subclass of `AwsOptions`. Its `s3_*`
attributes return the corresponding `aws_*` values, so existing code like
`options.view_as(S3Options).s3_region_name` keeps working.
- Both flag sets are registered on `AwsOptions` only. Since `S3Options` is
no longer a direct subclass of `PipelineOptions`, no flag is registered twice.
- **S3 client:** `boto3_client.Client` now reads its settings from
`AwsOptions`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
