shunping opened a new pull request, #40547:
URL: https://github.com/apache/beam/pull/40547
### Motivation
Today, AWS credentials have to be passed as plain text in pipeline options
(`--aws_access_key_id`, `--aws_secret_access_key`, `--aws_session_token`). That
means they show up in job graphs, launch commands and logs. This PR lets them
be references to secrets stored in a secret manager instead.
### Changes
- **New option `--aws_credentials_secret_manager`** in `AwsOptions`.
- When set, the credential options and their `--s3_*` overrides are secret
specs (a JSON string or plain text). They are resolved with
`Secret.from_json(spec, secret_manager)`.
- When not set, the options are used as raw credentials, as before.
- AWS Secrets Manager is rejected as the credentials manager, because you
need AWS credentials to reach it.
- **New module `apache_beam/io/aws/clients/client_options.py`**, so every
boto3 client Beam creates is configured the same way
### Example
```
python my_pipeline.py
--aws_credentials_secret_manager=GoogleCloudSecretManager
--aws_access_key_id='{"name": "aws-key-id", "project": "my-project"}'
--aws_secret_access_key='{"name": "aws-secret-key", "project": "my-project"}'
--aws_region_name=us-east-2
```
Part of #40528.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
