pabloem commented on a change in pull request #13180:
URL: https://github.com/apache/beam/pull/13180#discussion_r511132983
##########
File path: sdks/python/apache_beam/options/pipeline_options.py
##########
@@ -1321,3 +1320,39 @@ def augment_options(cls, options):
for name, value in override.items():
setattr(options, name, value)
return options
+
+
+class S3Options(PipelineOptions):
+ @classmethod
+ def _add_argparse_args(cls, parser):
+ # These options are passed to the S3 IO Client
+ parser.add_argument(
+ '--aws_access_key_id',
+ default=None,
+ help='The secret key to use when creating the s3 client.')
+ parser.add_argument(
+ '--aws_secret_access_key',
+ default=None,
+ help='The secret key to use when creating the s3 client.')
+ parser.add_argument(
+ '--aws_session_token',
+ default=None,
+ help='The session token to use when creating the s3 client.')
+ parser.add_argument(
+ '--endpoint_url',
+ default=None,
+ help='The complete URL to use for the constructed s3 client.')
+ parser.add_argument(
+ '--region_name',
+ default=None,
+ help='The name of the region associated with the client.')
+ parser.add_argument(
+ '--api_version', default=None, help='The API version to use.')
+ parser.add_argument(
+ '--verify',
+ default=None,
+ help='Whether or not to verify SSL certificates.')
+ parser.add_argument(
+ '--use_ssl',
+ default=True,
+ help='Whether or not to use SSL. By default, SSL is used.')
Review comment:
You are correct that it's desirable to use a sort of namespace prefix.
Perhaps `--aws_` or `--aws_s3_`. What do you think?
You must know more than me about s3 and AWS - I wonder if
`aws_session_token`, `aws_secret_access_key`, `aws_access_key_id` in this
context are specific for s3, or if they provide some sort of AWS-wide
authentication?
If they're s3-specific, then maybe we should namespace them as `aws_s3`? Let
me know what you think.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
