dandy10 commented on a change in pull request #13180:
URL: https://github.com/apache/beam/pull/13180#discussion_r511150097
##########
File path: sdks/python/apache_beam/options/pipeline_options.py
##########
@@ -1321,3 +1320,39 @@ def augment_options(cls, options):
for name, value in override.items():
setattr(options, name, value)
return options
+
+
+class S3Options(PipelineOptions):
+ @classmethod
+ def _add_argparse_args(cls, parser):
+ # These options are passed to the S3 IO Client
+ parser.add_argument(
+ '--aws_access_key_id',
+ default=None,
+ help='The secret key to use when creating the s3 client.')
+ parser.add_argument(
+ '--aws_secret_access_key',
+ default=None,
+ help='The secret key to use when creating the s3 client.')
+ parser.add_argument(
+ '--aws_session_token',
+ default=None,
+ help='The session token to use when creating the s3 client.')
+ parser.add_argument(
+ '--endpoint_url',
+ default=None,
+ help='The complete URL to use for the constructed s3 client.')
+ parser.add_argument(
+ '--region_name',
+ default=None,
+ help='The name of the region associated with the client.')
+ parser.add_argument(
+ '--api_version', default=None, help='The API version to use.')
+ parser.add_argument(
+ '--verify',
+ default=None,
+ help='Whether or not to verify SSL certificates.')
+ parser.add_argument(
+ '--use_ssl',
+ default=True,
+ help='Whether or not to use SSL. By default, SSL is used.')
Review comment:
Fair enough. I've moved them all to use the same s3 prefix which I think
makes sense given they are collected under the S3Options class. @pabloem I
believe the access keys can also be used for other AWS services, although I've
never actually used them. I think it makes sense to consolidate the more
generic aws options with the s3 options together for now given that this is the
only use case at the moment. If there is another AWS service added in the
future it could make sense then to split them up.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
