You can create a PipelineOptions interface with a unique endpoint
identifier for each service you want to use like:
public interface AwsEndpointOptions extends PipelineOptions {
@Default.InstanceFactory(FallbackToAwsOptions.class);
String getSnsEndpoint();
void setSnsEndpoint();
@Default.InstanceFactory(FallbackToAwsOptions.class);
String getS3Endpoint();
void setS3Endpoint();
...
class FallbackToAwsOptions implements DefaultValueFactory<String> {
@Override
public String create(PipelineOptions options) {
return options.as(AwsOptions.class).getAwsServiceEndpoint();
}
}
}
It looks like AWS IOs support passing in a provider[1] and/or configuring
the endpoint during pipeline construction. Then during pipeline
creation/execution you can configure each AWS IO instance with the specific
endpoint from this new PipelineOptions interface.
1:
https://github.com/apache/beam/blob/6fdde4f4eab72b49b10a8bb1cb3be263c5c416b5/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/SnsIO.java#L273
On Thu, Sep 24, 2020 at 4:24 PM <[email protected]> wrote:
> We're currently working on getting our Beam app working with localstack
> (and potentially other AWS regions). We're using SqsIO and S3 as part of
> our pipeline (with other AWS components likely to come into the mix).
> While I could cast the PipelineOptions to AwsOptions and then call
> AwsOptions.setAwsServiceEndpoint() prior to pipeline construction, that
> won't work as different AWS services make use of different endpoints --
> e.g. the endpoint for SqsIO isn't going to work for S3.
>
> What I'd really like to do is provide a different set of AwsOptions per
> AWS service. What's the best means of accomplishing this?
>
> Tim.
>
>
