Re: Setting AWS endponts per transform
It would also be ideal if someone extended the SqsIO transform to allow
specifying endpoints via a builder method.
On Fri, Sep 25, 2020 at 9:50 AM Luke Cwik wrote:
> You can create a PipelineOptions interface with a unique endpoint
> identifier for each service you want to use like:
> public interface AwsEndpointOptions extends PipelineOptions {
> @Default.InstanceFactory(FallbackToAwsOptions.class);
> String getSnsEndpoint();
> void setSnsEndpoint();
>
> @Default.InstanceFactory(FallbackToAwsOptions.class);
> String getS3Endpoint();
> void setS3Endpoint();
> ...
>
> class FallbackToAwsOptions implements DefaultValueFactory {
> @Override
> public String create(PipelineOptions options) {
> return options.as(AwsOptions.class).getAwsServiceEndpoint();
> }
> }
> }
>
> It looks like AWS IOs support passing in a provider[1] and/or configuring
> the endpoint during pipeline construction. Then during pipeline
> creation/execution you can configure each AWS IO instance with the specific
> endpoint from this new PipelineOptions interface.
>
> 1:
> https://github.com/apache/beam/blob/6fdde4f4eab72b49b10a8bb1cb3be263c5c416b5/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/SnsIO.java#L273
>
> On Thu, Sep 24, 2020 at 4:24 PM wrote:
>
>> We're currently working on getting our Beam app working with localstack
>> (and potentially other AWS regions). We're using SqsIO and S3 as part of
>> our pipeline (with other AWS components likely to come into the mix).
>> While I could cast the PipelineOptions to AwsOptions and then call
>> AwsOptions.setAwsServiceEndpoint() prior to pipeline construction, that
>> won't work as different AWS services make use of different endpoints --
>> e.g. the endpoint for SqsIO isn't going to work for S3.
>>
>> What I'd really like to do is provide a different set of AwsOptions per
>> AWS service. What's the best means of accomplishing this?
>>
>> Tim.
>>
>>
Re: Setting AWS endponts per transform
You can create a PipelineOptions interface with a unique endpoint
identifier for each service you want to use like:
public interface AwsEndpointOptions extends PipelineOptions {
@Default.InstanceFactory(FallbackToAwsOptions.class);
String getSnsEndpoint();
void setSnsEndpoint();
@Default.InstanceFactory(FallbackToAwsOptions.class);
String getS3Endpoint();
void setS3Endpoint();
...
class FallbackToAwsOptions implements DefaultValueFactory {
@Override
public String create(PipelineOptions options) {
return options.as(AwsOptions.class).getAwsServiceEndpoint();
}
}
}
It looks like AWS IOs support passing in a provider[1] and/or configuring
the endpoint during pipeline construction. Then during pipeline
creation/execution you can configure each AWS IO instance with the specific
endpoint from this new PipelineOptions interface.
1:
https://github.com/apache/beam/blob/6fdde4f4eab72b49b10a8bb1cb3be263c5c416b5/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/SnsIO.java#L273
On Thu, Sep 24, 2020 at 4:24 PM wrote:
> We're currently working on getting our Beam app working with localstack
> (and potentially other AWS regions). We're using SqsIO and S3 as part of
> our pipeline (with other AWS components likely to come into the mix).
> While I could cast the PipelineOptions to AwsOptions and then call
> AwsOptions.setAwsServiceEndpoint() prior to pipeline construction, that
> won't work as different AWS services make use of different endpoints --
> e.g. the endpoint for SqsIO isn't going to work for S3.
>
> What I'd really like to do is provide a different set of AwsOptions per
> AWS service. What's the best means of accomplishing this?
>
> Tim.
>
>
Setting AWS endponts per transform
We're currently working on getting our Beam app working with localstack (and potentially other AWS regions). We're using SqsIO and S3 as part of our pipeline (with other AWS components likely to come into the mix). While I could cast the PipelineOptions to AwsOptions and then call AwsOptions.setAwsServiceEndpoint() prior to pipeline construction, that won't work as different AWS services make use of different endpoints -- e.g. the endpoint for SqsIO isn't going to work for S3. What I'd really like to do is provide a different set of AwsOptions per AWS service. What's the best means of accomplishing this? Tim.
