If I’m not mistaken, it’s not supported in the current implementation of KinesisIO.
PS: Btw, if you still use KinesisIO connector based on AWS SDK v1 [1] then it’s highly recommended to switch to one based on AWS SDK v2 [2] since former is deprecated. [1] https://github.com/apache/beam/blob/master/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisIO.java [2] https://github.com/apache/beam/blob/master/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis/KinesisIO.java — Alexey > On 11 Nov 2022, at 12:34, Sankar Subramaniam > <sankar.subramaniam01....@bbc.co.uk> wrote: > > Hello there, > > Good morning. > > We are using Apache Beam (Java SDK 2.35.0) in our data pipeline to read from > AWS Kinesis Stream using AWS KDA (Kinesis Data Analytics) and so far it’s > working fine for few data pipelines. > > Now we have got a new requirement that AWS KDA (running an application > implemented using Apache Bean SDK 2.35.0) needs to read the data from AWS > Kinesis Stream in different account. > > I have followed AWS Documentation > <https://docs.aws.amazon.com/kinesisanalytics/latest/java/examples-cross.html> > to grant required permission and policies for the AWS KDA and in Apache Bean > implementation we have something like below, > > KinesisIO.read() > .withStreamName(getInputPattern()) > .withAWSClientsProvider( > new KinesisClientsProvider( > Regions.fromName(getAwsRegion()), > getAwsCredentialsProvider(), > getAwsVerifyCertificate(), > getAwsKinesisServiceEndpoint(), > getAwsCloudwatchServiceEndpoint())) > > Here, we could only set the Kinesis Stream and not the ARN. With the above > implementation, this application couldn’t read from the stream and from the > logs we are seeing it’s trying to connect to the stream in the same AWS > Account. The ARN formed using streamName assumes it’s in the same AWS Account > whereas we want to connect to Kinesis Stream in another AWS Account. > > Note: We are using ‘DefaultAWSCredentialsProviderChain’. > > With this situation, wondering am I missing something / doing incorrectly > here. Could you please give us some pointers how to use Beam to read from a > (Kinesis)Stream in different AWS Account. Thanks. > > Regards, > Sankar
