Hi Sankar,
First, as Alexey pointed out, please try and migrate to the Beam AWS SDK v2 as
soon as possible. The SDK v1 (including the Kinesis module) has been long
deprecated and will be removed some time soon.
The AWS API doesn’t support cross-account access for Kinesis using an ARN. This
is always based on the stream name as shown in the example you’ve linked [1].
You must use STS / assume role credentials to do this, it can’t be done using
DefaultAWSCredentialsProviderChain.
Assuming you’ve correctly configured all required policies and roles (on both
accounts) following [1], you can then use the StsAssumeRoleCredentialsProvider.
For the Beam AWS SDK v2 this can be done using pipeline options [2], or
programmatically of course.
Support for StsAssumeRoleCredentialsProvider is more limited in SDK v1, though
it should also work.
--awsCredentialsProvider={
"@type": "StsAssumeRoleCredentialsProvider",
"roleArn": "arn:aws:iam::SOURCE01234567:role/KA-Source-Stream-Role",
"roleSessionName": "ksassumedrolesession"
}
The STS client will implicitly use the DefaultAWSCredentialsProviderChain to
assume the above role of the source account using an authenticated principal of
the sink account (source/sink nomenclature as used in [1]). Please make sure
your environment is configured accordingly, this part is easy to miss.
Let me know if you have more questions!
Regards,
Moritz
[1] https://docs.aws.amazon.com/kinesisanalytics/latest/java/examples-cross.html
[2]
https://beam.apache.org/releases/javadoc/2.42.0/org/apache/beam/sdk/io/aws2/options/AwsOptions.html#getAwsCredentialsProvider--
On 14.11.22, 19:05, "Alexey Romanenko" <[email protected]> wrote:
If I’m not mistaken, it’s not supported in the current implementation of
KinesisIO. PS: Btw, if you still use KinesisIO connector based on AWS SDK v1
[1] then it’s highly recommended to switch to one based on AWS SDK v2 [2] since
former is
If I’m not mistaken, it’s not supported in the current implementation of
KinesisIO.
PS: Btw, if you still use KinesisIO connector based on AWS SDK v1 [1] then it’s
highly recommended to switch to one based on AWS SDK v2 [2] since former is
deprecated.
[1]
https://github.com/apache/beam/blob/master/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisIO.java<https://urldefense.com/v3/__https:/github.com/apache/beam/blob/master/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisIO.java__;!!CiXD_PY!XzTvAWqfcGyw-k0V1j7SeMMVBnlyM1dysSffSXz1PcoXs_RSX3GuJf_sXTE4Sk9QIou906p0z_tyKTCweGmqKQ$>
[2]
https://github.com/apache/beam/blob/master/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis/KinesisIO.java<https://urldefense.com/v3/__https:/github.com/apache/beam/blob/master/sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/kinesis/KinesisIO.java__;!!CiXD_PY!XzTvAWqfcGyw-k0V1j7SeMMVBnlyM1dysSffSXz1PcoXs_RSX3GuJf_sXTE4Sk9QIou906p0z_tyKTBLSThXSg$>
—
Alexey
On 11 Nov 2022, at 12:34, Sankar Subramaniam
<[email protected]> wrote:
Hello there,
Good morning.
We are using Apache Beam (Java SDK 2.35.0) in our data pipeline to read from
AWS Kinesis Stream using AWS KDA (Kinesis Data Analytics) and so far it’s
working fine for few data pipelines.
Now we have got a new requirement that AWS KDA (running an application
implemented using Apache Bean SDK 2.35.0) needs to read the data from AWS
Kinesis Stream in different account.
I have followed AWS
Documentation<https://urldefense.com/v3/__https:/docs.aws.amazon.com/kinesisanalytics/latest/java/examples-cross.html__;!!CiXD_PY!XzTvAWqfcGyw-k0V1j7SeMMVBnlyM1dysSffSXz1PcoXs_RSX3GuJf_sXTE4Sk9QIou906p0z_tyKTBEETAzMw$>
to grant required permission and policies for the AWS KDA and in Apache Bean
implementation we have something like below,
KinesisIO.read()
.withStreamName(getInputPattern())
.withAWSClientsProvider(
new KinesisClientsProvider(
Regions.fromName(getAwsRegion()),
getAwsCredentialsProvider(),
getAwsVerifyCertificate(),
getAwsKinesisServiceEndpoint(),
getAwsCloudwatchServiceEndpoint()))
Here, we could only set the Kinesis Stream and not the ARN. With the above
implementation, this application couldn’t read from the stream and from the
logs we are seeing it’s trying to connect to the stream in the same AWS
Account. The ARN formed using streamName assumes it’s in the same AWS Account
whereas we want to connect to Kinesis Stream in another AWS Account.
Note: We are using ‘DefaultAWSCredentialsProviderChain’.
With this situation, wondering am I missing something / doing incorrectly here.
Could you please give us some pointers how to use Beam to read from a
(Kinesis)Stream in different AWS Account. Thanks.
Regards,
Sankar
As a recipient of an email from Talend, your contact personal data will be on
our systems. Please see our privacy notice. <https://www.talend.com/privacy/>
