Re: [AWS][1.15.3] - Unable to use vpc endpoint

2022-03-08 Thread Peter Turcsanyi 
Hi Maxime,

It seems to be a similar issue as NIFI-8662 [1] where the AWS
client library cannot parse the Region from a VPC-style endpoint and we
need to get it from the Region property.
I will look into it in more detail and get back to you.

Regards,
Peter Turcsanyi

[1] https://issues.apache.org/jira/browse/NIFI-8662

On Tue, Mar 8, 2022 at 4:05 PM LEZIER Maxime (ITNOVEM) 
wrote:

> Hello,
>
>
>
> I would like to ask you again about this subject, would you have an answer
> for me please?
>
>
>
> *ML*
>
>
>
>
>
> *De : *LEZIER Maxime (ITNOVEM) 
> *Date : *vendredi, 4 mars 2022 à 08:47
> *À : *users@nifi.apache.org 
> *Objet : *[AWS][1.15.3] - Unable to use vpc endpoint
>
> Hello,
>
>
>
> I use nifi in 1.15.3 version.
>
>
>
> I have to reach aws ressources (S3,SQS..) which have VPCE in front.
>
>
>
> My test pipeline for this usecase is a simple ListS3 with AwsCredentials
> controller.
>
>
>
> In the listS3 i have set the bucket’s name, the region, point it to the
> credentials controller and set an endpoint override url with a url like
> this :
>
> https://bucketname.vpce-xxx-xxx.s3.eu-west-3.vpce.amazonaws.com
>
>
>
> For the controller, i set up access id/secret id, the role’s arn i have to
> assume, the role assume session name, and the sts vpce endpoint url which
> have this form :
>
> vpce-xxx-xxx.sts.eu-west-3.vpce.amazonaws.com (tried with and without
> https:// in front)d
>
>
>
> When i start this flow i’ve got this error :
>
>
>
> 2022-03-04 08:42:08,067 ERROR [Timer-Driven Process Thread-16]
> org.apache.nifi.processors.aws.s3.ListS3
> ListS3[id=501cd602-017f-1000--9f28cb25] Failed to list contents of
> bucket due to
> com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:
> Credential should be scoped to a valid region, not 'vpce'. (Service:
> AWSSecurityTokenService; Status Code: 403; Error Code:
> SignatureDoesNotMatch; Request ID: d57a4d34-7e59-41ba-9e71-4505dd801e75;
> Proxy: null):
> com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:
> *Credential should be scoped to a valid region, not 'vpce'*. (Service:
> AWSSecurityTokenService; Status Code: 403; Error Code:
> SignatureDoesNotMatch; Request ID: d57a4d34-7e59-41ba-9e71-4505dd801e75;
> Proxy: null)
>
> com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException:
> Credential should be scoped to a valid region, not 'vpce'. (Service:
> AWSSecurityTokenService; Status Code: 403; Error Code:
> SignatureDoesNotMatch; Request ID: d57a4d34-7e59-41ba-9e71-4505dd801e75;
> Proxy: null)
>
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1819)
>
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1403)
>
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1372)
>
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)
>
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
>
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
>
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
>
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
>
> at
> com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
>
> at
> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
>
> at
> com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
>
> at
> com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.doInvoke(AWSSecurityTokenServiceClient.java:1728)
>
> at
> com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(AWSSecurityTokenServiceClient.java:1695)
>
> at
> com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(AWSSecurityTokenServiceClient.java:1684)
>
> at
> com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.executeAssumeRole(AWSSecurityTokenServiceClient.java:488)
>
> at
> com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.assumeRole(AWSSecurityTokenServiceClient.java:457)
>
> at
> com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.newSession(STSAssumeRoleSessionCredentialsProvider.java:343)
>
> at
> com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.access$000(STSAssumeRoleSessionCredentialsProvider.java:41)
>
> at
> com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:90)
>
> at
>

Re: [AWS][1.15.3] - Unable to use vpc endpoint

2022-03-08 Thread LEZIER Maxime (ITNOVEM) 
Hello,

I would like to ask you again about this subject, would you have an answer for 
me please?

ML


De : LEZIER Maxime (ITNOVEM) 
Date : vendredi, 4 mars 2022 à 08:47
À : users@nifi.apache.org 
Objet : [AWS][1.15.3] - Unable to use vpc endpoint

Hello,



I use nifi in 1.15.3 version.



I have to reach aws ressources (S3,SQS..) which have VPCE in front.



My test pipeline for this usecase is a simple ListS3 with AwsCredentials 
controller.



In the listS3 i have set the bucket’s name, the region, point it to the 
credentials controller and set an endpoint override url with a url like this :

https://bucketname.vpce-xxx-xxx.s3.eu-west-3.vpce.amazonaws.com



For the controller, i set up access id/secret id, the role’s arn i have to 
assume, the role assume session name, and the sts vpce endpoint url which have 
this form :

vpce-xxx-xxx.sts.eu-west-3.vpce.amazonaws.com 
(tried with and without https:// in front)d



When i start this flow i’ve got this error :



2022-03-04 08:42:08,067 ERROR [Timer-Driven Process Thread-16] 
org.apache.nifi.processors.aws.s3.ListS3 
ListS3[id=501cd602-017f-1000--9f28cb25] Failed to list contents of 
bucket due to 
com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException: 
Credential should be scoped to a valid region, not 'vpce'. (Service: 
AWSSecurityTokenService; Status Code: 403; Error Code: SignatureDoesNotMatch; 
Request ID: d57a4d34-7e59-41ba-9e71-4505dd801e75; Proxy: null): 
com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException: 
Credential should be scoped to a valid region, not 'vpce'. (Service: 
AWSSecurityTokenService; Status Code: 403; Error Code: SignatureDoesNotMatch; 
Request ID: d57a4d34-7e59-41ba-9e71-4505dd801e75; Proxy: null)

com.amazonaws.services.securitytoken.model.AWSSecurityTokenServiceException: 
Credential should be scoped to a valid region, not 'vpce'. (Service: 
AWSSecurityTokenService; Status Code: 403; Error Code: SignatureDoesNotMatch; 
Request ID: d57a4d34-7e59-41ba-9e71-4505dd801e75; Proxy: null)

at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1819)

at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1403)

at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1372)

at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)

at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)

at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)

at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)

at 
com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)

at 
com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)

at 
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)

at 
com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)

at 
com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.doInvoke(AWSSecurityTokenServiceClient.java:1728)

at 
com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(AWSSecurityTokenServiceClient.java:1695)

at 
com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(AWSSecurityTokenServiceClient.java:1684)

at 
com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.executeAssumeRole(AWSSecurityTokenServiceClient.java:488)

at 
com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.assumeRole(AWSSecurityTokenServiceClient.java:457)

at 
com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.newSession(STSAssumeRoleSessionCredentialsProvider.java:343)

at 
com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.access$000(STSAssumeRoleSessionCredentialsProvider.java:41)

at 
com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:90)

at 
com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider$1.call(STSAssumeRoleSessionCredentialsProvider.java:87)

at 
com.amazonaws.auth.RefreshableTask.refreshValue(RefreshableTask.java:257)

at 
com.amazonaws.auth.RefreshableTask.blockingRefresh(RefreshableTask.java:213)

at 
com.amazonaws.auth.RefreshableTask.getValue(RefreshableTask.java:154)

at 
com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.getCredentials(STSAssumeRoleSessionCredentialsProvider.java:315)

at