[jira] [Commented] (NIFI-7900) Add AWS session token to AWSCredentialsProvider
[ https://issues.apache.org/jira/browse/NIFI-7900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17556874#comment-17556874 ] iain smith commented on NIFI-7900: -- [~john.wise] - Could this be used in your use case above, by setting 'Use Default Credentials' to 'true' then configuring your periodic STS-endpoint credentials retrieving flow to write the access key id, secret access key and session token into the AWS default credentials file $HOME/.aws/credentials (in the format shown below)? [default] aws_access_key_id = aws_secret_access_key = aws_session_token = X or set them as environment variables with the correct names maybe? That should also be picked up as long as 'Use Default Credentials' is 'true' > Add AWS session token to AWSCredentialsProvider > --- > > Key: NIFI-7900 > URL: https://issues.apache.org/jira/browse/NIFI-7900 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.9.2, 1.12.1 >Reporter: Jody >Assignee: Peter Turcsanyi >Priority: Major > > As a NiFi user, I want to use AWS processors, e.g. PutS3Object processor, > with temporary credentials to allow connecting to secure AWS environments > that make use of the AWS Security Token Service. > > The NiFi AWSCredentialsProviderControllerService is giving an option to add > the required fields for using temporary credentials. While access key id and > secret access key properties can be configured, the property "session token" > is not available. The session token property must be provided when temporary > credentials are used. If the session token is not presented, an error will be > thrown: "The AWS Access Key Id you provided does not exist in our records. > (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId" -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (NIFI-7900) Add AWS session token to AWSCredentialsProvider
[ https://issues.apache.org/jira/browse/NIFI-7900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17556793#comment-17556793 ] iain smith commented on NIFI-7900: -- I've just noticed on [https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-aws-nar/1.15.3/org.apache.nifi.processors.aws.credentials.provider.service.AWSCredentialsProviderControllerService/index.html] "Uses default credentials without configuration. Default credentials support EC2 instance profile/role, default user profile, environment variables, etc" Is using the temporary credentials from an EC2 instance profile/role already supported? If so, should they be being automatically detected and used as long as no key id / access key is configured in nifi? > Add AWS session token to AWSCredentialsProvider > --- > > Key: NIFI-7900 > URL: https://issues.apache.org/jira/browse/NIFI-7900 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.9.2, 1.12.1 >Reporter: Jody >Assignee: Peter Turcsanyi >Priority: Major > > As a NiFi user, I want to use AWS processors, e.g. PutS3Object processor, > with temporary credentials to allow connecting to secure AWS environments > that make use of the AWS Security Token Service. > > The NiFi AWSCredentialsProviderControllerService is giving an option to add > the required fields for using temporary credentials. While access key id and > secret access key properties can be configured, the property "session token" > is not available. The session token property must be provided when temporary > credentials are used. If the session token is not presented, an error will be > thrown: "The AWS Access Key Id you provided does not exist in our records. > (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId" -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (NIFI-7900) Add AWS session token to AWSCredentialsProvider
[ https://issues.apache.org/jira/browse/NIFI-7900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17556789#comment-17556789 ] iain smith commented on NIFI-7900: -- This really should be supported for Nifi running on EC2 instances... we should not have to create an IAM user with static API keys for Nifi to use, when the EC2 already provides local API credentials which are more secure (ie. regularly auto-rotated, and not likely to set off irritating security non-compliance alerts!) > Add AWS session token to AWSCredentialsProvider > --- > > Key: NIFI-7900 > URL: https://issues.apache.org/jira/browse/NIFI-7900 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.9.2, 1.12.1 >Reporter: Jody >Assignee: Peter Turcsanyi >Priority: Major > > As a NiFi user, I want to use AWS processors, e.g. PutS3Object processor, > with temporary credentials to allow connecting to secure AWS environments > that make use of the AWS Security Token Service. > > The NiFi AWSCredentialsProviderControllerService is giving an option to add > the required fields for using temporary credentials. While access key id and > secret access key properties can be configured, the property "session token" > is not available. The session token property must be provided when temporary > credentials are used. If the session token is not presented, an error will be > thrown: "The AWS Access Key Id you provided does not exist in our records. > (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId" -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (NIFI-7900) Add AWS session token to AWSCredentialsProvider
[
https://issues.apache.org/jira/browse/NIFI-7900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17556760#comment-17556760
]
iain smith commented on NIFI-7900:
--
Another similar use case is when Nifi is running on an ec2 instance, or in a
container hosted on an ec2 instance. If the instance has an associated IAM role
('instance profile') granting permissions to AWS services, the role's temporary
credentials (Access Key / Secret Key / Session Token) are available to a nifi
flow via a http query to the instance's metadata at the 'special url' (only
accessible locally on the ec2 and hosted containers)
http://169.254.169.254/latest/meta-data/iam/security-credentials/
> Add AWS session token to AWSCredentialsProvider
> ---
>
> Key: NIFI-7900
> URL: https://issues.apache.org/jira/browse/NIFI-7900
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Extensions
>Affects Versions: 1.9.2, 1.12.1
>Reporter: Jody
>Assignee: Peter Turcsanyi
>Priority: Major
>
> As a NiFi user, I want to use AWS processors, e.g. PutS3Object processor,
> with temporary credentials to allow connecting to secure AWS environments
> that make use of the AWS Security Token Service.
>
> The NiFi AWSCredentialsProviderControllerService is giving an option to add
> the required fields for using temporary credentials. While access key id and
> secret access key properties can be configured, the property "session token"
> is not available. The session token property must be provided when temporary
> credentials are used. If the session token is not presented, an error will be
> thrown: "The AWS Access Key Id you provided does not exist in our records.
> (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId"
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (NIFI-7900) Add AWS session token to AWSCredentialsProvider
[ https://issues.apache.org/jira/browse/NIFI-7900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17504352#comment-17504352 ] John Wise commented on NIFI-7900: - [~turcsanyip] - Although it's a bit unwieldy, the above is a viable use case for supporting temporary AWS credentials in the AWSCredentialsProviderControllerService. I presume that session token support would also require updates to most/all of the AWS processors, but it should be a minimal lift to add an additional configuration field & header. Due to a draconian policy decreed by our ill-informed & stubborn security organization, we're prohibited from creating new user accounts to acquire static access & secret keys; no, that doesn't make sense at all, but that's what we've been stuck with for several years now. Without that session token support, anyone required to use temporary credentials is essentially unable to use any AWS services with NiFi. > Add AWS session token to AWSCredentialsProvider > --- > > Key: NIFI-7900 > URL: https://issues.apache.org/jira/browse/NIFI-7900 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.9.2, 1.12.1 >Reporter: Jody >Assignee: Peter Turcsanyi >Priority: Major > > As a NiFi user, I want to use AWS processors, e.g. PutS3Object processor, > with temporary credentials to allow connecting to secure AWS environments > that make use of the AWS Security Token Service. > > The NiFi AWSCredentialsProviderControllerService is giving an option to add > the required fields for using temporary credentials. While access key id and > secret access key properties can be configured, the property "session token" > is not available. The session token property must be provided when temporary > credentials are used. If the session token is not presented, an error will be > thrown: "The AWS Access Key Id you provided does not exist in our records. > (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId" -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (NIFI-7900) Add AWS session token to AWSCredentialsProvider
[ https://issues.apache.org/jira/browse/NIFI-7900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17503064#comment-17503064 ] John Wise commented on NIFI-7900: - FWIW, I've created a flow which queries the AWS API every 4 hours for temporary credentials, grabs the access key, secret key, session token, and expiration into attributes, updates an AWSCredentialsProviderControllerService with the keys, then restarts it. Unfortunately, without the ability to pass in & use the session token, none of the attached AWS processors can authenticate, even though the access & secret keys are current at the time the flow runs. > Add AWS session token to AWSCredentialsProvider > --- > > Key: NIFI-7900 > URL: https://issues.apache.org/jira/browse/NIFI-7900 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.9.2, 1.12.1 >Reporter: Jody >Assignee: Peter Turcsanyi >Priority: Major > > As a NiFi user, I want to use AWS processors, e.g. PutS3Object processor, > with temporary credentials to allow connecting to secure AWS environments > that make use of the AWS Security Token Service. > > The NiFi AWSCredentialsProviderControllerService is giving an option to add > the required fields for using temporary credentials. While access key id and > secret access key properties can be configured, the property "session token" > is not available. The session token property must be provided when temporary > credentials are used. If the session token is not presented, an error will be > thrown: "The AWS Access Key Id you provided does not exist in our records. > (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId" -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (NIFI-7900) Add AWS session token to AWSCredentialsProvider
[ https://issues.apache.org/jira/browse/NIFI-7900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17496779#comment-17496779 ] Peter Turcsanyi commented on NIFI-7900: --- [~jmaaks] "Session Token" property can be added to AWSCredentialsProviderControllerService but how would you configure it? Using a static values would not make sense as the token is temporary. So if I understand it correctly, the token value would come from a FlowFile attribute in practice. The FlowFile attribute approach could work for PutS3Object which has incoming FlowFiles but not for ListS3 which has no incoming connection (currently, so it would be a bigger change to add the FlowFile attribute solution here). Another question: how do you acquire the Session Token that would be passed to AWSCredentialsProviderControllerService in the new property? In the NiFi flow using a custom processor? > Add AWS session token to AWSCredentialsProvider > --- > > Key: NIFI-7900 > URL: https://issues.apache.org/jira/browse/NIFI-7900 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.9.2, 1.12.1 >Reporter: Jody >Assignee: Peter Turcsanyi >Priority: Major > > As a NiFi user, I want to use AWS processors, e.g. PutS3Object processor, > with temporary credentials to allow connecting to secure AWS environments > that make use of the AWS Security Token Service. > > The NiFi AWSCredentialsProviderControllerService is giving an option to add > the required fields for using temporary credentials. While access key id and > secret access key properties can be configured, the property "session token" > is not available. The session token property must be provided when temporary > credentials are used. If the session token is not presented, an error will be > thrown: "The AWS Access Key Id you provided does not exist in our records. > (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId" -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (NIFI-7900) Add AWS session token to AWSCredentialsProvider
[ https://issues.apache.org/jira/browse/NIFI-7900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17454864#comment-17454864 ] Jeff Maaks commented on NIFI-7900: -- I'm also seeing this as a potential issue, explicitly when using MinIO configured with LDAP SSO. How would the AWSCredentialsProviderControllerService work around this? As originally noted above "While access key id and secret access key properties can be configured, the property "session token" is not available." – Can this not be added? > Add AWS session token to AWSCredentialsProvider > --- > > Key: NIFI-7900 > URL: https://issues.apache.org/jira/browse/NIFI-7900 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.9.2, 1.12.1 >Reporter: Jody >Assignee: Peter Turcsanyi >Priority: Major > > As a NiFi user, I want to use AWS processors, e.g. PutS3Object processor, > with temporary credentials to allow connecting to secure AWS environments > that make use of the AWS Security Token Service. > > The NiFi AWSCredentialsProviderControllerService is giving an option to add > the required fields for using temporary credentials. While access key id and > secret access key properties can be configured, the property "session token" > is not available. The session token property must be provided when temporary > credentials are used. If the session token is not presented, an error will be > thrown: "The AWS Access Key Id you provided does not exist in our records. > (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId" -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (NIFI-7900) Add AWS session token to AWSCredentialsProvider
[ https://issues.apache.org/jira/browse/NIFI-7900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17359084#comment-17359084 ] Peter Turcsanyi commented on NIFI-7900: --- [~j9dy], [~Svirsky] Could you please describe your use cases in more detail? AWSCredentialsProviderControllerService supports cross-account access via Access Key / Secret Key plus Assume Role* properties. No explicit Session Token property is needed in this case because it will be retrieved from STS in the background along with the temporary Access Key / Secret Key. You need to configure the primary Access Key / Secret Key on the controller service which will be used to access STS. Or do you have an external mechanism to get (and refresh) the temporary credentials and you wold like to pass them to controller service via FlowFile attributes? In this case only the Session Token property needs to be added to the controller service but it would be the flow designer's responsibility to provide the token value. > Add AWS session token to AWSCredentialsProvider > --- > > Key: NIFI-7900 > URL: https://issues.apache.org/jira/browse/NIFI-7900 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.9.2, 1.12.1 >Reporter: Jody >Assignee: Peter Turcsanyi >Priority: Major > > As a NiFi user, I want to use AWS processors, e.g. PutS3Object processor, > with temporary credentials to allow connecting to secure AWS environments > that make use of the AWS Security Token Service. > > The NiFi AWSCredentialsProviderControllerService is giving an option to add > the required fields for using temporary credentials. While access key id and > secret access key properties can be configured, the property "session token" > is not available. The session token property must be provided when temporary > credentials are used. If the session token is not presented, an error will be > thrown: "The AWS Access Key Id you provided does not exist in our records. > (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-7900) Add AWS session token to AWSCredentialsProvider
[ https://issues.apache.org/jira/browse/NIFI-7900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17324960#comment-17324960 ] Nir commented on NIFI-7900: --- It happens for me as well. It seems like nifi doesn't support AWS SSO authentication at the moment > Add AWS session token to AWSCredentialsProvider > --- > > Key: NIFI-7900 > URL: https://issues.apache.org/jira/browse/NIFI-7900 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.9.2, 1.12.1 >Reporter: Jody >Priority: Major > > As a NiFi user, I want to use AWS processors, e.g. PutS3Object processor, > with temporary credentials to allow connecting to secure AWS environments > that make use of the AWS Security Token Service. > > The NiFi AWSCredentialsProviderControllerService is giving an option to add > the required fields for using temporary credentials. While access key id and > secret access key properties can be configured, the property "session token" > is not available. The session token property must be provided when temporary > credentials are used. If the session token is not presented, an error will be > thrown: "The AWS Access Key Id you provided does not exist in our records. > (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId" -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NIFI-7900) Add AWS session token to AWSCredentialsProvider
[ https://issues.apache.org/jira/browse/NIFI-7900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17210083#comment-17210083 ] Jody commented on NIFI-7900: More information: [https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp_use-resources.html] [https://docs.aws.amazon.com/credref/latest/refdocs/setting-global-aws_session_token.html] [https://docs.aws.amazon.com/AmazonS3/latest/dev/AuthUsingTempSessionTokenJava.html] > Add AWS session token to AWSCredentialsProvider > --- > > Key: NIFI-7900 > URL: https://issues.apache.org/jira/browse/NIFI-7900 > Project: Apache NiFi > Issue Type: Improvement > Components: Extensions >Affects Versions: 1.9.2, 1.12.1 >Reporter: Jody >Priority: Major > > As a NiFi user, I want to use AWS processors, e.g. PutS3Object processor, > with temporary credentials to allow connecting to secure AWS environments > that make use of the AWS Security Token Service. > > The NiFi AWSCredentialsProviderControllerService is giving an option to add > the required fields for using temporary credentials. While access key id and > secret access key properties can be configured, the property "session token" > is not available. The session token property must be provided when temporary > credentials are used. If the session token is not presented, an error will be > thrown: "The AWS Access Key Id you provided does not exist in our records. > (Service: Amazon S3; Status Code: 403; Error Code: InvalidAccessKeyId" -- This message was sent by Atlassian Jira (v8.3.4#803005)
