Keren,
Which version of NiFi are you using?
One thing I noticed in your configuration of FetchS3Object is you are
setting both the Access Key and Secret Key properties with the AWS
Credentials Provider. When you are using the AWS Credentials Provider
Service, you should not specify keys.
A more certainly helpful thing to do is enable debug logging for the AWS
processor package by adding a line like the following to conf/logback.xml:
<logger name="org.apache.nifi.processors.aws" level="DEBUG"/>
With the debug logging enabled, there are messages indicating which
credential type is being attempted. Your settings for the AWS Credentials
Provider look appropriate. The controller service is indeed designed to
refresh the STS token automagically using the AWS SDK classes for temporary
credentials.
Last, you might experiment with configuring
AWSCredentialsProviderControllerService to use your named CLI profile
"crossaccountrole", which should also work.
Thanks,
James
On Wed, Aug 31, 2016 at 1:44 PM, Tseytlin, Keren <
[email protected]> wrote:
> Hi All!
>
> Looking for some help on enabling Cross Account communication within Nifi!
>
> My goal: There are files stored from CloudTrail in an S3 bucket in VPC B.
> My Nifi machines are in VPC A. I want Nifi to be able to get those files
> from VPC B. VPC A and VPC B need to be communicating in the FetchS3Object
> component.
>
> See this link for some additional info: http://docs.aws.amazon.com/
> awscloudtrail/latest/userguide/cloudtrail-sharing-logs-assume-role.html
>
> I have communication working manually on the Nifi machines in VPC A when I
> use the AWS CLI. The process is as follows:
>
> 1. Run sts -assume-role on my Nifi machine (VPC A) to assume a role
> I've created in VPC B that is configured to have access to the S3 bucket in
> VPC B.
>
> 2. This will generate temporary keys that need to be refreshed every
> hour. There is no way to have assume role create permanent keys. Export the
> keys as environment variables.
>
> 3. Set up ~/.aws/config to have a profile "crossaccountrole" that
> connects to the arn of the role created in VPC B.
>
> 4. Run the following command à "aws s3 cp s3://<object path> <object
> name locally> --profile crossaccountrole"
>
> Most importantly, if I ever try to run this without the --profile flag,
> then it will not allow me to download the file. It seems like perhaps to
> get it to work with Nifi I need a place to pass in the profile that needs
> to be used in order for the communication to work.
>
> I've been trying to implement this in Nifi. Within the FetchS3Object, I
> have created an AWSCredentialsProviderService which has the following
> properties:
>
> · Access Key: VPC A access key
>
> · Secret Key: VPC A secret key
>
> · Assume Role ARN: VPC B role
>
> · Assume Role Session Name: crossaccountrole
>
> · Session Time: 3600
> The general properties in the FetchS3Object are as follows:
>
> · Bucket: VPC B bucket name
>
> · Object: Filename of VPC B bucket object
>
> · Access Key: VPC A access key
>
> · Secret Key: VPC A secret key
>
> · AWS Credentials Provider Service: <see above>
>
> However, when this tries to run I get Access Denied. I've been going
> through the source code for Nifi and I'm not sure if short-lived tokens get
> passed through. Can anyone please provide me some guidance or suggestions
> on how to get this to work? J
>
> Best,
> Keren
> ________________________________________________________
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>
