Debug logging can be set in a processor itself in the UI, too. On Wed, Aug 31, 2016, 5:34 PM James Wing <[email protected]> wrote:
> Keren, > > Which version of NiFi are you using? > > One thing I noticed in your configuration of FetchS3Object is you are > setting both the Access Key and Secret Key properties with the AWS > Credentials Provider. When you are using the AWS Credentials Provider > Service, you should not specify keys. > > A more certainly helpful thing to do is enable debug logging for the AWS > processor package by adding a line like the following to conf/logback.xml: > > <logger name="org.apache.nifi.processors.aws" level="DEBUG"/> > > With the debug logging enabled, there are messages indicating which > credential type is being attempted. Your settings for the AWS Credentials > Provider look appropriate. The controller service is indeed designed to > refresh the STS token automagically using the AWS SDK classes for temporary > credentials. > > Last, you might experiment with configuring > AWSCredentialsProviderControllerService to use your named CLI profile > "crossaccountrole", which should also work. > > Thanks, > > James > > On Wed, Aug 31, 2016 at 1:44 PM, Tseytlin, Keren < > [email protected]> wrote: > > > Hi All! > > > > Looking for some help on enabling Cross Account communication within > Nifi! > > > > My goal: There are files stored from CloudTrail in an S3 bucket in VPC B. > > My Nifi machines are in VPC A. I want Nifi to be able to get those files > > from VPC B. VPC A and VPC B need to be communicating in the FetchS3Object > > component. > > > > See this link for some additional info: http://docs.aws.amazon.com/ > > awscloudtrail/latest/userguide/cloudtrail-sharing-logs-assume-role.html > > > > I have communication working manually on the Nifi machines in VPC A when > I > > use the AWS CLI. The process is as follows: > > > > 1. Run sts -assume-role on my Nifi machine (VPC A) to assume a role > > I've created in VPC B that is configured to have access to the S3 bucket > in > > VPC B. > > > > 2. This will generate temporary keys that need to be refreshed every > > hour. There is no way to have assume role create permanent keys. Export > the > > keys as environment variables. > > > > 3. Set up ~/.aws/config to have a profile "crossaccountrole" that > > connects to the arn of the role created in VPC B. > > > > 4. Run the following command à "aws s3 cp s3://<object path> <object > > name locally> --profile crossaccountrole" > > > > Most importantly, if I ever try to run this without the --profile flag, > > then it will not allow me to download the file. It seems like perhaps to > > get it to work with Nifi I need a place to pass in the profile that needs > > to be used in order for the communication to work. > > > > I've been trying to implement this in Nifi. Within the FetchS3Object, I > > have created an AWSCredentialsProviderService which has the following > > properties: > > > > · Access Key: VPC A access key > > > > · Secret Key: VPC A secret key > > > > · Assume Role ARN: VPC B role > > > > · Assume Role Session Name: crossaccountrole > > > > · Session Time: 3600 > > The general properties in the FetchS3Object are as follows: > > > > · Bucket: VPC B bucket name > > > > · Object: Filename of VPC B bucket object > > > > · Access Key: VPC A access key > > > > · Secret Key: VPC A secret key > > > > · AWS Credentials Provider Service: <see above> > > > > However, when this tries to run I get Access Denied. I've been going > > through the source code for Nifi and I'm not sure if short-lived tokens > get > > passed through. Can anyone please provide me some guidance or suggestions > > on how to get this to work? J > > > > Best, > > Keren > > ________________________________________________________ > > > > The information contained in this e-mail is confidential and/or > > proprietary to Capital One and/or its affiliates and may only be used > > solely in performance of work or services for Capital One. The > information > > transmitted herewith is intended only for use by the individual or entity > > to which it is addressed. If the reader of this message is not the > intended > > recipient, you are hereby notified that any review, retransmission, > > dissemination, distribution, copying or other use of, or taking of any > > action in reliance upon this information is strictly prohibited. If you > > have received this communication in error, please contact the sender and > > delete the material from your computer. > > >
