subject:"Nifi Cross Account Download With A Profile Flag"

Re: Nifi Cross Account Download With A Profile Flag

2019-07-03 Thread Craig Insley

Hello,

This is an old thread but I'm having the same issue on 1.9.2. I have cross
account access with an assumed role working just fine with the aws command
line using a profile tag. However, no matter how I configure
my AWSCredentialsProviderControllerService I get errors.

Using the profile tag only I get a 403.
Using a combination of the profile tag for credentials and the Assume Role
settings, I get  "The security token included in the request is invalid."
Setting the credentials and Assume Role settings explicitly in the
controller service I get the same "The security token included in the
request is invalid."
Using a credentials file, I get the same "The security token included in
the request is invalid."

I do have an External ID. I really hope I am just missing something. Any
help would be appreciated!

Thanks,
Craig

On Thu, Sep 1, 2016 at 2:45 PM James Wing  wrote:

> Keren,
>
> I'm certain cross-account access can work in 0.6.0, I've done it.
>
> The timeout error calling sts:AssumeRole is not the same error you started
> with, right?  Earlier, the error was 403 "Access Denied", which would have
> been farther in the auth process.  Any idea what changed?  Are you using
> the HTTP proxy settings on FetchS3Object?
>
>
> Thanks,
>
> James
>
> On Thu, Sep 1, 2016 at 10:30 AM, Tseytlin, Keren <
> keren.tseyt...@capitalone.com> wrote:
>
> > Hey James,
> >
> > No problem. I’ve tried running a bunch of different ways to do it
> manually
> > to try and avoid the CLI profile flag. But it seems like that is the only
> > it works locally (without Nifi). My cross account role doesn’t require an
> > External ID, and that feature doesn’t exist in 0.6.0 either.
> >
> > An small portion of the logs are below. The logs show that it starts to
> > attempt to get the S3 object, it times out on assuming the
> > role/credentials, and then fails.
> >
> > at
> > com.amazonaws.auth.STSAssumeRoleSessionCredential
> > sProvider.startSession(STS
> > AssumeRoleSessionCredentialsProvider.java:272)
> > [aws-java-sdk-sts-1.10.32.jar:na]
> > at
> > com.amazonaws.auth.STSAssumeRoleSessionCredential
> > sProvider.getCredentials(S
> > TSAssumeRoleSessionCredentialsProvider.java:247)
> > [aws-java-sdk-sts-1.10.32.jar:na]
> > at
> > com.amazonaws.auth.STSAssumeRoleSessionCredential
> > sProvider.getCredentials(S
> > TSAssumeRoleSessionCredentialsProvider.java:34)
> > [aws-java-sdk-sts-1.10.32.jar:na]
> > at
> > com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.
> > invoke(A
> > WSSecurityTokenServiceClient.java:1098) [aws-java-sdk-sts-1.10.32.jar:na]
> > at
> > com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.
> > assumeRo
> > le(AWSSecurityTokenServiceClient.java:1000)
> > [aws-java-sdk-sts-1.10.32.jar:na]
> > at
> > com.amazonaws.auth.STSAssumeRoleSessionCredential
> > sProvider.startSession(STS
> > AssumeRoleSessionCredentialsProvider.java:272)
> > [aws-java-sdk-sts-1.10.32.jar:na]
> > at
> > com.amazonaws.auth.STSAssumeRoleSessionCredential
> > sProvider.getCredentials(S
> > TSAssumeRoleSessionCredentialsProvider.java:247)
> > [aws-java-sdk-sts-1.10.32.jar:na]
> > at
> > com.amazonaws.auth.STSAssumeRoleSessionCredential
> > sProvider.getCredentials(S
> > TSAssumeRoleSessionCredentialsProvider.java:34)
> > [aws-java-sdk-sts-1.10.32.jar:na]
> > at
> > com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3589)
> > [aws-java-sdk-s3-1.10.32.jar:na]
> > at
> > com.amazonaws.services.s3.AmazonS3Client.getObject(
> > AmazonS3Client.java:1116
> > ) [aws-java-sdk-s3-1.10.32.jar:na]
> > at
> > org.apache.nifi.processors.aws.s3.FetchS3Object.
> > onTrigger(FetchS3Object.jav
> > a:105) [nifi-aws-processors-0.6.0.1.2.0.0-91.jar:0.6.0.1.2.0.0-91]
> > at
> > org.apache.nifi.processor.AbstractProcessor.onTrigger(
> > AbstractProcessor.jav
> > a:27) [nifi-api-0.6.0.1.2.0.0-91.jar:0.6.0.1.2.0.0-91]
> >
> >
> > If you (or anyone scanning the thread) can think of a way to do this
> > without upgrading that’d be awesome. Otherwise, I’ll start motivating the
> > masses.
> >
> > Best,
> > Keren
> >
> > On 9/1/16, 12:17 PM, "James Wing"  wrote:
> >
> > >Keren,
> > >
> > >I'm sorry if my advice is a bit confusing, there have been some changes
> to
> > >AWS credentials over the last few versions.  NiFi 0.6.0 does not have
> the
> > >option to use a CLI profile in AWSCredentialsProviderControllerService,
> I
> > >think that was introduced in 0.7.0.
> > >
> > >Would it be possible to share some of the log entries?  Was there a
> stack
> > >trace associated with the timeout?
> > >
> > >Also, does your cross-account Role require an External ID?  I do not
> > >believe that is supported in 0.6.0, but can be required to assume some
> > >roles.
> > >
> > >
> > >Thanks,
> > >
> > >James
> > >
> > >On Thu, Sep 1, 2016 at 7:57 AM, Tseytlin, Keren <
> > >keren.tseyt...@capitalone.com> wrote:
> > >
> > >> Thanks for your responses!

Re: Nifi Cross Account Download With A Profile Flag

2016-09-01 Thread James Wing

Keren,

I'm certain cross-account access can work in 0.6.0, I've done it.

The timeout error calling sts:AssumeRole is not the same error you started
with, right?  Earlier, the error was 403 "Access Denied", which would have
been farther in the auth process.  Any idea what changed?  Are you using
the HTTP proxy settings on FetchS3Object?


Thanks,

James

On Thu, Sep 1, 2016 at 10:30 AM, Tseytlin, Keren <
keren.tseyt...@capitalone.com> wrote:

> Hey James,
>
> No problem. I’ve tried running a bunch of different ways to do it manually
> to try and avoid the CLI profile flag. But it seems like that is the only
> it works locally (without Nifi). My cross account role doesn’t require an
> External ID, and that feature doesn’t exist in 0.6.0 either.
>
> An small portion of the logs are below. The logs show that it starts to
> attempt to get the S3 object, it times out on assuming the
> role/credentials, and then fails.
>
> at
> com.amazonaws.auth.STSAssumeRoleSessionCredential
> sProvider.startSession(STS
> AssumeRoleSessionCredentialsProvider.java:272)
> [aws-java-sdk-sts-1.10.32.jar:na]
> at
> com.amazonaws.auth.STSAssumeRoleSessionCredential
> sProvider.getCredentials(S
> TSAssumeRoleSessionCredentialsProvider.java:247)
> [aws-java-sdk-sts-1.10.32.jar:na]
> at
> com.amazonaws.auth.STSAssumeRoleSessionCredential
> sProvider.getCredentials(S
> TSAssumeRoleSessionCredentialsProvider.java:34)
> [aws-java-sdk-sts-1.10.32.jar:na]
> at
> com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.
> invoke(A
> WSSecurityTokenServiceClient.java:1098) [aws-java-sdk-sts-1.10.32.jar:na]
> at
> com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.
> assumeRo
> le(AWSSecurityTokenServiceClient.java:1000)
> [aws-java-sdk-sts-1.10.32.jar:na]
> at
> com.amazonaws.auth.STSAssumeRoleSessionCredential
> sProvider.startSession(STS
> AssumeRoleSessionCredentialsProvider.java:272)
> [aws-java-sdk-sts-1.10.32.jar:na]
> at
> com.amazonaws.auth.STSAssumeRoleSessionCredential
> sProvider.getCredentials(S
> TSAssumeRoleSessionCredentialsProvider.java:247)
> [aws-java-sdk-sts-1.10.32.jar:na]
> at
> com.amazonaws.auth.STSAssumeRoleSessionCredential
> sProvider.getCredentials(S
> TSAssumeRoleSessionCredentialsProvider.java:34)
> [aws-java-sdk-sts-1.10.32.jar:na]
> at
> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3589)
> [aws-java-sdk-s3-1.10.32.jar:na]
> at
> com.amazonaws.services.s3.AmazonS3Client.getObject(
> AmazonS3Client.java:1116
> ) [aws-java-sdk-s3-1.10.32.jar:na]
> at
> org.apache.nifi.processors.aws.s3.FetchS3Object.
> onTrigger(FetchS3Object.jav
> a:105) [nifi-aws-processors-0.6.0.1.2.0.0-91.jar:0.6.0.1.2.0.0-91]
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(
> AbstractProcessor.jav
> a:27) [nifi-api-0.6.0.1.2.0.0-91.jar:0.6.0.1.2.0.0-91]
>
>
> If you (or anyone scanning the thread) can think of a way to do this
> without upgrading that’d be awesome. Otherwise, I’ll start motivating the
> masses.
>
> Best,
> Keren
>
> On 9/1/16, 12:17 PM, "James Wing"  wrote:
>
> >Keren,
> >
> >I'm sorry if my advice is a bit confusing, there have been some changes to
> >AWS credentials over the last few versions.  NiFi 0.6.0 does not have the
> >option to use a CLI profile in AWSCredentialsProviderControllerService, I
> >think that was introduced in 0.7.0.
> >
> >Would it be possible to share some of the log entries?  Was there a stack
> >trace associated with the timeout?
> >
> >Also, does your cross-account Role require an External ID?  I do not
> >believe that is supported in 0.6.0, but can be required to assume some
> >roles.
> >
> >
> >Thanks,
> >
> >James
> >
> >On Thu, Sep 1, 2016 at 7:57 AM, Tseytlin, Keren <
> >keren.tseyt...@capitalone.com> wrote:
> >
> >> Thanks for your responses!
> >>
> >> @James - we are on version 0.6.0. Using Hortonworks Data Flow 1.2.0.0.
> >>
> >> I¹ve set up debugging, and it shows me that it¹s trying to connect, but
> >>it
> >> times out on connecting. It would be awesome if it would also return the
> >> account ID of the credentials it is trying to use.
> >>
> >> Is there any way to see the exact keys/tokens that Nifi is trying to use
> >> to get the S3 object? I¹m not seeing it in the logs.
> >>
> >> I tried to set the Profile in Nifi, but it complains that it¹s not a
> >>valid
> >> property.
> >>
> >> Best,
> >> Keren
> >>
> >> On 8/31/16, 6:24 PM, "Andrew Grande"  wrote:
> >>
> >> >Debug logging can be set in a processor itself in the UI, too.
> >> >
> >> >On Wed, Aug 31, 2016, 5:34 PM James Wing  wrote:
> >> >
> >> >> Keren,
> >> >>
> >> >> Which version of NiFi are you using?
> >> >>
> >> >> One thing I noticed in your configuration of FetchS3Object is you are
> >> >> setting both the Access Key and Secret Key properties with the AWS
> >> >> Credentials Provider.  When you are using the AWS

Re: Nifi Cross Account Download With A Profile Flag

2016-09-01 Thread Tseytlin, Keren

Hey James,

No problem. I’ve tried running a bunch of different ways to do it manually
to try and avoid the CLI profile flag. But it seems like that is the only
it works locally (without Nifi). My cross account role doesn’t require an
External ID, and that feature doesn’t exist in 0.6.0 either.

An small portion of the logs are below. The logs show that it starts to
attempt to get the S3 object, it times out on assuming the
role/credentials, and then fails.

at 
com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.startSession(STS
AssumeRoleSessionCredentialsProvider.java:272)
[aws-java-sdk-sts-1.10.32.jar:na]
at 
com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.getCredentials(S
TSAssumeRoleSessionCredentialsProvider.java:247)
[aws-java-sdk-sts-1.10.32.jar:na]
at 
com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.getCredentials(S
TSAssumeRoleSessionCredentialsProvider.java:34)
[aws-java-sdk-sts-1.10.32.jar:na]
at 
com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(A
WSSecurityTokenServiceClient.java:1098) [aws-java-sdk-sts-1.10.32.jar:na]
at 
com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.assumeRo
le(AWSSecurityTokenServiceClient.java:1000)
[aws-java-sdk-sts-1.10.32.jar:na]
at 
com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.startSession(STS
AssumeRoleSessionCredentialsProvider.java:272)
[aws-java-sdk-sts-1.10.32.jar:na]
at 
com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.getCredentials(S
TSAssumeRoleSessionCredentialsProvider.java:247)
[aws-java-sdk-sts-1.10.32.jar:na]
at 
com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.getCredentials(S
TSAssumeRoleSessionCredentialsProvider.java:34)
[aws-java-sdk-sts-1.10.32.jar:na]
at 
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3589)
[aws-java-sdk-s3-1.10.32.jar:na]
at 
com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1116
) [aws-java-sdk-s3-1.10.32.jar:na]
at 
org.apache.nifi.processors.aws.s3.FetchS3Object.onTrigger(FetchS3Object.jav
a:105) [nifi-aws-processors-0.6.0.1.2.0.0-91.jar:0.6.0.1.2.0.0-91]
at 
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.jav
a:27) [nifi-api-0.6.0.1.2.0.0-91.jar:0.6.0.1.2.0.0-91]


If you (or anyone scanning the thread) can think of a way to do this
without upgrading that’d be awesome. Otherwise, I’ll start motivating the
masses. 

Best,
Keren

On 9/1/16, 12:17 PM, "James Wing"  wrote:

>Keren,
>
>I'm sorry if my advice is a bit confusing, there have been some changes to
>AWS credentials over the last few versions.  NiFi 0.6.0 does not have the
>option to use a CLI profile in AWSCredentialsProviderControllerService, I
>think that was introduced in 0.7.0.
>
>Would it be possible to share some of the log entries?  Was there a stack
>trace associated with the timeout?
>
>Also, does your cross-account Role require an External ID?  I do not
>believe that is supported in 0.6.0, but can be required to assume some
>roles.
>
>
>Thanks,
>
>James
>
>On Thu, Sep 1, 2016 at 7:57 AM, Tseytlin, Keren <
>keren.tseyt...@capitalone.com> wrote:
>
>> Thanks for your responses!
>>
>> @James - we are on version 0.6.0. Using Hortonworks Data Flow 1.2.0.0.
>>
>> I¹ve set up debugging, and it shows me that it¹s trying to connect, but
>>it
>> times out on connecting. It would be awesome if it would also return the
>> account ID of the credentials it is trying to use.
>>
>> Is there any way to see the exact keys/tokens that Nifi is trying to use
>> to get the S3 object? I¹m not seeing it in the logs.
>>
>> I tried to set the Profile in Nifi, but it complains that it¹s not a
>>valid
>> property.
>>
>> Best,
>> Keren
>>
>> On 8/31/16, 6:24 PM, "Andrew Grande"  wrote:
>>
>> >Debug logging can be set in a processor itself in the UI, too.
>> >
>> >On Wed, Aug 31, 2016, 5:34 PM James Wing  wrote:
>> >
>> >> Keren,
>> >>
>> >> Which version of NiFi are you using?
>> >>
>> >> One thing I noticed in your configuration of FetchS3Object is you are
>> >> setting both the Access Key and Secret Key properties with the AWS
>> >> Credentials Provider.  When you are using the AWS Credentials
>>Provider
>> >> Service, you should not specify keys.
>> >>
>> >> A more certainly helpful thing to do is enable debug logging for the
>>AWS
>> >> processor package by adding a line like the following to
>> >>conf/logback.xml:
>> >>
>> >> 
>> >>
>> >> With the debug logging enabled, there are messages indicating which
>> >> credential type is being attempted.  Your settings for the AWS
>> >>Credentials
>> >> Provider look appropriate.  The controller service is indeed
>>designed to
>> >> refresh the STS token automagically using the AWS SDK classes for
>> >>temporary
>> >> credentials.
>> >>
>> >> Last, you might experiment with configuring
>> >>

Re: Nifi Cross Account Download With A Profile Flag

2016-09-01 Thread James Wing

Keren,

I'm sorry if my advice is a bit confusing, there have been some changes to
AWS credentials over the last few versions.  NiFi 0.6.0 does not have the
option to use a CLI profile in AWSCredentialsProviderControllerService, I
think that was introduced in 0.7.0.

Would it be possible to share some of the log entries?  Was there a stack
trace associated with the timeout?

Also, does your cross-account Role require an External ID?  I do not
believe that is supported in 0.6.0, but can be required to assume some
roles.


Thanks,

James

On Thu, Sep 1, 2016 at 7:57 AM, Tseytlin, Keren <
keren.tseyt...@capitalone.com> wrote:

> Thanks for your responses!
>
> @James - we are on version 0.6.0. Using Hortonworks Data Flow 1.2.0.0.
>
> I¹ve set up debugging, and it shows me that it¹s trying to connect, but it
> times out on connecting. It would be awesome if it would also return the
> account ID of the credentials it is trying to use.
>
> Is there any way to see the exact keys/tokens that Nifi is trying to use
> to get the S3 object? I¹m not seeing it in the logs.
>
> I tried to set the Profile in Nifi, but it complains that it¹s not a valid
> property.
>
> Best,
> Keren
>
> On 8/31/16, 6:24 PM, "Andrew Grande"  wrote:
>
> >Debug logging can be set in a processor itself in the UI, too.
> >
> >On Wed, Aug 31, 2016, 5:34 PM James Wing  wrote:
> >
> >> Keren,
> >>
> >> Which version of NiFi are you using?
> >>
> >> One thing I noticed in your configuration of FetchS3Object is you are
> >> setting both the Access Key and Secret Key properties with the AWS
> >> Credentials Provider.  When you are using the AWS Credentials Provider
> >> Service, you should not specify keys.
> >>
> >> A more certainly helpful thing to do is enable debug logging for the AWS
> >> processor package by adding a line like the following to
> >>conf/logback.xml:
> >>
> >> 
> >>
> >> With the debug logging enabled, there are messages indicating which
> >> credential type is being attempted.  Your settings for the AWS
> >>Credentials
> >> Provider look appropriate.  The controller service is indeed designed to
> >> refresh the STS token automagically using the AWS SDK classes for
> >>temporary
> >> credentials.
> >>
> >> Last, you might experiment with configuring
> >> AWSCredentialsProviderControllerService to use your named CLI profile
> >> "crossaccountrole", which should also work.
> >>
> >> Thanks,
> >>
> >> James
> >>
> >> On Wed, Aug 31, 2016 at 1:44 PM, Tseytlin, Keren <
> >> keren.tseyt...@capitalone.com> wrote:
> >>
> >> > Hi All!
> >> >
> >> > Looking for some help on enabling Cross Account communication within
> >> Nifi!
> >> >
> >> > My goal: There are files stored from CloudTrail in an S3 bucket in
> >>VPC B.
> >> > My Nifi machines are in VPC A. I want Nifi to be able to get those
> >>files
> >> > from VPC B. VPC A and VPC B need to be communicating in the
> >>FetchS3Object
> >> > component.
> >> >
> >> > See this link for some additional info: http://docs.aws.amazon.com/
> >> >
> >>awscloudtrail/latest/userguide/cloudtrail-sharing-logs-assume-role.html
> >> >
> >> > I have communication working manually on the Nifi machines in VPC A
> >>when
> >> I
> >> > use the AWS CLI. The process is as follows:
> >> >
> >> > 1. Run sts -assume-role on my Nifi machine (VPC A) to assume a
> >>role
> >> > I've created in VPC B that is configured to have access to the S3
> >>bucket
> >> in
> >> > VPC B.
> >> >
> >> > 2. This will generate temporary keys that need to be refreshed
> >>every
> >> > hour. There is no way to have assume role create permanent keys.
> >>Export
> >> the
> >> > keys as environment variables.
> >> >
> >> > 3. Set up ~/.aws/config to have a profile "crossaccountrole" that
> >> > connects to the arn of the role created in VPC B.
> >> >
> >> > 4. Run the following command à "aws s3 cp s3://
> >> >> > name locally> --profile crossaccountrole"
> >> >
> >> > Most importantly, if I ever try to run this without the --profile
> >>flag,
> >> > then it will not allow me to download the file.  It seems like
> >>perhaps to
> >> > get it to work with Nifi I need a place to pass in the profile that
> >>needs
> >> > to be used in order for the communication to work.
> >> >
> >> > I've been trying to implement this in Nifi. Within the FetchS3Object,
> >>I
> >> > have created an AWSCredentialsProviderService which has the following
> >> > properties:
> >> >
> >> > ·  Access Key: VPC A access key
> >> >
> >> > ·  Secret Key: VPC A secret key
> >> >
> >> > ·  Assume Role ARN: VPC B role
> >> >
> >> > ·  Assume Role Session Name: crossaccountrole
> >> >
> >> > ·  Session Time: 3600
> >> > The general properties in the FetchS3Object are as follows:
> >> >
> >> > ·  Bucket: VPC B bucket name
> >> >
> >> > ·  Object: Filename of VPC B bucket object
> >> >
> >> > ·  Access Key: VPC A access key
> >> >
> >> > ·  Secret Key: VPC A secret key
> >> >
> >> >

Re: Nifi Cross Account Download With A Profile Flag

2016-09-01 Thread Tseytlin, Keren

Thanks for your responses!

@James - we are on version 0.6.0. Using Hortonworks Data Flow 1.2.0.0.

I¹ve set up debugging, and it shows me that it¹s trying to connect, but it
times out on connecting. It would be awesome if it would also return the
account ID of the credentials it is trying to use.

Is there any way to see the exact keys/tokens that Nifi is trying to use
to get the S3 object? I¹m not seeing it in the logs.

I tried to set the Profile in Nifi, but it complains that it¹s not a valid
property.

Best,
Keren

On 8/31/16, 6:24 PM, "Andrew Grande"  wrote:

>Debug logging can be set in a processor itself in the UI, too.
>
>On Wed, Aug 31, 2016, 5:34 PM James Wing  wrote:
>
>> Keren,
>>
>> Which version of NiFi are you using?
>>
>> One thing I noticed in your configuration of FetchS3Object is you are
>> setting both the Access Key and Secret Key properties with the AWS
>> Credentials Provider.  When you are using the AWS Credentials Provider
>> Service, you should not specify keys.
>>
>> A more certainly helpful thing to do is enable debug logging for the AWS
>> processor package by adding a line like the following to
>>conf/logback.xml:
>>
>> 
>>
>> With the debug logging enabled, there are messages indicating which
>> credential type is being attempted.  Your settings for the AWS
>>Credentials
>> Provider look appropriate.  The controller service is indeed designed to
>> refresh the STS token automagically using the AWS SDK classes for
>>temporary
>> credentials.
>>
>> Last, you might experiment with configuring
>> AWSCredentialsProviderControllerService to use your named CLI profile
>> "crossaccountrole", which should also work.
>>
>> Thanks,
>>
>> James
>>
>> On Wed, Aug 31, 2016 at 1:44 PM, Tseytlin, Keren <
>> keren.tseyt...@capitalone.com> wrote:
>>
>> > Hi All!
>> >
>> > Looking for some help on enabling Cross Account communication within
>> Nifi!
>> >
>> > My goal: There are files stored from CloudTrail in an S3 bucket in
>>VPC B.
>> > My Nifi machines are in VPC A. I want Nifi to be able to get those
>>files
>> > from VPC B. VPC A and VPC B need to be communicating in the
>>FetchS3Object
>> > component.
>> >
>> > See this link for some additional info: http://docs.aws.amazon.com/
>> > 
>>awscloudtrail/latest/userguide/cloudtrail-sharing-logs-assume-role.html
>> >
>> > I have communication working manually on the Nifi machines in VPC A
>>when
>> I
>> > use the AWS CLI. The process is as follows:
>> >
>> > 1. Run sts -assume-role on my Nifi machine (VPC A) to assume a
>>role
>> > I've created in VPC B that is configured to have access to the S3
>>bucket
>> in
>> > VPC B.
>> >
>> > 2. This will generate temporary keys that need to be refreshed
>>every
>> > hour. There is no way to have assume role create permanent keys.
>>Export
>> the
>> > keys as environment variables.
>> >
>> > 3. Set up ~/.aws/config to have a profile "crossaccountrole" that
>> > connects to the arn of the role created in VPC B.
>> >
>> > 4. Run the following command à "aws s3 cp s3://
>>> > name locally> --profile crossaccountrole"
>> >
>> > Most importantly, if I ever try to run this without the --profile
>>flag,
>> > then it will not allow me to download the file.  It seems like
>>perhaps to
>> > get it to work with Nifi I need a place to pass in the profile that
>>needs
>> > to be used in order for the communication to work.
>> >
>> > I've been trying to implement this in Nifi. Within the FetchS3Object,
>>I
>> > have created an AWSCredentialsProviderService which has the following
>> > properties:
>> >
>> > ·  Access Key: VPC A access key
>> >
>> > ·  Secret Key: VPC A secret key
>> >
>> > ·  Assume Role ARN: VPC B role
>> >
>> > ·  Assume Role Session Name: crossaccountrole
>> >
>> > ·  Session Time: 3600
>> > The general properties in the FetchS3Object are as follows:
>> >
>> > ·  Bucket: VPC B bucket name
>> >
>> > ·  Object: Filename of VPC B bucket object
>> >
>> > ·  Access Key: VPC A access key
>> >
>> > ·  Secret Key: VPC A secret key
>> >
>> > ·  AWS Credentials Provider Service: 
>> >
>> > However, when this tries to run I get Access Denied. I've been going
>> > through the source code for Nifi and I'm not sure if short-lived
>>tokens
>> get
>> > passed through. Can anyone please provide me some guidance or
>>suggestions
>> > on how to get this to work? J
>> >
>> > Best,
>> > Keren
>> > 
>> >
>> > The information contained in this e-mail is confidential and/or
>> > proprietary to Capital One and/or its affiliates and may only be used
>> > solely in performance of work or services for Capital One. The
>> information
>> > transmitted herewith is intended only for use by the individual or
>>entity
>> > to which it is addressed. If the reader of this message is not the
>> intended
>> > recipient, you are hereby notified that any review, retransmission,
>> >

Re: Nifi Cross Account Download With A Profile Flag

2016-08-31 Thread Andrew Grande

Debug logging can be set in a processor itself in the UI, too.

On Wed, Aug 31, 2016, 5:34 PM James Wing  wrote:

> Keren,
>
> Which version of NiFi are you using?
>
> One thing I noticed in your configuration of FetchS3Object is you are
> setting both the Access Key and Secret Key properties with the AWS
> Credentials Provider.  When you are using the AWS Credentials Provider
> Service, you should not specify keys.
>
> A more certainly helpful thing to do is enable debug logging for the AWS
> processor package by adding a line like the following to conf/logback.xml:
>
> 
>
> With the debug logging enabled, there are messages indicating which
> credential type is being attempted.  Your settings for the AWS Credentials
> Provider look appropriate.  The controller service is indeed designed to
> refresh the STS token automagically using the AWS SDK classes for temporary
> credentials.
>
> Last, you might experiment with configuring
> AWSCredentialsProviderControllerService to use your named CLI profile
> "crossaccountrole", which should also work.
>
> Thanks,
>
> James
>
> On Wed, Aug 31, 2016 at 1:44 PM, Tseytlin, Keren <
> keren.tseyt...@capitalone.com> wrote:
>
> > Hi All!
> >
> > Looking for some help on enabling Cross Account communication within
> Nifi!
> >
> > My goal: There are files stored from CloudTrail in an S3 bucket in VPC B.
> > My Nifi machines are in VPC A. I want Nifi to be able to get those files
> > from VPC B. VPC A and VPC B need to be communicating in the FetchS3Object
> > component.
> >
> > See this link for some additional info: http://docs.aws.amazon.com/
> > awscloudtrail/latest/userguide/cloudtrail-sharing-logs-assume-role.html
> >
> > I have communication working manually on the Nifi machines in VPC A when
> I
> > use the AWS CLI. The process is as follows:
> >
> > 1. Run sts -assume-role on my Nifi machine (VPC A) to assume a role
> > I've created in VPC B that is configured to have access to the S3 bucket
> in
> > VPC B.
> >
> > 2. This will generate temporary keys that need to be refreshed every
> > hour. There is no way to have assume role create permanent keys. Export
> the
> > keys as environment variables.
> >
> > 3. Set up ~/.aws/config to have a profile "crossaccountrole" that
> > connects to the arn of the role created in VPC B.
> >
> > 4. Run the following command à "aws s3 cp s3://  > name locally> --profile crossaccountrole"
> >
> > Most importantly, if I ever try to run this without the --profile flag,
> > then it will not allow me to download the file.  It seems like perhaps to
> > get it to work with Nifi I need a place to pass in the profile that needs
> > to be used in order for the communication to work.
> >
> > I've been trying to implement this in Nifi. Within the FetchS3Object, I
> > have created an AWSCredentialsProviderService which has the following
> > properties:
> >
> > ·  Access Key: VPC A access key
> >
> > ·  Secret Key: VPC A secret key
> >
> > ·  Assume Role ARN: VPC B role
> >
> > ·  Assume Role Session Name: crossaccountrole
> >
> > ·  Session Time: 3600
> > The general properties in the FetchS3Object are as follows:
> >
> > ·  Bucket: VPC B bucket name
> >
> > ·  Object: Filename of VPC B bucket object
> >
> > ·  Access Key: VPC A access key
> >
> > ·  Secret Key: VPC A secret key
> >
> > ·  AWS Credentials Provider Service: 
> >
> > However, when this tries to run I get Access Denied. I've been going
> > through the source code for Nifi and I'm not sure if short-lived tokens
> get
> > passed through. Can anyone please provide me some guidance or suggestions
> > on how to get this to work? J
> >
> > Best,
> > Keren
> > 
> >
> > The information contained in this e-mail is confidential and/or
> > proprietary to Capital One and/or its affiliates and may only be used
> > solely in performance of work or services for Capital One. The
> information
> > transmitted herewith is intended only for use by the individual or entity
> > to which it is addressed. If the reader of this message is not the
> intended
> > recipient, you are hereby notified that any review, retransmission,
> > dissemination, distribution, copying or other use of, or taking of any
> > action in reliance upon this information is strictly prohibited. If you
> > have received this communication in error, please contact the sender and
> > delete the material from your computer.
> >
>

Re: Nifi Cross Account Download With A Profile Flag

2016-08-31 Thread James Wing

Keren,

Which version of NiFi are you using?

One thing I noticed in your configuration of FetchS3Object is you are
setting both the Access Key and Secret Key properties with the AWS
Credentials Provider.  When you are using the AWS Credentials Provider
Service, you should not specify keys.

A more certainly helpful thing to do is enable debug logging for the AWS
processor package by adding a line like the following to conf/logback.xml:



With the debug logging enabled, there are messages indicating which
credential type is being attempted.  Your settings for the AWS Credentials
Provider look appropriate.  The controller service is indeed designed to
refresh the STS token automagically using the AWS SDK classes for temporary
credentials.

Last, you might experiment with configuring
AWSCredentialsProviderControllerService to use your named CLI profile
"crossaccountrole", which should also work.

Thanks,

James

On Wed, Aug 31, 2016 at 1:44 PM, Tseytlin, Keren <
keren.tseyt...@capitalone.com> wrote:

> Hi All!
>
> Looking for some help on enabling Cross Account communication within Nifi!
>
> My goal: There are files stored from CloudTrail in an S3 bucket in VPC B.
> My Nifi machines are in VPC A. I want Nifi to be able to get those files
> from VPC B. VPC A and VPC B need to be communicating in the FetchS3Object
> component.
>
> See this link for some additional info: http://docs.aws.amazon.com/
> awscloudtrail/latest/userguide/cloudtrail-sharing-logs-assume-role.html
>
> I have communication working manually on the Nifi machines in VPC A when I
> use the AWS CLI. The process is as follows:
>
> 1. Run sts -assume-role on my Nifi machine (VPC A) to assume a role
> I've created in VPC B that is configured to have access to the S3 bucket in
> VPC B.
>
> 2. This will generate temporary keys that need to be refreshed every
> hour. There is no way to have assume role create permanent keys. Export the
> keys as environment variables.
>
> 3. Set up ~/.aws/config to have a profile "crossaccountrole" that
> connects to the arn of the role created in VPC B.
>
> 4. Run the following command à "aws s3 cp s3://  name locally> --profile crossaccountrole"
>
> Most importantly, if I ever try to run this without the --profile flag,
> then it will not allow me to download the file.  It seems like perhaps to
> get it to work with Nifi I need a place to pass in the profile that needs
> to be used in order for the communication to work.
>
> I've been trying to implement this in Nifi. Within the FetchS3Object, I
> have created an AWSCredentialsProviderService which has the following
> properties:
>
> ·  Access Key: VPC A access key
>
> ·  Secret Key: VPC A secret key
>
> ·  Assume Role ARN: VPC B role
>
> ·  Assume Role Session Name: crossaccountrole
>
> ·  Session Time: 3600
> The general properties in the FetchS3Object are as follows:
>
> ·  Bucket: VPC B bucket name
>
> ·  Object: Filename of VPC B bucket object
>
> ·  Access Key: VPC A access key
>
> ·  Secret Key: VPC A secret key
>
> ·  AWS Credentials Provider Service: 
>
> However, when this tries to run I get Access Denied. I've been going
> through the source code for Nifi and I'm not sure if short-lived tokens get
> passed through. Can anyone please provide me some guidance or suggestions
> on how to get this to work? J
>
> Best,
> Keren
> 
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>

Re: Nifi Cross Account Download With A Profile Flag

2016-08-31 Thread Tseytlin, Keren

Oops, I sent this out to the developer list serve, will send it out to the
Users list serve now.

My bad for flooding your inbox (but if you have suggestions please send
them my way)!

Best,
Keren

On 8/31/16, 4:44 PM, "Tseytlin, Keren" 
wrote:

>Hi All!
>
>Looking for some help on enabling Cross Account communication within Nifi!
>
>My goal: There are files stored from CloudTrail in an S3 bucket in VPC B.
>My Nifi machines are in VPC A. I want Nifi to be able to get those files
>from VPC B. VPC A and VPC B need to be communicating in the FetchS3Object
>component.
>
>See this link for some additional info:
>http://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-shari
>ng-logs-assume-role.html
>
>I have communication working manually on the Nifi machines in VPC A when
>I use the AWS CLI. The process is as follows:
>
>1. Run sts -assume-role on my Nifi machine (VPC A) to assume a role
>I've created in VPC B that is configured to have access to the S3 bucket
>in VPC B.
>
>2. This will generate temporary keys that need to be refreshed every
>hour. There is no way to have assume role create permanent keys. Export
>the keys as environment variables.
>
>3. Set up ~/.aws/config to have a profile "crossaccountrole" that
>connects to the arn of the role created in VPC B.
>
>4. Run the following command à "aws s3 cp s3:// name locally> --profile crossaccountrole"
>
>Most importantly, if I ever try to run this without the --profile flag,
>then it will not allow me to download the file.  It seems like perhaps to
>get it to work with Nifi I need a place to pass in the profile that needs
>to be used in order for the communication to work.
>
>I've been trying to implement this in Nifi. Within the FetchS3Object, I
>have created an AWSCredentialsProviderService which has the following
>properties:
>
>·  Access Key: VPC A access key
>
>·  Secret Key: VPC A secret key
>
>·  Assume Role ARN: VPC B role
>
>·  Assume Role Session Name: crossaccountrole
>
>·  Session Time: 3600
>The general properties in the FetchS3Object are as follows:
>
>·  Bucket: VPC B bucket name
>
>·  Object: Filename of VPC B bucket object
>
>·  Access Key: VPC A access key
>
>·  Secret Key: VPC A secret key
>
>·  AWS Credentials Provider Service: 
>
>However, when this tries to run I get Access Denied. I've been going
>through the source code for Nifi and I'm not sure if short-lived tokens
>get passed through. Can anyone please provide me some guidance or
>suggestions on how to get this to work? J
>
>Best,
>Keren
>
>
>The information contained in this e-mail is confidential and/or
>proprietary to Capital One and/or its affiliates and may only be used
>solely in performance of work or services for Capital One. The
>information transmitted herewith is intended only for use by the
>individual or entity to which it is addressed. If the reader of this
>message is not the intended recipient, you are hereby notified that any
>review, retransmission, dissemination, distribution, copying or other use
>of, or taking of any action in reliance upon this information is strictly
>prohibited. If you have received this communication in error, please
>contact the sender and delete the material from your computer.



The information contained in this e-mail is confidential and/or proprietary to 
Capital One and/or its affiliates and may only be used solely in performance of 
work or services for Capital One. The information transmitted herewith is 
intended only for use by the individual or entity to which it is addressed. If 
the reader of this message is not the intended recipient, you are hereby 
notified that any review, retransmission, dissemination, distribution, copying 
or other use of, or taking of any action in reliance upon this information is 
strictly prohibited. If you have received this communication in error, please 
contact the sender and delete the material from your computer.

Nifi Cross Account Download With A Profile Flag

2016-08-31 Thread Tseytlin, Keren

Hi All!

Looking for some help on enabling Cross Account communication within Nifi!

My goal: There are files stored from CloudTrail in an S3 bucket in VPC B. My
Nifi machines are in VPC A. I want Nifi to be able to get those files from VPC
B. VPC A and VPC B need to be communicating in the FetchS3Object component.

See this link for some additional info:
http://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-sharing-logs-assume-role.html

I have communication working manually on the Nifi machines in VPC A when I use
the AWS CLI. The process is as follows:

1. Run sts -assume-role on my Nifi machine (VPC A) to assume a role I've
created in VPC B that is configured to have access to the S3 bucket in VPC B.

2. This will generate temporary keys that need to be refreshed every hour.
There is no way to have assume role create permanent keys. Export the keys as
environment variables.

3. Set up ~/.aws/config to have a profile "crossaccountrole" that connects
to the arn of the role created in VPC B.

4. Run the following command à "aws s3 cp s3:// --profile crossaccountrole"

Most importantly, if I ever try to run this without the --profile flag, then it
will not allow me to download the file. It seems like perhaps to get it to
work with Nifi I need a place to pass in the profile that needs to be used in
order for the communication to work.

I've been trying to implement this in Nifi. Within the FetchS3Object, I have
created an AWSCredentialsProviderService which has the following properties:

· Access Key: VPC A access key

· Secret Key: VPC A secret key

· Assume Role ARN: VPC B role

· Assume Role Session Name: crossaccountrole

· Session Time: 3600
The general properties in the FetchS3Object are as follows:

· Bucket: VPC B bucket name

· Object: Filename of VPC B bucket object

· Access Key: VPC A access key

· Secret Key: VPC A secret key

· AWS Credentials Provider Service:

However, when this tries to run I get Access Denied. I've been going through
the source code for Nifi and I'm not sure if short-lived tokens get passed
through. Can anyone please provide me some guidance or suggestions on how to
get this to work? J

Best,
Keren

The information contained in this e-mail is confidential and/or proprietary to
Capital One and/or its affiliates and may only be used solely in performance of
work or services for Capital One. The information transmitted herewith is
intended only for use by the individual or entity to which it is addressed. If
the reader of this message is not the intended recipient, you are hereby
notified that any review, retransmission, dissemination, distribution, copying
or other use of, or taking of any action in reliance upon this information is
strictly prohibited. If you have received this communication in error, please
contact the sender and delete the material from your computer.

Re: Nifi Cross Account Download With A Profile Flag

Re: Nifi Cross Account Download With A Profile Flag

Re: Nifi Cross Account Download With A Profile Flag

Re: Nifi Cross Account Download With A Profile Flag

Re: Nifi Cross Account Download With A Profile Flag

Re: Nifi Cross Account Download With A Profile Flag

Re: Nifi Cross Account Download With A Profile Flag

Re: Nifi Cross Account Download With A Profile Flag

Nifi Cross Account Download With A Profile Flag

9 matches

Site Navigation

Mail list logo

Footer information