Re: Nifi 1.0.0 breaking changes?

[Joe Witt]

Devin,

You might find this helpful.  We try to maintain it for each release.

  https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance

Please let us know if you run into something you find wasn't here.  We
want to make it as accurate as possible but across more than 500 JIRAs
I am a bit skeptical it is complete.

Thanks
JOe

On Fri, Sep 2, 2016 at 5:00 AM, Bryan Bende  wrote:
> Devin,
>
> I know there are JIRAs in 1.0.0 that are marked with a "Migration" label,
> those are ones that might impact upgrading from 0.x.
>
> I'm not sure there is something that explains every change to the API.
> Is there a particular area you are interested in? are you talking about
> custom processors developed against 0.x?
>
> -Bryan
>
> On Wed, Aug 31, 2016 at 7:21 PM, Devin Fisher <
> devin.fis...@perfectsearchcorp.com> wrote:
>
>> Based on Version Scheme and API Compatibility[1] the recently released 1.0
>> (congratulations to all) could have some breaking changes for code
>> written against 0.* releases.  I was wondering if there are any resources
>> about what has broken with the new release? I looked at the release notes
>> but there is nothing there that looked like it would break anything.
>>
>> [1] :
>> https://cwiki.apache.org/confluence/display/NIFI/Version+Scheme+and+API+
>> Compatibility
>>

Re: PostHTTP Penalize file on HTTP 5xx response

[McDermott, Chris Kevin (MSDU - STaTS/StorefrontRemote)]

Hmmm, given this I wonder penalizing the flow file is going to help.  I’d like 
maintain delivery order as best I can.  If the web service is having 
intermittent problems some files might be penalized but others, added to the 
flow later, don’t get penalized and are sent out of order.

It might be better to yield the processor.  That would solve the out of order 
problem.  However, since the URL supports the EL a single processor could be 
talking to multiple web-services and yielding the processor could penalize 
files that destined for web services that are not having problems.Maybe 
that is OK though, since using a single processor for multiple web services is 
probably a corner case and routing to multiple PostHTTP processors could be 
used to handle such a case.

Chris McDermott
 
Remote Business Analytics
STaTS/StoreFront Remote
HPE Storage
Hewlett Packard Enterprise
Mobile: +1 978-697-5315
 


On 8/31/16, 11:28 PM, "Joe Witt"  wrote:

It will not be blocked by penalized things.  The queues are setup to
basically put those aside and move on to other things until their
penalty period passes. If you're seeing different behavior please
advise.

Thanks
Joe

On Thu, Sep 1, 2016 at 1:11 PM, McDermott, Chris Kevin (MSDU -
STaTS/StorefrontRemote)  wrote:
> Thanks, everyone for the feedback. I’ll file a JIRA for this and see if I 
can find some time to address it.
>
> Does anyone have any thoughts on my related question?
>
> (with spelling and grammar corrections:)
>
> ➢ If a penalized file is routed to a funnel that’s s connect to a 
processor via a connection with the OldestFlowFileFirst  prioritizer will the 
consumption of files from that connection be blocked until penalization period 
is over?
>
>
>
> Chris McDermott
>
> Remote Business Analytics
> STaTS/StoreFront Remote
> HPE Storage
> Hewlett Packard Enterprise
> Mobile: +1 978-697-5315
>
>
>
> On 8/31/16, 11:00 PM, "Matt Burgess"  wrote:
>
> Adam,
>
> A PenalizeFlowFile processor could be pretty useful, please feel free
> to file a New Feature Jira for this if you like.
>
> In the meantime you could use ExecuteScript (with Groovy for this
> example) and the following:
>
> def flowFile = session.get()
> if(!flowFile) return
> flowFile = session.penalize(flowFile)
> session.transfer(flowFile, REL_SUCCESS)
>
> In this case the "success" relationship is awkward, it means you
> successfully penalized the flow file. But then you can route it
> back/forward to the appropriate processor. If you create a template
> from this single processor, then dragging the template onto the canvas
> is somewhat equivalent to dragging a "PenalizeFlowFile" processor onto
> the canvas (meaning I suggest the template is named PenalizeFlowFile).
>
> Regards,
> Matt
>
> On Wed, Aug 31, 2016 at 10:46 PM, Adam Taft  wrote:
> > In the wild west of HTTP response codes, a 500 Server Error could 
mean
> > practically anything.  In my experience, you can't infer any 
semantic
> > meaning for what a 500 status code could mean, unless you're very 
familiar
> > with the server application.
> >
> > I'd even go so far as to suggest, if a modification is made to 
PostHTTP,
> > that all non-200 response codes should be penalized.  The dataflow 
manager
> > can always adjust the penalization timeout towards zero if a 
processing
> > delay is not warranted.
> >
> > Unrelated, but this also reminds me, we really need a 
PenalizeFlowFile
> > processor, which would allow a dataflow manager to penalize a 
flowfile
> > anywhere that is deemed necessary, even if other processors haven't 
done so
> > (have routed to success).
> >
> >
> > On Wed, Aug 31, 2016 at 1:54 PM, Andrew Grande  
wrote:
> >
> >> Wasn't HTTP 400 Bad Request meant for that? 500 only means the 
server
> >> failed, not necessarily due to user input.
> >>
> >> Andrew
> >>
> >> On Wed, Aug 31, 2016, 10:16 AM Mark Payne  
wrote:
> >>
> >> > Hey Chris,
> >> >
> >> > I think it is reasonable to penalize when we receive a 500 
response. 500
> >> > means Internal Server Error, and it is
> >> > very reasonable to believe that the Internal Server Error 
occurred due to
> >> > the specific input (i.e., that it may not
> >> > always occur with different input). So penalizing the FlowFile 
so that it
> >> > can be retried after a little bit is reasonable

Re: Back Pressure Indicators?

[Bryan Bende]

Peter,

This is something that has come up before, but hasn't been addressed yet.

https://issues.apache.org/jira/browse/NIFI-766

-Bryan

On Thu, Sep 1, 2016 at 10:25 AM, Peter Wicks (pwicks) 
wrote:

> One of the things I've been noticing that doesn't seem to have any status
> indicator for is back pressure.  When back pressure kicks in I usually
> notice because I start thinking that NiFi is being buggy or is broken, "Why
> aren't these Flow Files moving through the queue?".  Then I'll notice the
> GB's of data in the downstream queue and it will click.
>
> Am I missing it or is there no status indicator showing that Back Pressure
> is in action?
>

Re: Nifi 1.0.0 breaking changes?

[Bryan Bende]

Devin,

I know there are JIRAs in 1.0.0 that are marked with a "Migration" label,
those are ones that might impact upgrading from 0.x.

I'm not sure there is something that explains every change to the API.
Is there a particular area you are interested in? are you talking about
custom processors developed against 0.x?

-Bryan

On Wed, Aug 31, 2016 at 7:21 PM, Devin Fisher <
devin.fis...@perfectsearchcorp.com> wrote:

> Based on Version Scheme and API Compatibility[1] the recently released 1.0
> (congratulations to all) could have some breaking changes for code
> written against 0.* releases.  I was wondering if there are any resources
> about what has broken with the new release? I looked at the release notes
> but there is nothing there that looked like it would break anything.
>
> [1] :
> https://cwiki.apache.org/confluence/display/NIFI/Version+Scheme+and+API+
> Compatibility
>

Re: Apache NiFi - 3 tier Architecture

[Andrew Grande]

Nishad,

You can split NiFi clusters at will and connect them securely. Take a look
at Remote Process Group and an underlying site-to-site protocol.

In fact, linking data centers and layers is one of the mainstream NiFi use
cases.

Andrew

On Thu, Sep 1, 2016, 1:05 PM Nishad Hameed  wrote:

> Hi All,
>
> We are planning to use the "Apache NiFi" for our data flow and scheduling
> service. But as we have 3 tier architecture, we need to see if we can split
> NiFi.
>
> Like
>
> 1.   Web/Presentation layer
>
> 2.   Application layer
>
> 3.   Data Layer
>
> Is there any project already implemented this way?
>
> Waiting for your feedback.
>
> Thanks & Regards
> Nishad C Hameed
>
> MACBIS/T-MSIS
> M:+1(301) 335-5346
> L:+1(301) 977-7884 x 681
>
>
> This electronic mail (including any attachments) may contain information
> that is privileged, confidential, and/or otherwise protected from
> disclosure to anyone other than its intended recipient(s). Any
> dissemination or use of this electronic email or its contents (including
> any attachments) by persons other than the intended recipient(s) is
> strictly prohibited. If you have received this message in error, please
> notify the sender by reply email and delete the original message (including
> any attachments) in its entirety.
>

Re: Apache NiFi - 3 tier Architecture

[Bryan Bende]

Hello,

I'm not sure if this answers your question, but Apache NiFi is a single
application that can only be run the way it is packaged in the distribution.

It happens to run an embedded web server which is Jetty, but from the user
perspective you just launch a Java application.
There aren't different parts that can be broken apart and run separately.

-Bryan

On Thu, Sep 1, 2016 at 11:04 AM, Nishad Hameed 
wrote:

> Hi All,
>
> We are planning to use the "Apache NiFi" for our data flow and scheduling
> service. But as we have 3 tier architecture, we need to see if we can split
> NiFi.
>
> Like
>
> 1.   Web/Presentation layer
>
> 2.   Application layer
>
> 3.   Data Layer
>
> Is there any project already implemented this way?
>
> Waiting for your feedback.
>
> Thanks & Regards
> Nishad C Hameed
>
> MACBIS/T-MSIS
> M:+1(301) 335-5346
> L:+1(301) 977-7884 x 681
>
>
> This electronic mail (including any attachments) may contain information
> that is privileged, confidential, and/or otherwise protected from
> disclosure to anyone other than its intended recipient(s). Any
> dissemination or use of this electronic email or its contents (including
> any attachments) by persons other than the intended recipient(s) is
> strictly prohibited. If you have received this message in error, please
> notify the sender by reply email and delete the original message (including
> any attachments) in its entirety.
>

Re: Nifi Cross Account Download With A Profile Flag

[James Wing]

Keren,

I'm certain cross-account access can work in 0.6.0, I've done it.

The timeout error calling sts:AssumeRole is not the same error you started
with, right?  Earlier, the error was 403 "Access Denied", which would have
been farther in the auth process.  Any idea what changed?  Are you using
the HTTP proxy settings on FetchS3Object?


Thanks,

James

On Thu, Sep 1, 2016 at 10:30 AM, Tseytlin, Keren <
keren.tseyt...@capitalone.com> wrote:

> Hey James,
>
> No problem. I’ve tried running a bunch of different ways to do it manually
> to try and avoid the CLI profile flag. But it seems like that is the only
> it works locally (without Nifi). My cross account role doesn’t require an
> External ID, and that feature doesn’t exist in 0.6.0 either.
>
> An small portion of the logs are below. The logs show that it starts to
> attempt to get the S3 object, it times out on assuming the
> role/credentials, and then fails.
>
> at
> com.amazonaws.auth.STSAssumeRoleSessionCredential
> sProvider.startSession(STS
> AssumeRoleSessionCredentialsProvider.java:272)
> [aws-java-sdk-sts-1.10.32.jar:na]
> at
> com.amazonaws.auth.STSAssumeRoleSessionCredential
> sProvider.getCredentials(S
> TSAssumeRoleSessionCredentialsProvider.java:247)
> [aws-java-sdk-sts-1.10.32.jar:na]
> at
> com.amazonaws.auth.STSAssumeRoleSessionCredential
> sProvider.getCredentials(S
> TSAssumeRoleSessionCredentialsProvider.java:34)
> [aws-java-sdk-sts-1.10.32.jar:na]
> at
> com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.
> invoke(A
> WSSecurityTokenServiceClient.java:1098) [aws-java-sdk-sts-1.10.32.jar:na]
> at
> com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.
> assumeRo
> le(AWSSecurityTokenServiceClient.java:1000)
> [aws-java-sdk-sts-1.10.32.jar:na]
> at
> com.amazonaws.auth.STSAssumeRoleSessionCredential
> sProvider.startSession(STS
> AssumeRoleSessionCredentialsProvider.java:272)
> [aws-java-sdk-sts-1.10.32.jar:na]
> at
> com.amazonaws.auth.STSAssumeRoleSessionCredential
> sProvider.getCredentials(S
> TSAssumeRoleSessionCredentialsProvider.java:247)
> [aws-java-sdk-sts-1.10.32.jar:na]
> at
> com.amazonaws.auth.STSAssumeRoleSessionCredential
> sProvider.getCredentials(S
> TSAssumeRoleSessionCredentialsProvider.java:34)
> [aws-java-sdk-sts-1.10.32.jar:na]
> at
> com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3589)
> [aws-java-sdk-s3-1.10.32.jar:na]
> at
> com.amazonaws.services.s3.AmazonS3Client.getObject(
> AmazonS3Client.java:1116
> ) [aws-java-sdk-s3-1.10.32.jar:na]
> at
> org.apache.nifi.processors.aws.s3.FetchS3Object.
> onTrigger(FetchS3Object.jav
> a:105) [nifi-aws-processors-0.6.0.1.2.0.0-91.jar:0.6.0.1.2.0.0-91]
> at
> org.apache.nifi.processor.AbstractProcessor.onTrigger(
> AbstractProcessor.jav
> a:27) [nifi-api-0.6.0.1.2.0.0-91.jar:0.6.0.1.2.0.0-91]
>
>
> If you (or anyone scanning the thread) can think of a way to do this
> without upgrading that’d be awesome. Otherwise, I’ll start motivating the
> masses.
>
> Best,
> Keren
>
> On 9/1/16, 12:17 PM, "James Wing"  wrote:
>
> >Keren,
> >
> >I'm sorry if my advice is a bit confusing, there have been some changes to
> >AWS credentials over the last few versions.  NiFi 0.6.0 does not have the
> >option to use a CLI profile in AWSCredentialsProviderControllerService, I
> >think that was introduced in 0.7.0.
> >
> >Would it be possible to share some of the log entries?  Was there a stack
> >trace associated with the timeout?
> >
> >Also, does your cross-account Role require an External ID?  I do not
> >believe that is supported in 0.6.0, but can be required to assume some
> >roles.
> >
> >
> >Thanks,
> >
> >James
> >
> >On Thu, Sep 1, 2016 at 7:57 AM, Tseytlin, Keren <
> >keren.tseyt...@capitalone.com> wrote:
> >
> >> Thanks for your responses!
> >>
> >> @James - we are on version 0.6.0. Using Hortonworks Data Flow 1.2.0.0.
> >>
> >> I¹ve set up debugging, and it shows me that it¹s trying to connect, but
> >>it
> >> times out on connecting. It would be awesome if it would also return the
> >> account ID of the credentials it is trying to use.
> >>
> >> Is there any way to see the exact keys/tokens that Nifi is trying to use
> >> to get the S3 object? I¹m not seeing it in the logs.
> >>
> >> I tried to set the Profile in Nifi, but it complains that it¹s not a
> >>valid
> >> property.
> >>
> >> Best,
> >> Keren
> >>
> >> On 8/31/16, 6:24 PM, "Andrew Grande"  wrote:
> >>
> >> >Debug logging can be set in a processor itself in the UI, too.
> >> >
> >> >On Wed, Aug 31, 2016, 5:34 PM James Wing  wrote:
> >> >
> >> >> Keren,
> >> >>
> >> >> Which version of NiFi are you using?
> >> >>
> >> >> One thing I noticed in your configuration of FetchS3Object is you are
> >> >> setting both the Access Key and Secret Key properties with the AWS
> >> >> Credentials Provider.  When you are using the AWS 

Re: Nifi Cross Account Download With A Profile Flag

[Tseytlin, Keren]

Hey James,

No problem. I’ve tried running a bunch of different ways to do it manually
to try and avoid the CLI profile flag. But it seems like that is the only
it works locally (without Nifi). My cross account role doesn’t require an
External ID, and that feature doesn’t exist in 0.6.0 either.

An small portion of the logs are below. The logs show that it starts to
attempt to get the S3 object, it times out on assuming the
role/credentials, and then fails.

at 
com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.startSession(STS
AssumeRoleSessionCredentialsProvider.java:272)
[aws-java-sdk-sts-1.10.32.jar:na]
at 
com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.getCredentials(S
TSAssumeRoleSessionCredentialsProvider.java:247)
[aws-java-sdk-sts-1.10.32.jar:na]
at 
com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.getCredentials(S
TSAssumeRoleSessionCredentialsProvider.java:34)
[aws-java-sdk-sts-1.10.32.jar:na]
at 
com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.invoke(A
WSSecurityTokenServiceClient.java:1098) [aws-java-sdk-sts-1.10.32.jar:na]
at 
com.amazonaws.services.securitytoken.AWSSecurityTokenServiceClient.assumeRo
le(AWSSecurityTokenServiceClient.java:1000)
[aws-java-sdk-sts-1.10.32.jar:na]
at 
com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.startSession(STS
AssumeRoleSessionCredentialsProvider.java:272)
[aws-java-sdk-sts-1.10.32.jar:na]
at 
com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.getCredentials(S
TSAssumeRoleSessionCredentialsProvider.java:247)
[aws-java-sdk-sts-1.10.32.jar:na]
at 
com.amazonaws.auth.STSAssumeRoleSessionCredentialsProvider.getCredentials(S
TSAssumeRoleSessionCredentialsProvider.java:34)
[aws-java-sdk-sts-1.10.32.jar:na]
at 
com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3589)
[aws-java-sdk-s3-1.10.32.jar:na]
at 
com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1116
) [aws-java-sdk-s3-1.10.32.jar:na]
at 
org.apache.nifi.processors.aws.s3.FetchS3Object.onTrigger(FetchS3Object.jav
a:105) [nifi-aws-processors-0.6.0.1.2.0.0-91.jar:0.6.0.1.2.0.0-91]
at 
org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.jav
a:27) [nifi-api-0.6.0.1.2.0.0-91.jar:0.6.0.1.2.0.0-91]


If you (or anyone scanning the thread) can think of a way to do this
without upgrading that’d be awesome. Otherwise, I’ll start motivating the
masses. 

Best,
Keren

On 9/1/16, 12:17 PM, "James Wing"  wrote:

>Keren,
>
>I'm sorry if my advice is a bit confusing, there have been some changes to
>AWS credentials over the last few versions.  NiFi 0.6.0 does not have the
>option to use a CLI profile in AWSCredentialsProviderControllerService, I
>think that was introduced in 0.7.0.
>
>Would it be possible to share some of the log entries?  Was there a stack
>trace associated with the timeout?
>
>Also, does your cross-account Role require an External ID?  I do not
>believe that is supported in 0.6.0, but can be required to assume some
>roles.
>
>
>Thanks,
>
>James
>
>On Thu, Sep 1, 2016 at 7:57 AM, Tseytlin, Keren <
>keren.tseyt...@capitalone.com> wrote:
>
>> Thanks for your responses!
>>
>> @James - we are on version 0.6.0. Using Hortonworks Data Flow 1.2.0.0.
>>
>> I¹ve set up debugging, and it shows me that it¹s trying to connect, but
>>it
>> times out on connecting. It would be awesome if it would also return the
>> account ID of the credentials it is trying to use.
>>
>> Is there any way to see the exact keys/tokens that Nifi is trying to use
>> to get the S3 object? I¹m not seeing it in the logs.
>>
>> I tried to set the Profile in Nifi, but it complains that it¹s not a
>>valid
>> property.
>>
>> Best,
>> Keren
>>
>> On 8/31/16, 6:24 PM, "Andrew Grande"  wrote:
>>
>> >Debug logging can be set in a processor itself in the UI, too.
>> >
>> >On Wed, Aug 31, 2016, 5:34 PM James Wing  wrote:
>> >
>> >> Keren,
>> >>
>> >> Which version of NiFi are you using?
>> >>
>> >> One thing I noticed in your configuration of FetchS3Object is you are
>> >> setting both the Access Key and Secret Key properties with the AWS
>> >> Credentials Provider.  When you are using the AWS Credentials
>>Provider
>> >> Service, you should not specify keys.
>> >>
>> >> A more certainly helpful thing to do is enable debug logging for the
>>AWS
>> >> processor package by adding a line like the following to
>> >>conf/logback.xml:
>> >>
>> >> 
>> >>
>> >> With the debug logging enabled, there are messages indicating which
>> >> credential type is being attempted.  Your settings for the AWS
>> >>Credentials
>> >> Provider look appropriate.  The controller service is indeed
>>designed to
>> >> refresh the STS token automagically using the AWS SDK classes for
>> >>temporary
>> >> credentials.
>> >>
>> >> Last, you might experiment with configuring
>> >> 

Apache NiFi - 3 tier Architecture

[Nishad Hameed]

Hi All,

We are planning to use the "Apache NiFi" for our data flow and scheduling 
service. But as we have 3 tier architecture, we need to see if we can split 
NiFi.

Like

1.   Web/Presentation layer

2.   Application layer

3.   Data Layer

Is there any project already implemented this way?

Waiting for your feedback.

Thanks & Regards
Nishad C Hameed

MACBIS/T-MSIS
M:+1(301) 335-5346
L:+1(301) 977-7884 x 681


This electronic mail (including any attachments) may contain information that 
is privileged, confidential, and/or otherwise protected from disclosure to 
anyone other than its intended recipient(s). Any dissemination or use of this 
electronic email or its contents (including any attachments) by persons other 
than the intended recipient(s) is strictly prohibited. If you have received 
this message in error, please notify the sender by reply email and delete the 
original message (including any attachments) in its entirety.

Minifi question

[Gunjan Dave]

Hello everyone, minifi as i understand has specific use case of small
footprint data collection agent.
Hiwever, today i was trying it out and saw an interesting possibility,
would need NiFi teams opinion on it.

Can minifi be also modified to work as test harness agent? Like can
currrently processes a smaller part of the complete data flow. So with
certain enhacements as separate project, can it be also made to work as
test harness? I have further thoughts on it to share if you guys concur
with current thought

Re: Nifi Cross Account Download With A Profile Flag

[James Wing]

Keren,

I'm sorry if my advice is a bit confusing, there have been some changes to
AWS credentials over the last few versions.  NiFi 0.6.0 does not have the
option to use a CLI profile in AWSCredentialsProviderControllerService, I
think that was introduced in 0.7.0.

Would it be possible to share some of the log entries?  Was there a stack
trace associated with the timeout?

Also, does your cross-account Role require an External ID?  I do not
believe that is supported in 0.6.0, but can be required to assume some
roles.


Thanks,

James

On Thu, Sep 1, 2016 at 7:57 AM, Tseytlin, Keren <
keren.tseyt...@capitalone.com> wrote:

> Thanks for your responses!
>
> @James - we are on version 0.6.0. Using Hortonworks Data Flow 1.2.0.0.
>
> I¹ve set up debugging, and it shows me that it¹s trying to connect, but it
> times out on connecting. It would be awesome if it would also return the
> account ID of the credentials it is trying to use.
>
> Is there any way to see the exact keys/tokens that Nifi is trying to use
> to get the S3 object? I¹m not seeing it in the logs.
>
> I tried to set the Profile in Nifi, but it complains that it¹s not a valid
> property.
>
> Best,
> Keren
>
> On 8/31/16, 6:24 PM, "Andrew Grande"  wrote:
>
> >Debug logging can be set in a processor itself in the UI, too.
> >
> >On Wed, Aug 31, 2016, 5:34 PM James Wing  wrote:
> >
> >> Keren,
> >>
> >> Which version of NiFi are you using?
> >>
> >> One thing I noticed in your configuration of FetchS3Object is you are
> >> setting both the Access Key and Secret Key properties with the AWS
> >> Credentials Provider.  When you are using the AWS Credentials Provider
> >> Service, you should not specify keys.
> >>
> >> A more certainly helpful thing to do is enable debug logging for the AWS
> >> processor package by adding a line like the following to
> >>conf/logback.xml:
> >>
> >> 
> >>
> >> With the debug logging enabled, there are messages indicating which
> >> credential type is being attempted.  Your settings for the AWS
> >>Credentials
> >> Provider look appropriate.  The controller service is indeed designed to
> >> refresh the STS token automagically using the AWS SDK classes for
> >>temporary
> >> credentials.
> >>
> >> Last, you might experiment with configuring
> >> AWSCredentialsProviderControllerService to use your named CLI profile
> >> "crossaccountrole", which should also work.
> >>
> >> Thanks,
> >>
> >> James
> >>
> >> On Wed, Aug 31, 2016 at 1:44 PM, Tseytlin, Keren <
> >> keren.tseyt...@capitalone.com> wrote:
> >>
> >> > Hi All!
> >> >
> >> > Looking for some help on enabling Cross Account communication within
> >> Nifi!
> >> >
> >> > My goal: There are files stored from CloudTrail in an S3 bucket in
> >>VPC B.
> >> > My Nifi machines are in VPC A. I want Nifi to be able to get those
> >>files
> >> > from VPC B. VPC A and VPC B need to be communicating in the
> >>FetchS3Object
> >> > component.
> >> >
> >> > See this link for some additional info: http://docs.aws.amazon.com/
> >> >
> >>awscloudtrail/latest/userguide/cloudtrail-sharing-logs-assume-role.html
> >> >
> >> > I have communication working manually on the Nifi machines in VPC A
> >>when
> >> I
> >> > use the AWS CLI. The process is as follows:
> >> >
> >> > 1. Run sts -assume-role on my Nifi machine (VPC A) to assume a
> >>role
> >> > I've created in VPC B that is configured to have access to the S3
> >>bucket
> >> in
> >> > VPC B.
> >> >
> >> > 2. This will generate temporary keys that need to be refreshed
> >>every
> >> > hour. There is no way to have assume role create permanent keys.
> >>Export
> >> the
> >> > keys as environment variables.
> >> >
> >> > 3. Set up ~/.aws/config to have a profile "crossaccountrole" that
> >> > connects to the arn of the role created in VPC B.
> >> >
> >> > 4. Run the following command à "aws s3 cp s3://
> >> >> > name locally> --profile crossaccountrole"
> >> >
> >> > Most importantly, if I ever try to run this without the --profile
> >>flag,
> >> > then it will not allow me to download the file.  It seems like
> >>perhaps to
> >> > get it to work with Nifi I need a place to pass in the profile that
> >>needs
> >> > to be used in order for the communication to work.
> >> >
> >> > I've been trying to implement this in Nifi. Within the FetchS3Object,
> >>I
> >> > have created an AWSCredentialsProviderService which has the following
> >> > properties:
> >> >
> >> > ·  Access Key: VPC A access key
> >> >
> >> > ·  Secret Key: VPC A secret key
> >> >
> >> > ·  Assume Role ARN: VPC B role
> >> >
> >> > ·  Assume Role Session Name: crossaccountrole
> >> >
> >> > ·  Session Time: 3600
> >> > The general properties in the FetchS3Object are as follows:
> >> >
> >> > ·  Bucket: VPC B bucket name
> >> >
> >> > ·  Object: Filename of VPC B bucket object
> >> >
> >> > ·  Access Key: VPC A access key
> >> >
> >> > ·  Secret Key: VPC A secret key
> >> >
> >> > 

Re: Nifi Cross Account Download With A Profile Flag

[Tseytlin, Keren]

Thanks for your responses!

@James - we are on version 0.6.0. Using Hortonworks Data Flow 1.2.0.0.

I¹ve set up debugging, and it shows me that it¹s trying to connect, but it
times out on connecting. It would be awesome if it would also return the
account ID of the credentials it is trying to use.

Is there any way to see the exact keys/tokens that Nifi is trying to use
to get the S3 object? I¹m not seeing it in the logs.

I tried to set the Profile in Nifi, but it complains that it¹s not a valid
property.

Best,
Keren

On 8/31/16, 6:24 PM, "Andrew Grande"  wrote:

>Debug logging can be set in a processor itself in the UI, too.
>
>On Wed, Aug 31, 2016, 5:34 PM James Wing  wrote:
>
>> Keren,
>>
>> Which version of NiFi are you using?
>>
>> One thing I noticed in your configuration of FetchS3Object is you are
>> setting both the Access Key and Secret Key properties with the AWS
>> Credentials Provider.  When you are using the AWS Credentials Provider
>> Service, you should not specify keys.
>>
>> A more certainly helpful thing to do is enable debug logging for the AWS
>> processor package by adding a line like the following to
>>conf/logback.xml:
>>
>> 
>>
>> With the debug logging enabled, there are messages indicating which
>> credential type is being attempted.  Your settings for the AWS
>>Credentials
>> Provider look appropriate.  The controller service is indeed designed to
>> refresh the STS token automagically using the AWS SDK classes for
>>temporary
>> credentials.
>>
>> Last, you might experiment with configuring
>> AWSCredentialsProviderControllerService to use your named CLI profile
>> "crossaccountrole", which should also work.
>>
>> Thanks,
>>
>> James
>>
>> On Wed, Aug 31, 2016 at 1:44 PM, Tseytlin, Keren <
>> keren.tseyt...@capitalone.com> wrote:
>>
>> > Hi All!
>> >
>> > Looking for some help on enabling Cross Account communication within
>> Nifi!
>> >
>> > My goal: There are files stored from CloudTrail in an S3 bucket in
>>VPC B.
>> > My Nifi machines are in VPC A. I want Nifi to be able to get those
>>files
>> > from VPC B. VPC A and VPC B need to be communicating in the
>>FetchS3Object
>> > component.
>> >
>> > See this link for some additional info: http://docs.aws.amazon.com/
>> > 
>>awscloudtrail/latest/userguide/cloudtrail-sharing-logs-assume-role.html
>> >
>> > I have communication working manually on the Nifi machines in VPC A
>>when
>> I
>> > use the AWS CLI. The process is as follows:
>> >
>> > 1. Run sts -assume-role on my Nifi machine (VPC A) to assume a
>>role
>> > I've created in VPC B that is configured to have access to the S3
>>bucket
>> in
>> > VPC B.
>> >
>> > 2. This will generate temporary keys that need to be refreshed
>>every
>> > hour. There is no way to have assume role create permanent keys.
>>Export
>> the
>> > keys as environment variables.
>> >
>> > 3. Set up ~/.aws/config to have a profile "crossaccountrole" that
>> > connects to the arn of the role created in VPC B.
>> >
>> > 4. Run the following command à "aws s3 cp s3://
>>> > name locally> --profile crossaccountrole"
>> >
>> > Most importantly, if I ever try to run this without the --profile
>>flag,
>> > then it will not allow me to download the file.  It seems like
>>perhaps to
>> > get it to work with Nifi I need a place to pass in the profile that
>>needs
>> > to be used in order for the communication to work.
>> >
>> > I've been trying to implement this in Nifi. Within the FetchS3Object,
>>I
>> > have created an AWSCredentialsProviderService which has the following
>> > properties:
>> >
>> > ·  Access Key: VPC A access key
>> >
>> > ·  Secret Key: VPC A secret key
>> >
>> > ·  Assume Role ARN: VPC B role
>> >
>> > ·  Assume Role Session Name: crossaccountrole
>> >
>> > ·  Session Time: 3600
>> > The general properties in the FetchS3Object are as follows:
>> >
>> > ·  Bucket: VPC B bucket name
>> >
>> > ·  Object: Filename of VPC B bucket object
>> >
>> > ·  Access Key: VPC A access key
>> >
>> > ·  Secret Key: VPC A secret key
>> >
>> > ·  AWS Credentials Provider Service: 
>> >
>> > However, when this tries to run I get Access Denied. I've been going
>> > through the source code for Nifi and I'm not sure if short-lived
>>tokens
>> get
>> > passed through. Can anyone please provide me some guidance or
>>suggestions
>> > on how to get this to work? J
>> >
>> > Best,
>> > Keren
>> > 
>> >
>> > The information contained in this e-mail is confidential and/or
>> > proprietary to Capital One and/or its affiliates and may only be used
>> > solely in performance of work or services for Capital One. The
>> information
>> > transmitted herewith is intended only for use by the individual or
>>entity
>> > to which it is addressed. If the reader of this message is not the
>> intended
>> > recipient, you are hereby notified that any review, retransmission,
>> > 

Back Pressure Indicators?

[Peter Wicks (pwicks)]

One of the things I've been noticing that doesn't seem to have any status 
indicator for is back pressure.  When back pressure kicks in I usually notice 
because I start thinking that NiFi is being buggy or is broken, "Why aren't 
these Flow Files moving through the queue?".  Then I'll notice the GB's of data 
in the downstream queue and it will click.

Am I missing it or is there no status indicator showing that Back Pressure is 
in action?