Clarifications on getting flowfiles using FlowFileFilters

[Eric Chaves]

Hi folks,

I'd like to write InvokeScriptedProcessor that inspect all files in the
incoming queue without actually processing/transfering them until a given
business condition is met.

So I looked for some examples of how could I get all flowfiles in a queue
using session.get(FlowFileFilter) and I would like to confirm if I'm
getting it right.

1) If I have more than one thread executing my processor calling
session.get(FlowFileFilter) can the same flowfile be present to both
threads?

2) If I retrieve some flowfiles using session.get(FlowFileFilter) can I
add/modify their attributes but keep them in the current queue (ie set
attributes but don't transfer them to any relationship)?

3) Am I correct that session.get(FlowFileFilter) don't requires neither
session.commit nor session.rollback? Those were required only if I modify
the flowfile.

4) When obtain a list of flowfiles using session.get(FlowFileFilter) does
it marks/lock/hides the selected files in anyway?

5) I'm playing with this concept with some groovy script using a
InvokeScriptedProcessor and very often, after the second or third
processor's on trigger execution I'm receiving an error *java.io.IOException:
All Partitions have been blacklisted due to failures when attempting to
update. If the Write-Ahead Log is able to perform a checkpoint, this issue
may resolve itself. Otherwise, manual intervention will be required. *What
could be the source of this error? Could it be related to the use of
session.get(FlowFileFilter)? I'm having a hard time to detect what I'm
doing that raises this error.

Thanks in advance!

RE: Merge? Notify? Wait?

[William Gosse]

That is an awesome idea!  I’m definitely going to try that. Thank you.

From: Mark Payne 
Sent: Wednesday, March 27, 2019 2:42 PM
To: users@nifi.apache.org
Subject: Re: Merge? Notify? Wait?

[CAUTION: This email originated from outside of Kodak Alaris. Do not click 
links or open attachments unless you recognize the sender and know the content 
is safe.]

William,

You should be able to perform this using the Wait/Notify pattern. But I think a 
simpler (and likely better performing) alternative would be
to avoid splitting out into two directions at all and instead use a linear flow:

ListFile -> AttributesToJson -> ReplaceText -> InvokeHttp -> FetchFile -> 
UpdateAttribute (to rename file) -> PutFile

Because the first direction does not appear to need the contents of the file, 
since it is using attributes to create the JSON payload,
there's no need to fetch the contents of the file until after you've already 
made the web request and retrieved the client id. Once you
have that, you can fetch the contents and then use PutFile.

Thanks
-Mark



On Mar 27, 2019, at 2:25 PM, William Gosse 
mailto:william.go...@aifoundry.com>> wrote:

I’ve been trying to solve this problem in Nifi for the past couple of days and 
haven’t been able to find the secret sauce yet.

I goes something like this:
List/Fetch a file
Extract an client name from part of the filename using an UpdateAttribute

At this I send the flow in two different directions:
Direction 1 does this:
Take the original flow and create a new JSON content out of it 
using an AttributesToJson and ReplaceText
Post my new JSON content to a web service using an InvokeHttp 
to create a new client with the client name
Extract a client new client id from the web service response.

Direction 2 does this
Wait for the new client id return from the webservice  response 
(this is the part I’m not getting)
Use client id to rename original file
Put file to specified folder.

I’ll admit I could be doing this all wrong but what I’m having trouble with is 
bringing together my original flowfile with the result of my web service call.

I thought I could use a mergecontent somehow or a notify/wait but neither 
approach seems to work.

Any help will be greatly appreciated.

Re: Merge? Notify? Wait?

[Mark Payne]

William,

You should be able to perform this using the Wait/Notify pattern. But I think a 
simpler (and likely better performing) alternative would be
to avoid splitting out into two directions at all and instead use a linear flow:

ListFile -> AttributesToJson -> ReplaceText -> InvokeHttp -> FetchFile -> 
UpdateAttribute (to rename file) -> PutFile

Because the first direction does not appear to need the contents of the file, 
since it is using attributes to create the JSON payload,
there's no need to fetch the contents of the file until after you've already 
made the web request and retrieved the client id. Once you
have that, you can fetch the contents and then use PutFile.

Thanks
-Mark


On Mar 27, 2019, at 2:25 PM, William Gosse 
mailto:william.go...@aifoundry.com>> wrote:

I’ve been trying to solve this problem in Nifi for the past couple of days and 
haven’t been able to find the secret sauce yet.

I goes something like this:
List/Fetch a file
Extract an client name from part of the filename using an UpdateAttribute

At this I send the flow in two different directions:
Direction 1 does this:
Take the original flow and create a new JSON content out of it 
using an AttributesToJson and ReplaceText
Post my new JSON content to a web service using an InvokeHttp 
to create a new client with the client name
Extract a client new client id from the web service response.

Direction 2 does this
Wait for the new client id return from the webservice  response 
(this is the part I’m not getting)
Use client id to rename original file
Put file to specified folder.

I’ll admit I could be doing this all wrong but what I’m having trouble with is 
bringing together my original flowfile with the result of my web service call.

I thought I could use a mergecontent somehow or a notify/wait but neither 
approach seems to work.

Any help will be greatly appreciated.

Re: Empty "nifi users" page.

[Matt Gilman]

Is it possible that you have a user and a group with the same
name/identity? Can you compare what is in your users.xml file and what is
being loaded from your directory server? To see what's being loaded from
your directory server, if your running 1.9.x you should be able to enable
debug level logging on

org.apache.nifi.ldap.tenants.LdapUserGroupProvider

Thanks

Matt

On Wed, Mar 27, 2019 at 5:46 AM DEHAY Aurelien 
wrote:

> Thanks for you answer.
>
>
>
> Like Bryan and you made me investigate on the browser side, I’ve check
> cache and dev tools.
>
>
>
> Indeed the data are correctly sent to the browser. The users & groups api
> are correctly called and results are good.
>
>
>
> So I suppose it’s something on browser side (tested with FF and chrome
> with same issue).
>
>
>
> I got these errors in the console without guarantee that it’s linked to
> the issue:
>
>
>
> Content Security Policy: « x-frame-options » ignoré en raison de la
> directive « frame-ancestors ».
>
> CURRENT_SITE:  par01prdedge1.soft.fau:18443/nifi/users
> KeeperFill-var.js:30:935
>
> uncaught exception: Each data element must implement a unique 'id' property
>
>
>
> Thanks.
>
>
>
> *From:* Matt Gilman 
> *Sent:* mardi 26 mars 2019 17:49
> *To:* users@nifi.apache.org
> *Subject:* Re: Empty "nifi users" page.
>
>
>
> Hi.
>
>
>
> Could you open up your browsers Developer Tools and check the Console for
> any errors and look at the Network requests and ensure that they are
> returning correctly. The behavior your describing indicate one of two
> things. Either the request is not returning from the server or there is a
> front end code bug.
>
>
>
> We have addressed a couple of issues recently that are scheduled to land
> in 1.10.0 where users are removed (from LDAP externally) and they are still
> referenced in NiFi policies or groups. However, I don't think the behavior
> is exactly how you describe.
>
>
>
> Also, are you able to build current master and try that out? If your
> running into the same problem, it's possible its already addressed.
>
>
>
> Thanks
>
>
>
> Matt
>
>
>
> On Tue, Mar 26, 2019 at 10:29 AM Bryan Bende  wrote:
>
> What browser and browser version are you using?
>
> Have you tried clearing your browser cache just to make sure the page
> is loading properly?
>
> On Tue, Mar 26, 2019 at 10:21 AM DEHAY Aurelien
>  wrote:
> >
> > Hello.
> >
> > I expect to see at least the local users with associated rights (e.g.
> content of users.xml and authorizations.xml), I've got 3 local users and a
> bunch of associated rights with ldap users/groups.
> >
> > I'll try the search filter, but the list is ok when modifying the
> policies of objects, I can see groups 
> >
> > Thanks.
> >
> > -Original Message-
> > From: Kevin Doran 
> > Sent: samedi 23 mars 2019 18:52
> > To: users@nifi.apache.org
> > Subject: Re: Empty "nifi users" page.
> >
> > How many users and groups do you expect to sync? Are you able to test
> your search base and filter are correct using a command line tool such as
> ldapsearch? I'm not sure if a search filter is required (although it would
> probably fail with an error if it was), but maybe try setting a "match all"
> filter such as (uid=*).
> >
> > Hope this helps,
> > Kevin
> >
> > On Fri, Mar 22, 2019 at 11:08 AM DEHAY Aurelien <
> aurelien.de...@faurecia.com> wrote:
> > >
> > > Hello.
> > >
> > >
> > >
> > > My “nifi users” page is empty, with only the spinner spinning at the
> bottom of the page.
> > >
> > >
> > >
> > > I’ve tried a lot of conf, and for now everything work fine, except
> that. There is nothing in the log, even when I tried to put “debug” for
> authorization parts in logback.
> > >
> > >
> > >
> > > I had the same behavior with 1.7, 1.8, 1.9 and 1.9.1.
> > >
> > >
> > >
> > > My configuration authorizers.xml and the part of user mapping in the
> > > nifi.properties is
> > > https://gist.github.com/zorel/f76c3924f2b266e8a878397271a54f24
> > >
> > >
> > >
> > > I’ve read Pierre Villard blog post, nifi doc and a lot of post in
> hortonworks site, but I’ve must I missed something. Any pointer is
> appreciated.
> > >
> > >
> > >
> > > Thanks.
> > >
> > >
> > > This electronic transmission (and any attachments thereto) is intended
> solely for the use of the addressee(s). It may contain confidential or
> legally privileged information. If you are not the intended recipient of
> this message, you must delete it immediately and notify the sender. Any
> unauthorized use or disclosure of this message is strictly prohibited.
> Faurecia does not guarantee the integrity of this transmission and shall
> therefore never be liable if the message is altered or falsified nor for
> any virus, interception or damage to your system.
> >
> > This electronic transmission (and any attachments thereto) is intended
> solely for the use of the addressee(s). It may contain confidential or
> legally privileged information. If you are not the intended recipient of
> this message, you must delete it 

Merge? Notify? Wait?

[William Gosse]

I’ve been trying to solve this problem in Nifi for the past couple of days and 
haven’t been able to find the secret sauce yet.

I goes something like this:
List/Fetch a file
Extract an client name from part of the filename using an UpdateAttribute

At this I send the flow in two different directions:
Direction 1 does this:
Take the original flow and create a new JSON content out of it 
using an AttributesToJson and ReplaceText
Post my new JSON content to a web service using an InvokeHttp 
to create a new client with the client name
Extract a client new client id from the web service response.

Direction 2 does this
Wait for the new client id return from the webservice  response 
(this is the part I’m not getting)
Use client id to rename original file
Put file to specified folder.

I’ll admit I could be doing this all wrong but what I’m having trouble with is 
bringing together my original flowfile with the result of my web service call.

I thought I could use a mergecontent somehow or a notify/wait but neither 
approach seems to work.

Any help will be greatly appreciated.

Re: Changing All Stopped Processors to Disabled

[Ryan H]

Sounds like the ability to disable at the Process Group exists after 1.7, I
retract my question!

-Ryan H

On Wed, Mar 27, 2019 at 2:18 PM Ryan H 
wrote:

> @Mark, we are on version 1.5 and there is a definite slow down, at least
> for us with UI responsiveness (initial load of canvas upon logging in is
> close to a minute with about 2,000 processors on the canvas, with about
> 1,500 in the stopped state). We are planning on upgrading to 1.8.0 soon...
> Should there be a significant improvement in 1.8.0?
>
> @Andy, thanks for the suggestion. I agree that NiPyApi would foot the bill
> on this one.
>
> Would this be a decent feature request? The ability to start/stop at the
> process group level currently exists. Does it make sense to also be able to
> disable at the Process Group level (not sure if there is a good case for it
> other than this scenario)?
>
>
> -Ryan H
>
> On Wed, Mar 27, 2019 at 2:07 PM Andy LoPresto 
> wrote:
>
>> Ryan,
>>
>> Reading through your email, my immediate suggestion was NiPyAPI. I think
>> Dan has wrapped some useful query methods there that could make this quite
>> easy. Obviously you are aware of it, but it’s still my best recommendation
>> for now.
>>
>>
>> Andy LoPresto
>> alopre...@apache.org
>> *alopresto.apa...@gmail.com *
>> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>>
>> On Mar 27, 2019, at 10:58 AM, Ryan H 
>> wrote:
>>
>> Hi All,
>>
>> Is there a good way to change all processors on the canvas that are
>> stopped to a disabled state instead? The problem is that we have a large
>> amount of processors on our canvas that are in the stopped state which is
>> killing the UI performance (wouldn't want to go to each of the 2,000
>> stopped processors individually and mark as disabled). We just learned that
>> this isn't an issue (with regard to UI performance) when processors are in
>> the disabled state due to the way status checks are performed. I'm sure
>> this could be scripted with NiPy or something else, but just wanted to
>> throw the question out to the community first before delving into this.
>>
>> Cheers,
>>
>> Ryan H
>>
>>
>>

Re: Changing All Stopped Processors to Disabled

[Ryan H]

@Mark, we are on version 1.5 and there is a definite slow down, at least
for us with UI responsiveness (initial load of canvas upon logging in is
close to a minute with about 2,000 processors on the canvas, with about
1,500 in the stopped state). We are planning on upgrading to 1.8.0 soon...
Should there be a significant improvement in 1.8.0?

@Andy, thanks for the suggestion. I agree that NiPyApi would foot the bill
on this one.

Would this be a decent feature request? The ability to start/stop at the
process group level currently exists. Does it make sense to also be able to
disable at the Process Group level (not sure if there is a good case for it
other than this scenario)?


-Ryan H

On Wed, Mar 27, 2019 at 2:07 PM Andy LoPresto  wrote:

> Ryan,
>
> Reading through your email, my immediate suggestion was NiPyAPI. I think
> Dan has wrapped some useful query methods there that could make this quite
> easy. Obviously you are aware of it, but it’s still my best recommendation
> for now.
>
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Mar 27, 2019, at 10:58 AM, Ryan H 
> wrote:
>
> Hi All,
>
> Is there a good way to change all processors on the canvas that are
> stopped to a disabled state instead? The problem is that we have a large
> amount of processors on our canvas that are in the stopped state which is
> killing the UI performance (wouldn't want to go to each of the 2,000
> stopped processors individually and mark as disabled). We just learned that
> this isn't an issue (with regard to UI performance) when processors are in
> the disabled state due to the way status checks are performed. I'm sure
> this could be scripted with NiPy or something else, but just wanted to
> throw the question out to the community first before delving into this.
>
> Cheers,
>
> Ryan H
>
>
>

Re: Changing All Stopped Processors to Disabled

[Matt Gilman]

Ryan,

If you're running 1.7 or later you can enable or disable in bulk just like
you can start or stop. It's supported right in NiFi UI. You can use actions
out of the Operate palette or the context menu.

https://issues.apache.org/jira/browse/NIFI-5066

Matt

On Wed, Mar 27, 2019 at 2:07 PM Andy LoPresto  wrote:

> Ryan,
>
> Reading through your email, my immediate suggestion was NiPyAPI. I think
> Dan has wrapped some useful query methods there that could make this quite
> easy. Obviously you are aware of it, but it’s still my best recommendation
> for now.
>
>
> Andy LoPresto
> alopre...@apache.org
> *alopresto.apa...@gmail.com *
> PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69
>
> On Mar 27, 2019, at 10:58 AM, Ryan H 
> wrote:
>
> Hi All,
>
> Is there a good way to change all processors on the canvas that are
> stopped to a disabled state instead? The problem is that we have a large
> amount of processors on our canvas that are in the stopped state which is
> killing the UI performance (wouldn't want to go to each of the 2,000
> stopped processors individually and mark as disabled). We just learned that
> this isn't an issue (with regard to UI performance) when processors are in
> the disabled state due to the way status checks are performed. I'm sure
> this could be scripted with NiPy or something else, but just wanted to
> throw the question out to the community first before delving into this.
>
> Cheers,
>
> Ryan H
>
>
>

Re: Changing All Stopped Processors to Disabled

[Andy LoPresto]

Ryan, 

Reading through your email, my immediate suggestion was NiPyAPI. I think Dan 
has wrapped some useful query methods there that could make this quite easy. 
Obviously you are aware of it, but it’s still my best recommendation for now. 


Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Mar 27, 2019, at 10:58 AM, Ryan H  
> wrote:
> 
> Hi All,
> 
> Is there a good way to change all processors on the canvas that are stopped 
> to a disabled state instead? The problem is that we have a large amount of 
> processors on our canvas that are in the stopped state which is killing the 
> UI performance (wouldn't want to go to each of the 2,000 stopped processors 
> individually and mark as disabled). We just learned that this isn't an issue 
> (with regard to UI performance) when processors are in the disabled state due 
> to the way status checks are performed. I'm sure this could be scripted with 
> NiPy or something else, but just wanted to throw the question out to the 
> community first before delving into this. 
> 
> Cheers,
> 
> Ryan H

Re: Changing All Stopped Processors to Disabled

[Mark Payne]

Ryan,

In later versions of NiFi (1.8.0 I believe) the validation is done in the 
background instead of in the web request,
so stopped processors won't slow down the UI. We also do allow the user to 
select 1 or more components and
disable/enable the components in bulk. Older versions did not allow for that.

Thanks
-Mark


> On Mar 27, 2019, at 1:58 PM, Ryan H  wrote:
> 
> Hi All,
> 
> Is there a good way to change all processors on the canvas that are stopped 
> to a disabled state instead? The problem is that we have a large amount of 
> processors on our canvas that are in the stopped state which is killing the 
> UI performance (wouldn't want to go to each of the 2,000 stopped processors 
> individually and mark as disabled). We just learned that this isn't an issue 
> (with regard to UI performance) when processors are in the disabled state due 
> to the way status checks are performed. I'm sure this could be scripted with 
> NiPy or something else, but just wanted to throw the question out to the 
> community first before delving into this. 
> 
> Cheers,
> 
> Ryan H

Changing All Stopped Processors to Disabled

[Ryan H]

Hi All,

Is there a good way to change all processors on the canvas that are stopped
to a disabled state instead? The problem is that we have a large amount of
processors on our canvas that are in the stopped state which is killing the
UI performance (wouldn't want to go to each of the 2,000 stopped processors
individually and mark as disabled). We just learned that this isn't an
issue (with regard to UI performance) when processors are in the disabled
state due to the way status checks are performed. I'm sure this could be
scripted with NiPy or something else, but just wanted to throw the question
out to the community first before delving into this.

Cheers,

Ryan H

NiFi 1.9.1 release contains a bug causing content repos to fill...

[Joe Witt]

All,

We will work to promptly produce an Apache NiFi 1.9.2 release which will
correct a regression introduced in the 1.9.1 release.  The issue, NIFI-6150
has been identified and fixed on master and once further testing confirms
we're good we'll get the RC going.

I've updated the release guide to discourage use of 1.9.1 until the fix is
out.

Thanks
Joe

RE: Empty "nifi users" page.

[DEHAY Aurelien]

Thanks for you answer.

Like Bryan and you made me investigate on the browser side, I’ve check cache 
and dev tools.

Indeed the data are correctly sent to the browser. The users & groups api are 
correctly called and results are good.

So I suppose it’s something on browser side (tested with FF and chrome with 
same issue).

I got these errors in the console without guarantee that it’s linked to the 
issue:

Content Security Policy: « x-frame-options » ignoré en raison de la directive « 
frame-ancestors ».
CURRENT_SITE:  par01prdedge1.soft.fau:18443/nifi/users KeeperFill-var.js:30:935
uncaught exception: Each data element must implement a unique 'id' property

Thanks.

From: Matt Gilman 
Sent: mardi 26 mars 2019 17:49
To: users@nifi.apache.org
Subject: Re: Empty "nifi users" page.

Hi.

Could you open up your browsers Developer Tools and check the Console for any 
errors and look at the Network requests and ensure that they are returning 
correctly. The behavior your describing indicate one of two things. Either the 
request is not returning from the server or there is a front end code bug.

We have addressed a couple of issues recently that are scheduled to land in 
1.10.0 where users are removed (from LDAP externally) and they are still 
referenced in NiFi policies or groups. However, I don't think the behavior is 
exactly how you describe.

Also, are you able to build current master and try that out? If your running 
into the same problem, it's possible its already addressed.

Thanks

Matt

On Tue, Mar 26, 2019 at 10:29 AM Bryan Bende 
mailto:bbe...@gmail.com>> wrote:
What browser and browser version are you using?

Have you tried clearing your browser cache just to make sure the page
is loading properly?

On Tue, Mar 26, 2019 at 10:21 AM DEHAY Aurelien
mailto:aurelien.de...@faurecia.com>> wrote:
>
> Hello.
>
> I expect to see at least the local users with associated rights (e.g. content 
> of users.xml and authorizations.xml), I've got 3 local users and a bunch of 
> associated rights with ldap users/groups.
>
> I'll try the search filter, but the list is ok when modifying the policies of 
> objects, I can see groups 
>
> Thanks.
>
> -Original Message-
> From: Kevin Doran mailto:kdo...@apache.org>>
> Sent: samedi 23 mars 2019 18:52
> To: users@nifi.apache.org
> Subject: Re: Empty "nifi users" page.
>
> How many users and groups do you expect to sync? Are you able to test your 
> search base and filter are correct using a command line tool such as 
> ldapsearch? I'm not sure if a search filter is required (although it would 
> probably fail with an error if it was), but maybe try setting a "match all" 
> filter such as (uid=*).
>
> Hope this helps,
> Kevin
>
> On Fri, Mar 22, 2019 at 11:08 AM DEHAY Aurelien 
> mailto:aurelien.de...@faurecia.com>> wrote:
> >
> > Hello.
> >
> >
> >
> > My “nifi users” page is empty, with only the spinner spinning at the bottom 
> > of the page.
> >
> >
> >
> > I’ve tried a lot of conf, and for now everything work fine, except that. 
> > There is nothing in the log, even when I tried to put “debug” for 
> > authorization parts in logback.
> >
> >
> >
> > I had the same behavior with 1.7, 1.8, 1.9 and 1.9.1.
> >
> >
> >
> > My configuration authorizers.xml and the part of user mapping in the
> > nifi.properties is
> > https://gist.github.com/zorel/f76c3924f2b266e8a878397271a54f24
> >
> >
> >
> > I’ve read Pierre Villard blog post, nifi doc and a lot of post in 
> > hortonworks site, but I’ve must I missed something. Any pointer is 
> > appreciated.
> >
> >
> >
> > Thanks.
> >
> >
> > This electronic transmission (and any attachments thereto) is intended 
> > solely for the use of the addressee(s). It may contain confidential or 
> > legally privileged information. If you are not the intended recipient of 
> > this message, you must delete it immediately and notify the sender. Any 
> > unauthorized use or disclosure of this message is strictly prohibited.  
> > Faurecia does not guarantee the integrity of this transmission and shall 
> > therefore never be liable if the message is altered or falsified nor for 
> > any virus, interception or damage to your system.
>
> This electronic transmission (and any attachments thereto) is intended solely 
> for the use of the addressee(s). It may contain confidential or legally 
> privileged information. If you are not the intended recipient of this 
> message, you must delete it immediately and notify the sender. Any 
> unauthorized use or disclosure of this message is strictly prohibited.  
> Faurecia does not guarantee the integrity of this transmission and shall 
> therefore never be liable if the message is altered or falsified nor for any 
> virus, interception or damage to your system.

This electronic transmission (and any attachments thereto) is intended solely 
for the use of the addressee(s). It may contain confidential or legally 
privileged information. If you are not the intended