Re: Non Guaranteed Processing

Thanks , Andy. On Thu, Oct 5, 2017 at 7:55 AM, Andy LoPresto wrote: > Vikram, > > The content [1], flowfile [2], and provenance [3] repositories all have > volatile implementations which do not persist to the file system. You > reduce the overhead by removing I/O to disk,

Non Guaranteed Processing

Hi, Is it possible to have a dataflow with Non Guaranteed processing semantics in Nifi/MiniFi? I am aware that Nifi provides Guaranteed processing by storing the data in its Content Repository. I am wondering if there was any settings which will turn off the data persistence and keep data in

Re: Non Guaranteed Processing

Vikram, The content [1], flowfile [2], and provenance [3] repositories all have volatile implementations which do not persist to the file system. You reduce the overhead by removing I/O to disk, but in the event of power loss, you will lose the data. [1]

Re: A bag of groovy questions regarding the ExecuteScript processor

Hi Andy, That's very helpful, thanks! Inline my comments, waiting for Matt to come home :) On 3 Oct 2017, at 22:44, Andy LoPresto wrote: Giovanni, A lot of great questions here. I’ll try to go through them but I hope Matt weighs in as well (he is on vacation for the next few days

nifi 1.4 - variables

Hello,   in 1.4 when you right-click the processor group then there is an entry "variables". I have defined some, but I wonder how I can use them.   I have not found documentation on this.  Does somebody have details or a link to documentation or an example?   Rgds,   Uwe

Nifi 1.4: problem with QueryRecord Precessor

Hello,   I have created a flow: GetFile >> QueryRecord >> Putfile. GetFile reads an avro file. QueryRecord has one property/sql and the result is routed to PutFile.   When I run the processor, I get following error:   failed to process session due to java.lang.IllegalStateException

Re: nifi 1.4 - variables

Hello, You can use them in any fields that support expression language, just like variables in a properties file (nifi.variable.registry.properties= in nifi.properties) or system properties in bootstrap.conf. In the properties file approach you had to restart NiFi for any changes in the

Aw: nifi 1.4 - variables

I answer myself: you can use the variables with the expression language. so if you have a variable defined named "test123" then in a property of a processor that supports the expression language you can use it as ${test123}. And the value of the processor property will be replaced with the

Re: Nifi 1.4: problem with QueryRecord Precessor

Dan, Uwe, This is something that I will be looking into shortly. It is a known issue that exists in both 1.3 and 1.4. This occurs whenever there is some other sort of failure, when the processor attempts to route the FlowFile to 'failure'. This occurs most often when there is a problem

Re: Re: Nifi 1.4: problem with QueryRecord Precessor

All, The known issue Mark is referring to is NIFI-4349 [1], however it is not causing the problem; rather it is hiding the problem. If an error occurs (due to misonfiguration, schema errors, etc.), certain cleanup activities aren't being performed so the processor ends up with the error about

Re: Nifi 1.4: problem with QueryRecord Precessor

We're seeing the same thing. Works fine in 1.3 Dano On Wed, Oct 4, 2017, 8:13 AM Uwe Geercken wrote: > Hello, > > I have created a flow: GetFile >> QueryRecord >> Putfile. GetFile reads an > avro file. QueryRecord has one property/sql and the result is routed to > PutFile.

Aw: Re: Re: nifi 1.4 - variables

  cool. I will check.   I really like this - to get rid of the hardcoding of paths e.g.   Nifi gets better and better.   Rgds,   Uwe Gesendet: Mittwoch, 04. Oktober 2017 um 17:39 Uhr Von: "Bryan Bende" An: users@nifi.apache.org Betreff: Re: Re: nifi 1.4 - variables

Aw: Re: Nifi 1.4: problem with QueryRecord Precessor

thanks Dano.   Any other comments? Is this a bug in 1.4 then?   Rgds,   Uwe   Gesendet: Mittwoch, 04. Oktober 2017 um 16:33 Uhr Von: "dan young" An: nifi Betreff: Re: Nifi 1.4: problem with QueryRecord Precessor We're seeing the same thing.

Aw: Re: Nifi 1.4: problem with QueryRecord Precessor

  Mark,   I stopped 1.4 and started 1.3 and created the same flow. And in 1.3 it works without a problem.   And I looked into the 1.4 log (I should have looked before, but only looked at the processor's tooltip). You are right, it's a problem with the schema. Here is the real error:  

Re: Nifi 1.4: problem with QueryRecord Precessor

Let me check a few things.. On Wed, Oct 4, 2017 at 8:55 AM Mark Payne wrote: > Dan, Uwe, > > This is something that I will be looking into shortly. It is a known issue > that exists in both 1.3 > and 1.4. This occurs whenever there is some other sort of failure, when >

Re: A bag of groovy questions regarding the ExecuteScript processor

Giovanni, I second all of Andy's answers, they are spot-on. For the each() construct, they are "safe" in the sense that you will be working with one flow file at a time, but remember that there is only one "session". If you throw an Exception from inside the each(), then it will be caught by

Re: Re: Nifi 1.4: problem with QueryRecord Precessor

Arrghh..sorry folks for the noiseit was a schema issue with a conversion/cast exception. ;( Regards, Dano On Wed, Oct 4, 2017 at 9:33 AM Matt Burgess wrote: > All, > > The known issue Mark is referring to is NIFI-4349 [1], however it is > not causing the problem;

Re: Re: Nifi 1.4: problem with QueryRecord Precessor

It isn't a problem at all Dano. We need to hear about these user experience issues. Configuration mistakes will be made. But we need to make sure we provide pointers to get you to the right configuration. So we still have work to do! Thanks and glad you're making progress! On Wed, Oct 4,

Re: Re: Nifi 1.4: problem with QueryRecord Precessor

Hey Matt, Thank you. I think my specific issue is a misconfig with the schema types.looking into it an will report back On Wed, Oct 4, 2017 at 9:33 AM Matt Burgess wrote: > All, > > The known issue Mark is referring to is NIFI-4349 [1], however it is > not

Re: nifi 1.4 - variables

Can't say which is more useful--registry or variables, but I'm delighted to have both now. The registry was already a huge step toward solving problems of this sort. These variables are also very, very welcome! I love the scope semantic that comes along with them. --what Uwe said about NiFi.

Aw: Re: nifi 1.4 - variables

thank you Bryan.   is there documentation around for the variables?   If I add variables to the processor group and I want to reference a variable of the parent flow, will I have to add the scope somehow like ${scope.variable}?   If I have variables in different hierarchies of the flow

Re: Re: nifi 1.4 - variables

Thanks! There is a JIRA to add additional documentation to the user guide [1]. For the previously existing capabilities with the properties file, there is some mention in the admin guide [2]. The variables work in a hierarchical manner, meaning if a variable "foo" is declared at the root group

Aw: Re: Re: Nifi 1.4: problem with QueryRecord Precessor

  thanks Matt.   Uwe Gesendet: Mittwoch, 04. Oktober 2017 um 17:32 Uhr Von: "Matt Burgess" An: users@nifi.apache.org Betreff: Re: Re: Nifi 1.4: problem with QueryRecord Precessor All, The known issue Mark is referring to is NIFI-4349 [1], however it is not causing the

Re: A bag of groovy questions regarding the ExecuteScript processor

Hi Matt, Thanks for the answers. session.get(N).each) Good to know, I thought a roll-back was inevitable with uncatched exceptions; ScriptTester) Since you're here: I've could only get the script to download when adding this to the `repositories` in the `.build` ``` maven {

3 of 7 Expand all Print all In new window SiteToSiteProvenanceReportingTask reports no data when event type filter is defined

Dear community. I'm currently trying to set up a SiteToSiteProvenanceReportingTask to keep track of DOWNLOAD provenance events. If I set it up without any filters at all everything gets sent to my dedicated NiFi instance for handling these messages. When I set a filter in the "Event Type"

Re: SiteToSiteProvenanceReportingTask reports no data when event type filter is defined

*sigh* - Sorry about the subject, accidental paste. On Wed, Oct 4, 2017 at 10:50 PM, Fred S wrote: > Dear community. > > I'm currently trying to set up a SiteToSiteProvenanceReportingTask to > keep track of DOWNLOAD provenance events. > > If I set it up without any filters