Andy, Bryan , Thanks, ok, so I wasn't off mark (I call it flow expiration in my original email).
Now another question: I was aware of expiration due to time, but not aware of discarding based on queue levels. Is there a way to do so and if not, should we allow users to discard flowfiles that are stuck in a queue when it reaches a certain level? Kind regards On 4 Mar 2017 05:29, "Andy LoPresto" <[email protected]> wrote: I agree with Bryan’s interpretation here. My understanding of that feature reference is in relation to high volume streams of data coming in and the DFM making proactive and/or realtime decisions about how to prioritize data when they don’t have the capability to capture/operate on all of it. Flowfile expiration and routing to failure (the provenance event is called “drop” for a reason) are explicit design decisions that address the scenario in legacy systems when "bad/less valuable/not currently able to be operated on" data could prevent the "better/more valuable" data from flowing. Andy LoPresto [email protected] *[email protected] <[email protected]>* PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69 On Mar 3, 2017, at 6:15 AM, Bryan Bende <[email protected]> wrote: Andre, My understanding of that statement is that NiFi lets the user decide how to build their dataflow, and allows them to build it in a way that favors guaranteed delivery, or favors loss tolerance, depending on their needs. I would think loss tolerance is referencing the idea of expiring data from queues after a certain threshold, and maybe terminating failure relationships. The overall point being that the user makes the decision and not the framework, which is the same motto for a lot of other parts of NiFi. -Bryan On Thu, Mar 2, 2017 at 11:54 PM, Andre <[email protected]> wrote: All, A good number of NiFi related material makes the following statement about NiFi: Loss tolerant vs guaranteed delivery I have not historically used NiFi in a loss tolerant environment but after investigating the documentation it isn't clear to me how NiFi implements loss tolerance? Are we referring to Clustering? Flow expiration? To the ability to discard flowfiles that get piped to the REL_FAILURE relationships? etc? Would anyone mind shedding some light about what are we referring to when presenting the "Loss tolerant vs guaranteed delivery" I thank you in advance
