Just to recap/level set... The distinct between yielding and penalization is important. Penalization is an action taken on a FlowFile because the FlowFile cannot be processed right now (like a naming conflict for instance). The Processor is indicating that it cannot process that specific FlowFile at the moment but may be able to process the next. Yielding is an indication that the Processor is unable to work at all at the moment likely due to an environmental issue (like the out of service comment).
If the concept of penalization were moved to a connection, does it automatically penalize all FlowFile transferred to it? We would lose some granularity if a Processor wanted to penalize some FlowFile routed to a given Relationship but not others. I'm not sure if this is done in practice or not, just wanted to mention it. Outside of this minor concern, I like the idea. I especially like that it would help with the consistency of Processor behavior and transparency about what the data flow is actually doing. Matt On Thu, Jan 28, 2016 at 12:00 PM, Michael de Courci < [email protected]> wrote: > Hi > I think it would be better/simpler to have one “out of service” concept > to replace penalizing and yielding and when a plugin throws an exception > then the plugin is deemed out of service, for a duration and so the > connection to that plugin is disabled for the out of service duration. > > When a plugin is out of service and the connection disabled - then > resources that it uses will be freed(yielded). > > The question then is what the behaviour of the plugin before the disabled > connection - should be. My thought is to tend towards stability and make > sure resources are freed, so there may need to be a “domino effect”/cascade > affect where all plugins before are gradually put out of service. > > > With Kind Regards > Michael de Courci > [email protected] > > > > > > On 28 Jan 2016, at 16:34, Mark Payne <[email protected]> wrote: > > > > All, > > > > I've been thinking about how we handle the concept of penalizing > FlowFiles. We've had a lot of questions > > lately about how penalization works & the concept in general. Seems the > following problems exist: > > > > - Confusion about difference between penalization & yielding > > - DFM sees option to configure penalization period on all processors, > even if they don't penalize FlowFiles. > > - DFM cannot set penalty duration in 1 case and set a different value > for a different case (different relationship, for example). > > - Developers often forget to call penalize() > > - Developer has to determine whether or not to penalize when building a > processor. It is based on what the developer will > > think may make sense, but in reality DFM's sometimes want to penalize > things when the processor doesn't behave that way. > > > > I'm wondering if it doesn't make sense to remove the concept of > penalization all together from Processors and instead > > move the Penalty Duration so that it's a setting on the Connection. I > think this would clear up the confusion and give the DFM > > more control over when/how long to penalize. Could set to the default to > 30 seconds for self-looping connections and no penalization > > for other connections. > > > > Any thoughts? > > > > Thanks > > -Mark > >
