Re: UpdateAttribute Failure Relationship

[Phillip Lord]

IMO... UpdateAttribute has been around since the beginning of time, I can't
see adding a failure relationship. At the same time I understand the want
for such exceptions to be handled more gracefully rather than rolling back
indefinitely.
I'd vote in favor of considering Moser's option #2... and being able to
implement an "if this then that" logic within your flow.

Also just thinking... for every UA failure you have to consider a good
failure-management strategy, which MIGHT add a lot of noise to the flow.
Something that might otherwise easily be identified in a downstream
component and/or database/etc.

My 2 cents **
Phil





On Wed, Feb 7, 2024 at 5:18 PM Adam Taft  wrote:

> Or better, the failure relationship just doesn't even exist until the
> property "Has Failure Relationship" is set to True.  This involves updating
> UpdateAttribute to have dynamic relationships (the failure relationships
> appearing on true), which isn't hard to do in processor code.
>
> This has the advantage of being backwards compatible for existing users and
> allows the failure relationship to exist for new configurations. Obviously
> the processor would need an update to catch Expression Language exceptions
> and then route conditionally to failure.
>
> Just thinking out loud.
> /Adam
>
>
>
> On Wed, Feb 7, 2024 at 1:48 PM u...@moosheimer.com 
> wrote:
>
> > Hi Mike,
> >
> > How about the option of introducing a new property that decides whether
> to
> > route to the 'failure' relationship in the event of an error?
> > If this property is set to false, then the 'failure' relationship is
> > automatically set to 'terminate' (since nothing is routed there anyway).
> >
> > Then everyone can decide whether and where they want to use this new
> > feature or not.
> > All other options would still be possible with such a solution.
> >
> > -- Uwe
> >
> > > Am 07.02.2024 um 22:15 schrieb Michael Moser :
> > >
> > > Hi Dan,
> > >
> > > This has been discussed in the past, as you found with those two Jira
> > > tickets.  Personally, I'm still not sure whether a new failure
> > relationship
> > > on UpdateAttribute in 2.0 is a good approach.  I have heard from some
> > > dataflow managers that would not want to go through their entire graph
> > when
> > > upgrading to 2.0 and update every UpdateAttribute configuration.
> > >
> > > I have heard some alternatives to a 'failure' relationship that I would
> > > like to share as options.
> > >
> > > 1) Add a new property to UpdateAttribute that controls whether a
> flowfile
> > > that causes an expression language exception either yields and rolls
> > back,
> > > or silently fails to update the attribute and sends the flowfile to
> > > success.  I personally don't like this, because the use case for
> "silent
> > > failure" seems really like a rarely needed edge case.
> > >
> > > 2) Identify all expression language methods that can throw an exception
> > and
> > > document that fact in the Expression Language Guide (some methods
> already
> > > mention they can throw an "exception bulletin").  Then implement new
> > > expression methods to check if an expression could fail, and use that
> in
> > > UpdateAttribute advanced rules.  For example, if the format() and
> > > formatInstant() methods can fail on a negative number, we create a new
> > > method such as isValidMilliseconds().  This already exists for some
> > cases,
> > > such as isJson() which can do a quick check of some value before
> calling
> > > jsonPathDelete() on it.
> > >
> > > I'm curious to hear more thoughts on this.
> > >
> > > -- Mike
> > >
> > >
> > >
> > >> On Wed, Jan 31, 2024 at 11:02 AM Dan S  wrote:
> > >>
> > >> My team is requesting a failure relationship for UpdateAttribute as
> > seen in
> > >> NIFI-5448  and
> > NIFI-6344
> > >>  as we are
> > >> experiencing the same problem where a NIFI Expression Language is
> > throwing
> > >> an exception. In the PR for NIFI-5448 it was mentioned this feature
> > would
> > >> have to wait until NIFI 2.0.0. I wanted to know if there is any active
> > work
> > >> regarding this and whether eventually there will be a failure
> > relationship
> > >> added to UpdateAttribute?
> > >>
> >
> >
>

Re: UpdateAttribute Failure Relationship

[Adam Taft]

Or better, the failure relationship just doesn't even exist until the
property "Has Failure Relationship" is set to True.  This involves updating
UpdateAttribute to have dynamic relationships (the failure relationships
appearing on true), which isn't hard to do in processor code.

This has the advantage of being backwards compatible for existing users and
allows the failure relationship to exist for new configurations. Obviously
the processor would need an update to catch Expression Language exceptions
and then route conditionally to failure.

Just thinking out loud.
/Adam



On Wed, Feb 7, 2024 at 1:48 PM u...@moosheimer.com 
wrote:

> Hi Mike,
>
> How about the option of introducing a new property that decides whether to
> route to the 'failure' relationship in the event of an error?
> If this property is set to false, then the 'failure' relationship is
> automatically set to 'terminate' (since nothing is routed there anyway).
>
> Then everyone can decide whether and where they want to use this new
> feature or not.
> All other options would still be possible with such a solution.
>
> -- Uwe
>
> > Am 07.02.2024 um 22:15 schrieb Michael Moser :
> >
> > Hi Dan,
> >
> > This has been discussed in the past, as you found with those two Jira
> > tickets.  Personally, I'm still not sure whether a new failure
> relationship
> > on UpdateAttribute in 2.0 is a good approach.  I have heard from some
> > dataflow managers that would not want to go through their entire graph
> when
> > upgrading to 2.0 and update every UpdateAttribute configuration.
> >
> > I have heard some alternatives to a 'failure' relationship that I would
> > like to share as options.
> >
> > 1) Add a new property to UpdateAttribute that controls whether a flowfile
> > that causes an expression language exception either yields and rolls
> back,
> > or silently fails to update the attribute and sends the flowfile to
> > success.  I personally don't like this, because the use case for "silent
> > failure" seems really like a rarely needed edge case.
> >
> > 2) Identify all expression language methods that can throw an exception
> and
> > document that fact in the Expression Language Guide (some methods already
> > mention they can throw an "exception bulletin").  Then implement new
> > expression methods to check if an expression could fail, and use that in
> > UpdateAttribute advanced rules.  For example, if the format() and
> > formatInstant() methods can fail on a negative number, we create a new
> > method such as isValidMilliseconds().  This already exists for some
> cases,
> > such as isJson() which can do a quick check of some value before calling
> > jsonPathDelete() on it.
> >
> > I'm curious to hear more thoughts on this.
> >
> > -- Mike
> >
> >
> >
> >> On Wed, Jan 31, 2024 at 11:02 AM Dan S  wrote:
> >>
> >> My team is requesting a failure relationship for UpdateAttribute as
> seen in
> >> NIFI-5448  and
> NIFI-6344
> >>  as we are
> >> experiencing the same problem where a NIFI Expression Language is
> throwing
> >> an exception. In the PR for NIFI-5448 it was mentioned this feature
> would
> >> have to wait until NIFI 2.0.0. I wanted to know if there is any active
> work
> >> regarding this and whether eventually there will be a failure
> relationship
> >> added to UpdateAttribute?
> >>
>
>

Re: UpdateAttribute Failure Relationship

[u...@moosheimer.com]

Hi Mike,

How about the option of introducing a new property that decides whether to 
route to the 'failure' relationship in the event of an error?
If this property is set to false, then the 'failure' relationship is 
automatically set to 'terminate' (since nothing is routed there anyway).

Then everyone can decide whether and where they want to use this new feature or 
not.
All other options would still be possible with such a solution.

-- Uwe

> Am 07.02.2024 um 22:15 schrieb Michael Moser :
> 
> Hi Dan,
> 
> This has been discussed in the past, as you found with those two Jira
> tickets.  Personally, I'm still not sure whether a new failure relationship
> on UpdateAttribute in 2.0 is a good approach.  I have heard from some
> dataflow managers that would not want to go through their entire graph when
> upgrading to 2.0 and update every UpdateAttribute configuration.
> 
> I have heard some alternatives to a 'failure' relationship that I would
> like to share as options.
> 
> 1) Add a new property to UpdateAttribute that controls whether a flowfile
> that causes an expression language exception either yields and rolls back,
> or silently fails to update the attribute and sends the flowfile to
> success.  I personally don't like this, because the use case for "silent
> failure" seems really like a rarely needed edge case.
> 
> 2) Identify all expression language methods that can throw an exception and
> document that fact in the Expression Language Guide (some methods already
> mention they can throw an "exception bulletin").  Then implement new
> expression methods to check if an expression could fail, and use that in
> UpdateAttribute advanced rules.  For example, if the format() and
> formatInstant() methods can fail on a negative number, we create a new
> method such as isValidMilliseconds().  This already exists for some cases,
> such as isJson() which can do a quick check of some value before calling
> jsonPathDelete() on it.
> 
> I'm curious to hear more thoughts on this.
> 
> -- Mike
> 
> 
> 
>> On Wed, Jan 31, 2024 at 11:02 AM Dan S  wrote:
>> 
>> My team is requesting a failure relationship for UpdateAttribute as seen in
>> NIFI-5448  and NIFI-6344
>>  as we are
>> experiencing the same problem where a NIFI Expression Language is throwing
>> an exception. In the PR for NIFI-5448 it was mentioned this feature would
>> have to wait until NIFI 2.0.0. I wanted to know if there is any active work
>> regarding this and whether eventually there will be a failure relationship
>> added to UpdateAttribute?
>> 

Re: UpdateAttribute Failure Relationship

[Michael Moser]

Hi Dan,

This has been discussed in the past, as you found with those two Jira
tickets.  Personally, I'm still not sure whether a new failure relationship
on UpdateAttribute in 2.0 is a good approach.  I have heard from some
dataflow managers that would not want to go through their entire graph when
upgrading to 2.0 and update every UpdateAttribute configuration.

I have heard some alternatives to a 'failure' relationship that I would
like to share as options.

1) Add a new property to UpdateAttribute that controls whether a flowfile
that causes an expression language exception either yields and rolls back,
or silently fails to update the attribute and sends the flowfile to
success.  I personally don't like this, because the use case for "silent
failure" seems really like a rarely needed edge case.

2) Identify all expression language methods that can throw an exception and
document that fact in the Expression Language Guide (some methods already
mention they can throw an "exception bulletin").  Then implement new
expression methods to check if an expression could fail, and use that in
UpdateAttribute advanced rules.  For example, if the format() and
formatInstant() methods can fail on a negative number, we create a new
method such as isValidMilliseconds().  This already exists for some cases,
such as isJson() which can do a quick check of some value before calling
jsonPathDelete() on it.

I'm curious to hear more thoughts on this.

-- Mike



On Wed, Jan 31, 2024 at 11:02 AM Dan S  wrote:

> My team is requesting a failure relationship for UpdateAttribute as seen in
> NIFI-5448  and NIFI-6344
>  as we are
> experiencing the same problem where a NIFI Expression Language is throwing
> an exception. In the PR for NIFI-5448 it was mentioned this feature would
> have to wait until NIFI 2.0.0. I wanted to know if there is any active work
> regarding this and whether eventually there will be a failure relationship
> added to UpdateAttribute?
>