Yes, that certainly involves a lot of effort. I wonder whether it's a good idea to fix a possible "design flaw" with a construct that is neither consistent nor easy to handle.
I also think the argument that you have to adjust a lot in the flow is questionable. With Release 2.0, so much has to be adapted (cron jobs, database schema controller etc.) that it hardly matters whether you adapt the UpdateAttribute processors or not. The effort to adapt and test everything is already present with the upgrade to 2.0. Doesn't Release 2.0 actually offer a really good opportunity to adapt everything that has been put off for a long time? Who knows when the next opportunity will come? With Release 3.0? Or not at all, because NiFi is becoming more and more widespread and the customer lobby is getting bigger and bigger? Cloudera will find it increasingly difficult to pack "large-scale" customizations into a release. Whether we wait 3-4 months or 5-6 months for the 2.0 release doesn't really matter, does it? But what do I know ... the NiFi experts and full-time maintainers will certainly do the right thing. -- Uwe > Am 09.02.2024 um 22:04 schrieb Michael Moser <moser...@gmail.com>: > > These are great ideas! I do love Adam's dynamic relationship idea over > creating a failure relationship that is auto-terminated. This would make > flow migrations in Registry and NiFi easier. > > After some more pondering, I (slowly) realized that this problem affects > more than UpdateAttribute, though. You can easily get expression language > (explang) exceptions anywhere that explang is used. I'm sure all of us can > create a RouteOnAttribute configuration that causes an explang exception > which rolls back the flowfile. If we spend so much effort on a solution it > would be a shame for that to only apply to UpdateAttribute. > > For this reason I would favor Matt's idea of a try() or trycatch() explang > method. How it might work isn't intuitive, though. Would we have to make > it aware of a data type to return? RouteOnAttribute expects boolean but > UpdateAttribute expects string (or convertible to string). > > I can see why the rollback/admin yield solution has been status quo for so > long. > > -- Mike > > > > >> On Fri, Feb 9, 2024 at 10:03 AM u...@moosheimer.com <u...@moosheimer.com> >> wrote: >> >> 👍 >> --Uwe >> >>>> Am 09.02.2024 um 13:50 schrieb Mike Thomsen <mikerthom...@gmail.com>: >>> >>> How about a third option which is to provide three options: >>> >>> 1) Default - status quo, exceptions cause it to yield >>> 2) Exception = moves forward to success w/ an error attribute, an error >> log >>> statement that triggers a bulletin, etc to let data manages know what's >>> happening. >>> 3) Exception = moves to a failure relationship that is otherwise >>> autoterminated >>> >>>> On Thu, Feb 8, 2024 at 7:12 PM Matt Burgess <mattyb...@apache.org> >> wrote: >>>> >>>> Mike's option #2 seems solid but would take a lot of work and there will >>>> always be inputs we don't account for. I support that work but in code >>>> sometimes we just do a "catch(Throwable)" just so it doesn't blow up. >> What >>>> about a subjectless "try" or "trycatch" function you can wrap around >> your >>>> whole expression? If no exception is thrown, the evaluated value will be >>>> returned but if one is thrown, you can provide some alternate value that >>>> you can check downstream. As this is optional it would retain the >> current >>>> behavior unless you use it, and then it takes the place of all those >>>> ifElse(isXYZValid()) calls we'd need throughout the expression. >>>> >>>> Regards, >>>> Matt >>>> >>>> >>>> On Wed, Feb 7, 2024 at 8:11 PM Phillip Lord <phillord0...@gmail.com> >>>> wrote: >>>> >>>>> IMO... UpdateAttribute has been around since the beginning of time, I >>>> can't >>>>> see adding a failure relationship. At the same time I understand the >> want >>>>> for such exceptions to be handled more gracefully rather than rolling >>>> back >>>>> indefinitely. >>>>> I'd vote in favor of considering Moser's option #2... and being able to >>>>> implement an "if this then that" logic within your flow. >>>>> >>>>> Also just thinking... for every UA failure you have to consider a good >>>>> failure-management strategy, which MIGHT add a lot of noise to the >> flow. >>>>> Something that might otherwise easily be identified in a downstream >>>>> component and/or database/etc. >>>>> >>>>> My 2 cents ** >>>>> Phil >>>>> >>>>> >>>>> >>>>> >>>>> >>>>>> On Wed, Feb 7, 2024 at 5:18 PM Adam Taft <a...@adamtaft.com> wrote: >>>>> >>>>>> Or better, the failure relationship just doesn't even exist until the >>>>>> property "Has Failure Relationship" is set to True. This involves >>>>> updating >>>>>> UpdateAttribute to have dynamic relationships (the failure >>>> relationships >>>>>> appearing on true), which isn't hard to do in processor code. >>>>>> >>>>>> This has the advantage of being backwards compatible for existing >> users >>>>> and >>>>>> allows the failure relationship to exist for new configurations. >>>>> Obviously >>>>>> the processor would need an update to catch Expression Language >>>>> exceptions >>>>>> and then route conditionally to failure. >>>>>> >>>>>> Just thinking out loud. >>>>>> /Adam >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Feb 7, 2024 at 1:48 PM u...@moosheimer.com <u...@moosheimer.com >>> >>>>>> wrote: >>>>>> >>>>>>> Hi Mike, >>>>>>> >>>>>>> How about the option of introducing a new property that decides >>>> whether >>>>>> to >>>>>>> route to the 'failure' relationship in the event of an error? >>>>>>> If this property is set to false, then the 'failure' relationship is >>>>>>> automatically set to 'terminate' (since nothing is routed there >>>>> anyway). >>>>>>> >>>>>>> Then everyone can decide whether and where they want to use this new >>>>>>> feature or not. >>>>>>> All other options would still be possible with such a solution. >>>>>>> >>>>>>> -- Uwe >>>>>>> >>>>>>>> Am 07.02.2024 um 22:15 schrieb Michael Moser <moser...@gmail.com>: >>>>>>>> >>>>>>>> Hi Dan, >>>>>>>> >>>>>>>> This has been discussed in the past, as you found with those two >>>> Jira >>>>>>>> tickets. Personally, I'm still not sure whether a new failure >>>>>>> relationship >>>>>>>> on UpdateAttribute in 2.0 is a good approach. I have heard from >>>> some >>>>>>>> dataflow managers that would not want to go through their entire >>>>> graph >>>>>>> when >>>>>>>> upgrading to 2.0 and update every UpdateAttribute configuration. >>>>>>>> >>>>>>>> I have heard some alternatives to a 'failure' relationship that I >>>>> would >>>>>>>> like to share as options. >>>>>>>> >>>>>>>> 1) Add a new property to UpdateAttribute that controls whether a >>>>>> flowfile >>>>>>>> that causes an expression language exception either yields and >>>> rolls >>>>>>> back, >>>>>>>> or silently fails to update the attribute and sends the flowfile to >>>>>>>> success. I personally don't like this, because the use case for >>>>>> "silent >>>>>>>> failure" seems really like a rarely needed edge case. >>>>>>>> >>>>>>>> 2) Identify all expression language methods that can throw an >>>>> exception >>>>>>> and >>>>>>>> document that fact in the Expression Language Guide (some methods >>>>>> already >>>>>>>> mention they can throw an "exception bulletin"). Then implement >>>> new >>>>>>>> expression methods to check if an expression could fail, and use >>>> that >>>>>> in >>>>>>>> UpdateAttribute advanced rules. For example, if the format() and >>>>>>>> formatInstant() methods can fail on a negative number, we create a >>>>> new >>>>>>>> method such as isValidMilliseconds(). This already exists for some >>>>>>> cases, >>>>>>>> such as isJson() which can do a quick check of some value before >>>>>> calling >>>>>>>> jsonPathDelete() on it. >>>>>>>> >>>>>>>> I'm curious to hear more thoughts on this. >>>>>>>> >>>>>>>> -- Mike >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> On Wed, Jan 31, 2024 at 11:02 AM Dan S <dsti...@gmail.com> wrote: >>>>>>>>> >>>>>>>>> My team is requesting a failure relationship for UpdateAttribute >>>> as >>>>>>> seen in >>>>>>>>> NIFI-5448 <https://issues.apache.org/jira/browse/NIFI-5448> and >>>>>>> NIFI-6344 >>>>>>>>> <https://issues.apache.org/jira/browse/NIFI-6344> as we are >>>>>>>>> experiencing the same problem where a NIFI Expression Language is >>>>>>> throwing >>>>>>>>> an exception. In the PR for NIFI-5448 it was mentioned this >>>> feature >>>>>>> would >>>>>>>>> have to wait until NIFI 2.0.0. I wanted to know if there is any >>>>> active >>>>>>> work >>>>>>>>> regarding this and whether eventually there will be a failure >>>>>>> relationship >>>>>>>>> added to UpdateAttribute? >>>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>
