Hi Josef, I would like to inform you that there is an adjustment for the ticket I mentioned in my previous mail, which is needed for the proper behaviour of the function in every case. The change is in PR phase but I hope it will be merged soon: https://github.com/apache/nifi/pull/6741 <https://github.com/apache/nifi/pull/6741> I would not recommend to go forward without this additional change!
Regards, Bence > On 2022. Nov 29., at 16:21, Simon Bence <[email protected]> wrote: > > Hi Josef, > > Your welcome! In your case it looks like the information is lost during > transformations of the internal representation. > > For technical details, here you can take a look on the code changes: > https://github.com/apache/nifi/commit/df2147829742129c039b37c5d6f4f11aa54785a2 > > <https://github.com/apache/nifi/commit/df2147829742129c039b37c5d6f4f11aa54785a2> > > Regards, > Bence > >> On 2022. Nov 29., at 16:08, <[email protected] >> <mailto:[email protected]>> <[email protected] >> <mailto:[email protected]>> wrote: >> >> Hi Bence, Joe >> >> Thank you guys for your fast response. >> >> @Bence you are right, the whole issue seems to be related to nested flows >> which we try to sync. We are relying heavily on this as we are doing the >> integration on one NiFi system and then sync it to production via the NiFi >> Registry. Our parent flow contains multiple nested flows. >> >> Ok, as NIFI-10874 will be included in the next release we will skip the >> 1.19.0 and wait for 1.19.1 / 1.20.0. Sadly it will probably take more than a >> few days until we see the next release... Do you have an idea why the flow >> with the nested flow could lose the storageLocation? Because, we just change >> a small thing in one of the nested flows and suddenly one of the other >> nested flows have been losing it’s storageLocation in the parent flow and >> from this point the whole flow was broken. Now we are scared to change >> anything as it could brake anytime again. >> >> Cheers Josef >> >> >> >> From: Simon Bence <[email protected] >> <mailto:[email protected]>> >> Reply to: "[email protected] <mailto:[email protected]>" >> <[email protected] <mailto:[email protected]>> >> Date: Tuesday, 29 November 2022 at 13:31 >> To: "[email protected] <mailto:[email protected]>" >> <[email protected] <mailto:[email protected]>> >> Subject: Re: NiFi Registry Bug which brakes the flow sync with NiFi 1.18.0 >> (and same version of the registry) on nested flows >> >> Hi Josef, >> >> Thanks for your patience! >> >> I took a deeper look on what you were writing. In general, this is a sign >> for the case where the registry client cannot find the nested flow. As far >> as I understood in your case this happens when you are having a sync. Based >> on this you may be hitting https://issues.apache.org/jira/browse/NIFI-10874 >> <https://issues.apache.org/jira/browse/NIFI-10874> which is not part of NiFi >> 1.19.0. The next release should contain the fix that addresses your issue. >> >> Regards, >> Bence >> >> >>> On 2022. Nov 28., at 17:48, Simon Bence <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> Hi Josef, >>> >>> Thank you for raising the attention to this. >>> >>> The storageLocation is a new concept to generalise the information used to >>> locate nested flows in a versioned flow. In general it can be there and >>> does not cause issues, as of now NiFi falls back the previous way to locate >>> nested flows. As Joe mentioned, there is a fix in 1.19, but I need to >>> double check if it relates to this situation. Please give me some time, I >>> will find you back. >>> >>> Regards, >>> Bence >>> >>> >>>> On 2022. Nov 28., at 17:17, Joe Witt <[email protected] >>>> <mailto:[email protected]>> wrote: >>>> >>>> Josef >>>> >>>> Sorry for the challenges you've hit there. I do think in 1.18 we had a >>>> bug/regression as we refactored our registry client. That issue should be >>>> resolved in 1.19 which just went live thanks to >>>> https://issues.apache.org/jira/browse/NIFI-10787 >>>> <https://issues.apache.org/jira/browse/NIFI-10787>. However, I am not >>>> positive if this will solve the scenario you've hit now but please if able >>>> try it out. >>>> >>>> Thanks >>>> >>>> On Mon, Nov 28, 2022 at 8:50 AM <[email protected] >>>> <mailto:[email protected]>> wrote: >>>> Hi guys >>>> >>>> >>>> >>>> We had the following bug twice already and it broke the whole sync with >>>> the NiFi registry for the Flow/PG. First time was directly after we have >>>> upgraded from NiFi 1.15.3 to 1.18.0, but we ignored it as we thought it >>>> could be because of the upgrade, however it occurred again after a few >>>> NiFi Registry commits on NiFi 1.18.0… The error was the following when we >>>> tried to change the flow version or when we tried to start from scratch >>>> with that version from the NiFi Registry, so the version was broken in the >>>> NiFi Registry: >>>> >>>> >>>> >>>> <image001.png> >>>> >>>> >>>> >>>> >>>> >>>> We investigated the last NiFi Registry Commit and we saw in our GIT repo >>>> (we sync the “flow_storage” to GIT) on line 4847 that NiFi removed a >>>> “storageLocation” from another PG which makes no sense at all. We changed >>>> nothing there and especially why should NiFi remove only the >>>> storageLocation line… We have one specialty, as we have nested NiFi >>>> Registry flows, so one of the flows where the storageLocation has been >>>> removed was such a nested flow. >>>> >>>> >>>> >>>> >>>> >>>> <image002.png> >>>> >>>> >>>> >>>> >>>> >>>> Luckily we were able to resolve the error. We tried to add the line and >>>> commit it to GIT plus we dropped the the database to repopulate the DB, >>>> however it was broken again after a commit from NiFi. So we tried to >>>> manually create a new fake version on bucket.yml in the corresponding >>>> bucket folder and added as well the line in the snapshot again. We dropped >>>> then the DB and restarted NiFi Registry and voilà it was working again. >>>> >>>> >>>> >>>> However it was a nightmare to get it working again as the flow was >>>> completely broken, we couldn’t checkout the affected version at all. >>>> >>>> >>>> >>>> Any thoughts on this? Shall I fill a Jira Ticket? The problem is, we can’t >>>> really reproduce it, it looks like it happens randomly. As you could >>>> imagine, we can’t share our template as it contains a lot of confidential >>>> material. >>>> >>>> >>>> >>>> Cheers Josef >>>> >
