Hi Josef, Your welcome! In your case it looks like the information is lost during transformations of the internal representation.
For technical details, here you can take a look on the code changes: https://github.com/apache/nifi/commit/df2147829742129c039b37c5d6f4f11aa54785a2 <https://github.com/apache/nifi/commit/df2147829742129c039b37c5d6f4f11aa54785a2> Regards, Bence > On 2022. Nov 29., at 16:08, <[email protected]> > <[email protected]> wrote: > > Hi Bence, Joe > > Thank you guys for your fast response. > > @Bence you are right, the whole issue seems to be related to nested flows > which we try to sync. We are relying heavily on this as we are doing the > integration on one NiFi system and then sync it to production via the NiFi > Registry. Our parent flow contains multiple nested flows. > > Ok, as NIFI-10874 will be included in the next release we will skip the > 1.19.0 and wait for 1.19.1 / 1.20.0. Sadly it will probably take more than a > few days until we see the next release... Do you have an idea why the flow > with the nested flow could lose the storageLocation? Because, we just change > a small thing in one of the nested flows and suddenly one of the other nested > flows have been losing it’s storageLocation in the parent flow and from this > point the whole flow was broken. Now we are scared to change anything as it > could brake anytime again. > > Cheers Josef > > > > From: Simon Bence <[email protected] <mailto:[email protected]>> > Reply to: "[email protected] <mailto:[email protected]>" > <[email protected] <mailto:[email protected]>> > Date: Tuesday, 29 November 2022 at 13:31 > To: "[email protected]" <[email protected]> > Subject: Re: NiFi Registry Bug which brakes the flow sync with NiFi 1.18.0 > (and same version of the registry) on nested flows > > Hi Josef, > > Thanks for your patience! > > I took a deeper look on what you were writing. In general, this is a sign for > the case where the registry client cannot find the nested flow. As far as I > understood in your case this happens when you are having a sync. Based on > this you may be hitting https://issues.apache.org/jira/browse/NIFI-10874 > <https://issues.apache.org/jira/browse/NIFI-10874> which is not part of NiFi > 1.19.0. The next release should contain the fix that addresses your issue. > > Regards, > Bence > > >> On 2022. Nov 28., at 17:48, Simon Bence <[email protected] >> <mailto:[email protected]>> wrote: >> >> Hi Josef, >> >> Thank you for raising the attention to this. >> >> The storageLocation is a new concept to generalise the information used to >> locate nested flows in a versioned flow. In general it can be there and does >> not cause issues, as of now NiFi falls back the previous way to locate >> nested flows. As Joe mentioned, there is a fix in 1.19, but I need to double >> check if it relates to this situation. Please give me some time, I will find >> you back. >> >> Regards, >> Bence >> >> >>> On 2022. Nov 28., at 17:17, Joe Witt <[email protected] >>> <mailto:[email protected]>> wrote: >>> >>> Josef >>> >>> Sorry for the challenges you've hit there. I do think in 1.18 we had a >>> bug/regression as we refactored our registry client. That issue should be >>> resolved in 1.19 which just went live thanks to >>> https://issues.apache.org/jira/browse/NIFI-10787. However, I am not >>> positive if this will solve the scenario you've hit now but please if able >>> try it out. >>> >>> Thanks >>> >>> On Mon, Nov 28, 2022 at 8:50 AM <[email protected]> wrote: >>> Hi guys >>> >>> >>> >>> We had the following bug twice already and it broke the whole sync with the >>> NiFi registry for the Flow/PG. First time was directly after we have >>> upgraded from NiFi 1.15.3 to 1.18.0, but we ignored it as we thought it >>> could be because of the upgrade, however it occurred again after a few NiFi >>> Registry commits on NiFi 1.18.0… The error was the following when we tried >>> to change the flow version or when we tried to start from scratch with that >>> version from the NiFi Registry, so the version was broken in the NiFi >>> Registry: >>> >>> >>> >>> <image001.png> >>> >>> >>> >>> >>> >>> We investigated the last NiFi Registry Commit and we saw in our GIT repo >>> (we sync the “flow_storage” to GIT) on line 4847 that NiFi removed a >>> “storageLocation” from another PG which makes no sense at all. We changed >>> nothing there and especially why should NiFi remove only the >>> storageLocation line… We have one specialty, as we have nested NiFi >>> Registry flows, so one of the flows where the storageLocation has been >>> removed was such a nested flow. >>> >>> >>> >>> >>> >>> <image002.png> >>> >>> >>> >>> >>> >>> Luckily we were able to resolve the error. We tried to add the line and >>> commit it to GIT plus we dropped the the database to repopulate the DB, >>> however it was broken again after a commit from NiFi. So we tried to >>> manually create a new fake version on bucket.yml in the corresponding >>> bucket folder and added as well the line in the snapshot again. We dropped >>> then the DB and restarted NiFi Registry and voilà it was working again. >>> >>> >>> >>> However it was a nightmare to get it working again as the flow was >>> completely broken, we couldn’t checkout the affected version at all. >>> >>> >>> >>> Any thoughts on this? Shall I fill a Jira Ticket? The problem is, we can’t >>> really reproduce it, it looks like it happens randomly. As you could >>> imagine, we can’t share our template as it contains a lot of confidential >>> material. >>> >>> >>> >>> Cheers Josef >>>
