I tried version 1.28.1 and I have the same results.
-----Original Message----- From: Bryan Bende <bbe...@gmail.com> Sent: Wednesday, February 19, 2025 10:18 AM To: users@nifi.apache.org Subject: Re: How do you update various instances/environments based on a single source of a Flow stored and maintained in Git? CAUTION: This email originated from outside the organization. ________________________________ When you get this error message: Failed to perform update flow request due to StandardProcessGroup[identifier=d80b3ed3-8e45-3cf4-d1e9-da6b28c97392,name=Baseline Processing Group] cannot be updated to the proposed flow because the proposed flow does not contain a match for Connection[ID=7ad36423-e0ef-3287-d087-87633b60ab1e, Source ID=7f446e33-2bfc-3e52-8736-910297920464, Dest ID=aaa13731-1598-3ae2-249b-2e96c0af1212] and the connection currently has data in the queue. Can you determine if this connection is one that would still exist in the next version of the flow? Meaning, saying you have ProcessorA -> Connection1 -> ProcessorB, if there is data in the queue for Connection1 and the next version of the flow no longer contains Connection1, then the above error message is correct and you can't change version because it would cause data loss to remove the connection. However, I suspect you might be hitting a bug that has since been resolved, and NiFi is confused about comparing the components from the next version against existing components, and it thinks this connection is going away, but it isn't. I don't have a JIRA off the top of my head, but I would consider trying to get on the latest 1.x first, or better yet 2x. In general, using separate registries per environment and needing to do exports and imports is fighting against the grain a bit. Registry itself was really meant to represent a git repo, meaning there would be one that all NiFi's would connect to, in practice this doesn't always work out. In NiFi 2.x there are not direct Git-based impls of a registry client so that NiFi can directly use GitHub and BitBucket and eventually others, without needing to have NiFi Registry in the middle. Most organization would be ok with Dev, Test, Prod all connecting to same GitHub repo, but not ok with them all connecting to the same NiFi Registry. -Bryan On Wed, Feb 19, 2025 at 8:17 AM Marton Szasz <sza...@apache.org> wrote: > > I've sent you a Slack invitation. > > Cheers, > Marton > > On 2/19/25 2:02 PM, David Steiner wrote: > > > > This is what I see for Slack – either use Gmail or Apple or some > > address @apahe.org. My work e-mail is none of those things. > > > > > > > > > > As far as my questions go: > > > > I have attempted to use a versioned flow in one environment’s > > Registry to export and import into another environment’s Registry. > > We are using Parameter Context and I have set up each environment to > > have that environment’s specific values for each Parameter Context set. > > I have called the Parameter Context in every environment > > BaselineContext – the same thing in each instance. > > > > Every time I try to do the export from one and the important into > > another, I attempt to “upgrade” to NiFi Flow to the new version and > > I get a failure unless all queues are empty and when they are > > emptied and I can “upgrade”, all processors that were running are turned > > off. > > > > Here are details of the last thing I tried, after many, many > > different > > things: > > > > I recreated NiFi from the us-43949 tag of the Development branch in > > Git. This us-43949 tag is the point where a commit was made PRIOR > > to the changes to the Baseline Template went into the Development > > branch. This created a versioned Flow. > > > > I deleted the Flow and the Controller Services from NiFi. I kept > > the Context Parameters. > > > > I uploaded the Template from Development, which contains an updated > > BaselineTemplate – it even had the name of the template changed for > > some reason – not sure why Developer did this. > > > > I dragged the new Template onto the canvas and went through the > > process of setting all of the Context Parameters and then enabling > > all Controller Services. Then turning on all appropriate Processors > > > > > > > > And starting Version Control on it, so now I have two flows in > > Registry – the original one and the new one: > > > > > > Now, in theory, I should be able to change NiFi back to the original > > flow by downloading the original flow from Registry and uploading it > > to the new flow’s version control and then changing the version in > > NiFi. Not really the right direction (we’d normally be downloading > > a new version and uploading it to an environment that has the old > > version so that we can “upgrade” the old version in a different > > environment), but I’d assume this should work. > > > > And/Or, in theory, I should be able to once again tear down the > > existing Flow and Controller Services to start with a “clean slate”. > > Create a ProcessGroup by importing the Original one from Registry. > > Then, export the new one from Registry and upload it into the > > original version in Registry. This should then allow me to update > > the Version in NiFi which would bring in the new version. > > > > > > > > If this works while the previous methods have failed, it means that > > there is literally no way to have development done on one NiFi host > > “transferred/transported” to another NiFi host. > > > > > > This does not really jive with what can be found in the pictures of > > this article: > > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fme > > dium.com%2Fdevops-central%2Fintegrating-git-with-nifi-registries-a-s > > tep-by-step-guide-f9f74c2ba046&data=05%7C02%7CDavid.Steiner%40voyate > > k.com%7C284e0434cd454704db0c08dd50f8b2bb%7C2ba07ab2dc574ebf81ff21f6c > > 5ae437d%7C0%7C0%7C638755751281899098%7CUnknown%7CTWFpbGZsb3d8eyJFbXB > > 0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCI > > sIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=kaTT1Nhggg%2F8il4uBkbCmSomC%2F7 > > rWHXf81P2LAqKaXE%3D&reserved=0 > > > > > > > > > > So, going with the “original” added to the “new” baseline flow: > > > > > > > > > > > > > > > > > > It actually didn’t fail, but it does stop all Processors. > > > > > > > > And I can upload the new flow again to put it at the “top”: > > > > > > > > But I didn’t have anything in the queues, so I have to test that. I > > can get 31 flowfiles into the queue by starting the Real Time > > Processing because of the Test Processors that start: > > > > > > > > So, now I try to change versions, and I’m back to getting the Error: > > > > Failed to perform update flow request due to > > StandardProcessGroup[identifier=d80b3ed3-8e45-3cf4-d1e9-da6b28c97392 > > ,name=Baseline Processing Group] cannot be updated to the proposed > > flow because the proposed flow does not contain a match for > > Connection[ID=7ad36423-e0ef-3287-d087-87633b60ab1e, Source > > ID=7f446e33-2bfc-3e52-8736-910297920464, Dest > > ID=aaa13731-1598-3ae2-249b-2e96c0af1212] and the connection > > currently has data in the queue. > > > > > > Therefore, I conclude that it is apparently impossible to push > > changes to flows to ANY instance of NiFi, even from a copy of a Flow > > from that instance. > > > > I can’t see how this application can be updated from a manual method > > let alone an automated method. > > > > > > David > > > > > > *From:*Joe Witt <joe.w...@gmail.com> > > *Sent:* Tuesday, February 18, 2025 3:04 PM > > *To:* users@nifi.apache.org > > *Subject:* Re: How do you update various instances/environments > > based on a single source of a Flow stored and maintained in Git? > > > > CAUTION: This email originated from outside the organization. > > > > -------------------------------------------------------------------- > > ---- > > > > > > > > David, > > > > This link [1] does not work for a slack invite? > > > > A higher level response: > > > > - The intent is you store versioned flows in the registry > > > > - In NiFi instances you instantiate an instance of a versioned flow. > > > > - As you make changes to the flow and bump its version in the > > registry you elect/choose when to take the updates in your various nifi > > instances. > > > > - You do not have to empty out flows - it will take care of most > > situations. The caveat being if you make changes that could create > > state problems like removing connections which could have data in > > them the flow update would fail until you resolve the connections > > with data in them. > > > > - The automation to update a given flow version change can/should be > > done in a CI/CD tool or process that you control which leverages > > NiFi APIs to programmatically do what a user would otherwise do. > > > > - Leverage parameters in your versioned flows and then parameter > > contexts unique to each environment so the flow versions don't have > > to change but your nifi instances have their own param values. > > > > [1] > > https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fjo > > in.slack.com%2Ft%2Fapachenifi%2Fshared_invite%2Fzt-2ccusmst2-l2KrTzJ > > LrGcHOO0V7~XD4g&data=05%7C02%7CDavid.Steiner%40voyatek.com%7C284e043 > > 4cd454704db0c08dd50f8b2bb%7C2ba07ab2dc574ebf81ff21f6c5ae437d%7C0%7C0 > > %7C638755751281920605%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydW > > UsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D% > > 3D%7C0%7C%7C%7C&sdata=TVPGITcOxcOpwqbYk2ky%2F5xnbU4jqMRLGNGvtC%2Bx1z > > E%3D&reserved=0 > > > > Thanks > > > > On Tue, Feb 18, 2025 at 12:47 PM David Steiner > > <david.stei...@voyatek.com> wrote: > > > > We are operating with version 1.24.0 of NiFi and Registry. > > > > We have been struggling with how to get a Baseline Flow for a > > Product to be 1) Deployable to a Brand new Instance of > > NiFi/Registry environment and more so to 2) updated in that > > environment. > > > > We have somewhat tackled the deployment, but we have no idea > > whether it is sustainable because we can’t figure out how to > > update all our existing environments – DEV, QA, UAT, PROD, etc…. > > > > We created a Baseline Flow Template and stored that in Azure > > DevOps Git. > > We then wrote scripts that we take that Template and “deploy” it > > into a NiFi/Registry environment – we start with a local Docker > > Desktop environment, but these scripts could be used for any > > “blank slate” environment that has a NiFi and Registry installed. > > This all works fine – we use a set of Context Parameters that all > > contain the specific things for each Environment, like server > > names, IDs, Passwords, etc… Thus, we establish the Baseline Flow > > in NiFi and start Version Control on it which puts the Flow in > > Registry. > > > > But we don’t know what to do when we want to update this Template > > or Flow. > > > > We have tried several things to figure out how to modify the flow > > and then save these changes in our local instances of NiFi and > > Registry, which is the place where Development happens, – using > > both the “Download flow definition” from NiFi and the “Export > > version” from Registry. But neither option allows us to load these > > “Flows” into the Registry of another environment and update the > > existing Baseline Flows there with the use of Version Control. We > > always get errors about there being flowfiles in queues. Message > > like this: > > Failed to perform update flow request due to > > > > StandardProcessGroup[identifier=d80b3ed3-8e45-3cf4-d1e9-da6b28c97392,name=Baseline > > Processing Group] cannot be updated to the proposed flow because > > the proposed flow does not contain a match for > > Connection[ID=7ad36423-e0ef-3287-d087-87633b60ab1e, Source > > ID=7f446e33-2bfc-3e52-8736-910297920464, Dest > > ID=aaa13731-1598-3ae2-249b-2e96c0af1212] and the connection > > currently has data in the queue. > > > > I have watch a video about how a Flow can be connected to a Git > > repository that is set up on the Registry machine, but I’m not > > sure how that would enable changes made locally to be applied to > > other environments. > > > > Is there some guidance or documentation about how to do updates to > > various instances/environments based on a single source of a Flow > > stored and maintained in Git? > > > > Is it always the case that if you update the Flow with Version > > Control that all queues have to be empty and all of the running > > processors are turned off after the update? Does this mean that > > the pattern for a Baseline Flow must have “entry” processes > > outside of it that will all flowfiles to queue up outside of it so > > that updates to the Flow can happen after the queues in the Flow > > have been emptied? > > > > If there is a better place to ask this question, what is it? I > > couldn’t figure out how to get access to the Slack group. > > > > Thanks, > > David > > > > A black and blue text Description automatically generated > > > > *David Steiner* > > > > *Technical Architect* > > > > *RevHub* > > > > *Cell: (513) 594-1737* > >
