We are operating with version 1.24.0 of NiFi and Registry. We have been struggling with how to get a Baseline Flow for a Product to be 1) Deployable to a Brand new Instance of NiFi/Registry environment and more so to 2) updated in that environment.
We have somewhat tackled the deployment, but we have no idea whether it is sustainable because we can't figure out how to update all our existing environments - DEV, QA, UAT, PROD, etc.... We created a Baseline Flow Template and stored that in Azure DevOps Git. We then wrote scripts that we take that Template and "deploy" it into a NiFi/Registry environment - we start with a local Docker Desktop environment, but these scripts could be used for any "blank slate" environment that has a NiFi and Registry installed. This all works fine - we use a set of Context Parameters that all contain the specific things for each Environment, like server names, IDs, Passwords, etc... Thus, we establish the Baseline Flow in NiFi and start Version Control on it which puts the Flow in Registry. But we don't know what to do when we want to update this Template or Flow. We have tried several things to figure out how to modify the flow and then save these changes in our local instances of NiFi and Registry, which is the place where Development happens, - using both the "Download flow definition" from NiFi and the "Export version" from Registry. But neither option allows us to load these "Flows" into the Registry of another environment and update the existing Baseline Flows there with the use of Version Control. We always get errors about there being flowfiles in queues. Message like this: Failed to perform update flow request due to StandardProcessGroup[identifier=d80b3ed3-8e45-3cf4-d1e9-da6b28c97392,name=Baseline Processing Group] cannot be updated to the proposed flow because the proposed flow does not contain a match for Connection[ID=7ad36423-e0ef-3287-d087-87633b60ab1e, Source ID=7f446e33-2bfc-3e52-8736-910297920464, Dest ID=aaa13731-1598-3ae2-249b-2e96c0af1212] and the connection currently has data in the queue. I have watch a video about how a Flow can be connected to a Git repository that is set up on the Registry machine, but I'm not sure how that would enable changes made locally to be applied to other environments. Is there some guidance or documentation about how to do updates to various instances/environments based on a single source of a Flow stored and maintained in Git? Is it always the case that if you update the Flow with Version Control that all queues have to be empty and all of the running processors are turned off after the update? Does this mean that the pattern for a Baseline Flow must have "entry" processes outside of it that will all flowfiles to queue up outside of it so that updates to the Flow can happen after the queues in the Flow have been emptied? If there is a better place to ask this question, what is it? I couldn't figure out how to get access to the Slack group. Thanks, David [A black and blue text Description automatically generated] David Steiner Technical Architect RevHub Cell: (513) 594-1737
