I've sent you a Slack invitation.
Cheers,
Marton
On 2/19/25 2:02 PM, David Steiner wrote:
This is what I see for Slack – either use Gmail or Apple or some
address @apahe.org. My work e-mail is none of those things.
As far as my questions go:
I have attempted to use a versioned flow in one environment’s Registry
to export and import into another environment’s Registry.
We are using Parameter Context and I have set up each environment to
have that environment’s specific values for each Parameter Context set.
I have called the Parameter Context in every environment
BaselineContext – the same thing in each instance.
Every time I try to do the export from one and the important into
another, I attempt to “upgrade” to NiFi Flow to the new version and I
get a failure unless all queues are empty and when they are emptied
and I can “upgrade”, all processors that were running are turned off.
Here are details of the last thing I tried, after many, many different
things:
I recreated NiFi from the us-43949 tag of the Development branch in
Git. This us-43949 tag is the point where a commit was made PRIOR to
the changes to the Baseline Template went into the Development
branch. This created a versioned Flow.
I deleted the Flow and the Controller Services from NiFi. I kept the
Context Parameters.
I uploaded the Template from Development, which contains an updated
BaselineTemplate – it even had the name of the template changed for
some reason – not sure why Developer did this.
I dragged the new Template onto the canvas and went through the
process of setting all of the Context Parameters and then enabling all
Controller Services. Then turning on all appropriate Processors
And starting Version Control on it, so now I have two flows in
Registry – the original one and the new one:
Now, in theory, I should be able to change NiFi back to the original
flow by downloading the original flow from Registry and uploading it
to the new flow’s version control and then changing the version in
NiFi. Not really the right direction (we’d normally be downloading a
new version and uploading it to an environment that has the old
version so that we can “upgrade” the old version in a different
environment), but I’d assume this should work.
And/Or, in theory, I should be able to once again tear down the
existing Flow and Controller Services to start with a “clean slate”.
Create a ProcessGroup by importing the Original one from Registry.
Then, export the new one from Registry and upload it into the original
version in Registry. This should then allow me to update the Version
in NiFi which would bring in the new version.
If this works while the previous methods have failed, it means that
there is literally no way to have development done on one NiFi host
“transferred/transported” to another NiFi host.
This does not really jive with what can be found in the pictures of
this article:
https://medium.com/devops-central/integrating-git-with-nifi-registries-a-step-by-step-guide-f9f74c2ba046
So, going with the “original” added to the “new” baseline flow:
It actually didn’t fail, but it does stop all Processors.
And I can upload the new flow again to put it at the “top”:
But I didn’t have anything in the queues, so I have to test that. I
can get 31 flowfiles into the queue by starting the Real Time
Processing because of the Test Processors that start:
So, now I try to change versions, and I’m back to getting the Error:
Failed to perform update flow request due to
StandardProcessGroup[identifier=d80b3ed3-8e45-3cf4-d1e9-da6b28c97392,name=Baseline
Processing Group] cannot be updated to the proposed flow because the
proposed flow does not contain a match for
Connection[ID=7ad36423-e0ef-3287-d087-87633b60ab1e, Source
ID=7f446e33-2bfc-3e52-8736-910297920464, Dest
ID=aaa13731-1598-3ae2-249b-2e96c0af1212] and the connection currently
has data in the queue.
Therefore, I conclude that it is apparently impossible to push changes
to flows to ANY instance of NiFi, even from a copy of a Flow from that
instance.
I can’t see how this application can be updated from a manual method
let alone an automated method.
David
*From:*Joe Witt <joe.w...@gmail.com>
*Sent:* Tuesday, February 18, 2025 3:04 PM
*To:* users@nifi.apache.org
*Subject:* Re: How do you update various instances/environments based
on a single source of a Flow stored and maintained in Git?
CAUTION: This email originated from outside the organization.
------------------------------------------------------------------------
David,
This link [1] does not work for a slack invite?
A higher level response:
- The intent is you store versioned flows in the registry
- In NiFi instances you instantiate an instance of a versioned flow.
- As you make changes to the flow and bump its version in the registry
you elect/choose when to take the updates in your various nifi instances.
- You do not have to empty out flows - it will take care of most
situations. The caveat being if you make changes that could create
state problems like removing connections which could have data in them
the flow update would fail until you resolve the connections with data
in them.
- The automation to update a given flow version change can/should be
done in a CI/CD tool or process that you control which leverages NiFi
APIs to programmatically do what a user would otherwise do.
- Leverage parameters in your versioned flows and then parameter
contexts unique to each environment so the flow versions don't have to
change but your nifi instances have their own param values.
[1]
https://join.slack.com/t/apachenifi/shared_invite/zt-2ccusmst2-l2KrTzJLrGcHOO0V7~XD4g
Thanks
On Tue, Feb 18, 2025 at 12:47 PM David Steiner
<david.stei...@voyatek.com> wrote:
We are operating with version 1.24.0 of NiFi and Registry.
We have been struggling with how to get a Baseline Flow for a
Product to be 1) Deployable to a Brand new Instance of
NiFi/Registry environment and more so to 2) updated in that
environment.
We have somewhat tackled the deployment, but we have no idea
whether it is sustainable because we can’t figure out how to
update all our existing environments – DEV, QA, UAT, PROD, etc….
We created a Baseline Flow Template and stored that in Azure
DevOps Git.
We then wrote scripts that we take that Template and “deploy” it
into a NiFi/Registry environment – we start with a local Docker
Desktop environment, but these scripts could be used for any
“blank slate” environment that has a NiFi and Registry installed.
This all works fine – we use a set of Context Parameters that all
contain the specific things for each Environment, like server
names, IDs, Passwords, etc… Thus, we establish the Baseline Flow
in NiFi and start Version Control on it which puts the Flow in
Registry.
But we don’t know what to do when we want to update this Template
or Flow.
We have tried several things to figure out how to modify the flow
and then save these changes in our local instances of NiFi and
Registry, which is the place where Development happens, – using
both the “Download flow definition” from NiFi and the “Export
version” from Registry. But neither option allows us to load these
“Flows” into the Registry of another environment and update the
existing Baseline Flows there with the use of Version Control. We
always get errors about there being flowfiles in queues. Message
like this:
Failed to perform update flow request due to
StandardProcessGroup[identifier=d80b3ed3-8e45-3cf4-d1e9-da6b28c97392,name=Baseline
Processing Group] cannot be updated to the proposed flow because
the proposed flow does not contain a match for
Connection[ID=7ad36423-e0ef-3287-d087-87633b60ab1e, Source
ID=7f446e33-2bfc-3e52-8736-910297920464, Dest
ID=aaa13731-1598-3ae2-249b-2e96c0af1212] and the connection
currently has data in the queue.
I have watch a video about how a Flow can be connected to a Git
repository that is set up on the Registry machine, but I’m not
sure how that would enable changes made locally to be applied to
other environments.
Is there some guidance or documentation about how to do updates to
various instances/environments based on a single source of a Flow
stored and maintained in Git?
Is it always the case that if you update the Flow with Version
Control that all queues have to be empty and all of the running
processors are turned off after the update? Does this mean that
the pattern for a Baseline Flow must have “entry” processes
outside of it that will all flowfiles to queue up outside of it so
that updates to the Flow can happen after the queues in the Flow
have been emptied?
If there is a better place to ask this question, what is it? I
couldn’t figure out how to get access to the Slack group.
Thanks,
David
A black and blue text Description automatically generated
*David Steiner*
*Technical Architect*
*RevHub*
*Cell: (513) 594-1737*
