Hi Stéphane,

This is a request that has grown popular recently. NiFi was not initially 
designed with environment promotion in mind, so it is something we are 
currently investigating and trying to address.

The development/QA/production environment promotion process [1] (sometimes 
referred to as "SDLC" or "D2P" in conversation) is a topic of much discussion 
amongst the NiFi development community. Currently, there are plans to improve 
this process in a future release. For now, I will discuss some common 
behaviors/workflows that we have seen.

* The $NIFI_HOME/conf/flow.xml.gz file contains the entire flow serialized to 
XML. This file contains all processor configuration values, even sensitive 
values (encrypted). With the new Variable Registry [2] effort, you can refer to 
environment-specific variables transparently, and promote the same flow between 
environments without having to update specific values in the flow itself.
* The XML flow definition or specific templates can be committed and versioned 
using Git (or any other source code control tool). Recent improvements like 
"deterministic template diffs” [3] have made this versioning easier.
* The NiFi REST API [4] can be used to "automate" the deployment of a template 
or flow to an instance of NiFi.
* A script (Groovy, Python, etc.) could be used to integrate with both your 
source code control tool and your various NiFi instances to semi-automate this 
process (i.e. tap into Git hooks detecting a commit, and promote automatically 
to the next environment), but you probably want some human interaction to 
remain for verification at each state.
* To ensure that no “data is left hanging”, you can stop your input sources and 
allow the system to “drain” (i.e. check that all queues are empty before 
stopping the application, if replacing the flow.xml.gz file). If adding a new 
template, you can add this during normal operation and then configure and start 
the new flow while the old continues to run.

We understand that the current state of NiFi is not ideal for the promotion of 
the flow between dev/QA/prod environments. There are ongoing efforts to improve 
this, but I can't describe anything concrete at this time. If these points 
raise specific questions or you think of something else, please follow up.

[2] https://cwiki.apache.org/confluence/display/NIFI/Variable+Registry 
[3] https://issues.apache.org/jira/browse/NIFI-826 
[4] https://nifi.apache.org/docs/nifi-docs/rest-api/index.html

Andy LoPresto
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Oct 11, 2016, at 7:37 PM, Stéphane Maarek <stephane.maa...@gmail.com> 
> wrote:
> Hi,
> How should we proceed to promote flow from dev to test to prod?
> Basically, our stakeholders are not happy with having people modifying 
> anything in production (so it's a read-only environment), and basically want 
> people to draft / edit their flow in dev, push them to test, and then to prod
> I think this has a couple of challenges, such as :
> 0) How do you capture all the changes that were made? Is it a template, or a 
> list of API calls, or something else?
> 1) How do you programatically promote stuff from dev to test to prod? Stuff 
> that change between these environments are probably hostnames, etc
> 2) How do you edit a flow that's already been promoted to prod?
> 3) How to ensure that no data is left "hanging" in between processors before 
> a change to the flow? Or is it fine?
> Interested in knowing how other companies do it, or manage their NiFi cluster 
> in production (what can be changed vs what can't)
> Cheers,
> Stephane

Attachment: signature.asc
Description: Message signed with OpenPGP using GPGMail

Reply via email to