Hello Oleg,

Many thanks for your reply!


I remembered having diff'ed NiFi templates in the past and that was a bit 
messy. However, this was certainly before NiFi 1.0, so from a quick look it 
appears that things are better now in that regard. Thanks for the hint!


However, the situation with moving the controller services associated with a 
processor seems still problematic in my humble opinion. Let's say that in my 
source VM, I have created a Database and an HTTPContext map service at the top 
level of my canvas. These are being used inside many different processors that 
belong to different process groups. Now, if I export my canvas as a template 
and import it on another VM, I don't see these services at the top level of my 
canvas on my target VM. The services are being "compartmentalised" into the 
process groups and sub process groups of my template. Therefore, if I have to 
do a change (say the port of the database is different) I need to go through 
all of them manually as a change in a "compartmentalised" service is not 
reflected on another process group.


An alternative, which I have been using, is to recreate the service at the top 
level of the canvas on the target VM, but this means that I need to go through 
all the processors of the template I moved and associate them with the new top 
level service of my target VM, which is very error-prone in an CI/CD pipeline.


Isn't there an easier way to migrate templates with their associated services?


Many thanks again,

Panos



Ps. writing all this, it came to  mind that I could include all my canvas in 
one process group in my source VM, generate my service inside that and then 
move all of that in my target VM. Not ideal, but should work [😉]


________________________________
From: Oleg Zhurakousky <ozhurakou...@hortonworks.com>
Sent: Friday, January 13, 2017 1:18 PM
To: users@nifi.apache.org
Subject: Re: Migrating NiFi templates/canvas

Panos

What version of NiFi are you using?
The issue with the NiFi templates diffs has actually been addressed (see 
https://issues.apache.org/jira/browse/NIFI-826). So as of NiFi 1.0 you can 
export templates in a deterministic way and then execute diff on them to see 
only what was changed. It was addressed specifically for the SDLC issue you are 
describing. Also, when processor is associated with controller service and you 
created a template that includes such processor, its dependent controller 
service will end up on the template and back in your environment once such 
template is imported.
[NIFI-826] Export templates in a deterministic way - ASF 
JIRA<https://issues.apache.org/jira/browse/NIFI-826>
issues.apache.org
Templates should be exported in a deterministic way so that they can be 
compared or diff'ed with another. Items to consider... The ordering of 
components The id's ...



Anyway, give it a shot and let us know.
Cheers
Oleg

On Jan 13, 2017, at 4:49 AM, Panos Geo 
<panospanos1...@outlook.com<mailto:panospanos1...@outlook.com>> wrote:


Hello all,

We have a Continuous Integration/Continuous Development pipeline that involves 
for each stage (development, testing, customer deployment) a dedicated virtual 
machine running a NiFi instance. As such, for each stage of the pipeline we 
have to migrate the changes of our NiFi canvas from one virtual machine to 
another.

For this we encounter two big problems :

  1.  As far as I know, there is no easy way to diff NiFi canvases (or 
templates) so that we can recognize the changes and don’t have to export and 
import the full canvas each time (as there are configuration changes in the 
target VM that we wouldn’t like to overwrite).
  2.  If we have to export the full canvas, is there a way to reassociate 
processors with the required services on the target virtual machine (e.g. 
DBCPConnectionPool or HTTPContextMap) without having to go through each 
processor explicitly and do the reassignment?


Many thanks in advance,
Panos

Reply via email to