Hello Matt,
NIFI-3173 is exactly what I am referring to. We are currently using 1.0.0 but will upgrade to 1.1.1 to get the fix [😊] Many thanks, Panos ________________________________ From: Matt Gilman <[email protected]> Sent: Friday, January 13, 2017 8:16 PM To: [email protected] Subject: Re: Migrating NiFi templates/canvas Panos, I'm not sure what version of NiFi you're using but in 1.1.1 another issue [1] was addressed that may be what you're seeing now. If you're still on 1.0.0 or 1.1.0 maybe try upgrading to 1.1.1. Matt [1] https://issues.apache.org/jira/browse/NIFI-3173 On Fri, Jan 13, 2017 at 1:51 PM, Panos Geo <[email protected]<mailto:[email protected]>> wrote: Hello Matt, Thanks for your reply. I fully get that, but my requirement is in addition to this. The (top) controller service is being referenced by multiple processors in my template, which however belong in different sub process groups. When I import this template in another NiFi instance this controller service is present, but it is being "split" to different process group controller services, which I cannot configure centrally (as I would have done if that was a top level controller service as it was in the original NiFi instance before exporting the template). Many thanks, Panos ________________________________ From: Matt Gilman <[email protected]<mailto:[email protected]>> Sent: Friday, January 13, 2017 6:27 PM To: [email protected]<mailto:[email protected]> Subject: Re: Migrating NiFi templates/canvas Panos, The current behavior is that Controller Services are only included if they are referenced by a component in the template. While this was initially by designed, we have received comments about it from folks looking to export their entire configuration. There is an outstanding JIRA [1] to update this behavior. Matt [1] https://issues.apache.org/jira/browse/NIFI-2895 On Fri, Jan 13, 2017 at 12:01 PM, Panos Geo <[email protected]<mailto:[email protected]>> wrote: Having just checked the idea in my PS, it doesn't seem to work as the transferred controller services are at a process group level and hence are "compartmentalised" after the migration... Any thoughts on this would be highly appreciated. Many thanks, Panos ________________________________ From: Panos Geo <[email protected]<mailto:[email protected]>> Sent: Friday, January 13, 2017 4:47 PM To: [email protected]<mailto:[email protected]> Subject: Re: Migrating NiFi templates/canvas Hello Oleg, Many thanks for your reply! I remembered having diff'ed NiFi templates in the past and that was a bit messy. However, this was certainly before NiFi 1.0, so from a quick look it appears that things are better now in that regard. Thanks for the hint! However, the situation with moving the controller services associated with a processor seems still problematic in my humble opinion. Let's say that in my source VM, I have created a Database and an HTTPContext map service at the top level of my canvas. These are being used inside many different processors that belong to different process groups. Now, if I export my canvas as a template and import it on another VM, I don't see these services at the top level of my canvas on my target VM. The services are being "compartmentalised" into the process groups and sub process groups of my template. Therefore, if I have to do a change (say the port of the database is different) I need to go through all of them manually as a change in a "compartmentalised" service is not reflected on another process group. An alternative, which I have been using, is to recreate the service at the top level of the canvas on the target VM, but this means that I need to go through all the processors of the template I moved and associate them with the new top level service of my target VM, which is very error-prone in an CI/CD pipeline. Isn't there an easier way to migrate templates with their associated services? Many thanks again, Panos Ps. writing all this, it came to mind that I could include all my canvas in one process group in my source VM, generate my service inside that and then move all of that in my target VM. Not ideal, but should work [😉] ________________________________ From: Oleg Zhurakousky <[email protected]<mailto:[email protected]>> Sent: Friday, January 13, 2017 1:18 PM To: [email protected]<mailto:[email protected]> Subject: Re: Migrating NiFi templates/canvas Panos What version of NiFi are you using? The issue with the NiFi templates diffs has actually been addressed (see https://issues.apache.org/jira/browse/NIFI-826). So as of NiFi 1.0 you can export templates in a deterministic way and then execute diff on them to see only what was changed. It was addressed specifically for the SDLC issue you are describing. Also, when processor is associated with controller service and you created a template that includes such processor, its dependent controller service will end up on the template and back in your environment once such template is imported. [NIFI-826] Export templates in a deterministic way - ASF JIRA<https://issues.apache.org/jira/browse/NIFI-826> issues.apache.org<http://issues.apache.org> Templates should be exported in a deterministic way so that they can be compared or diff'ed with another. Items to consider... The ordering of components The id's ... Anyway, give it a shot and let us know. Cheers Oleg On Jan 13, 2017, at 4:49 AM, Panos Geo <[email protected]<mailto:[email protected]>> wrote: Hello all, We have a Continuous Integration/Continuous Development pipeline that involves for each stage (development, testing, customer deployment) a dedicated virtual machine running a NiFi instance. As such, for each stage of the pipeline we have to migrate the changes of our NiFi canvas from one virtual machine to another. For this we encounter two big problems : 1. As far as I know, there is no easy way to diff NiFi canvases (or templates) so that we can recognize the changes and don’t have to export and import the full canvas each time (as there are configuration changes in the target VM that we wouldn’t like to overwrite). 2. If we have to export the full canvas, is there a way to reassociate processors with the required services on the target virtual machine (e.g. DBCPConnectionPool or HTTPContextMap) without having to go through each processor explicitly and do the reassignment? Many thanks in advance, Panos
