> It seems that the version control information (registry references) shouldn't > be included when creating templates or there should be an option to > include/exclude when creating the template. I would guess that there are a > lot of use cases where a template is being created to be used in environments > other than the one which it was created in.
Ryan, I agree with you on this point. New features (NiFi Registry integration) should not break old features (NiFi flow templates), unless the community explicitly deprecates those features, in which case that should be documented, and, ideally, migration tools and guides should be provided. This case you’ve encountered is definitely one where we could improve the documentation, and possibly even NiFi’s handling of the Registry metadata when exporting or importing templates if it is not a huge effort. Thanks, Kevin From: Ryan H <[email protected]> Reply-To: <[email protected]> Date: Monday, May 7, 2018 at 08:30 To: <[email protected]> Subject: Re: Error Reference When Creating Template From PG Under NiFi Registry Version Control Hi Kevin, Thanks for the multiple options as it relates to FDLC; good articles. The workaround for our current error was to manually search/destroy all references to the registry in the created template xml file (<versionControlInformation> tag in the template). Then when the template is imported into a the new environment, the issue was resolved as all of the bad references were not ported into the new environment. It seems that the version control information (registry references) shouldn't be included when creating templates or there should be an option to include/exclude when creating the template. I would guess that there are a lot of use cases where a template is being created to be used in environments other than the one which it was created in. While it may not fully align with best practices/suggested methods for FDLC, this does cause issues if done this way. It may just be the case that this is documented as being and issue and the recommendation is to follow one of the suggested practices to avoid issues such as this one. Cheers, Ryan On Mon, May 7, 2018 at 7:10 AM, Kevin Doran <[email protected]> wrote: Hi Ryan, I’ve never tried creating a template from a process group that was saved to a NiFi Registry, so I haven’t run into this exact error. However, there are users that cannot connect multiple environments (e.g., dev and production) to the same NiFi Registry and therefore have the need you’ve described of having to move flow definitions without a shared NiFi Registry instance. >From what you’ve described and the error you’ve encountered, I’ve gathered >that the approach you are using to migrate flows across environments is: 1. [Env A] Design / test a flow in NiFi, saving/loading from environment’s NiFi Registry as needed 2. [Env A] Export to template 3. [Env B] Import template into NiFI 4. [Env B] Save to environment’s NiFi Registry Is that more or less correct? If so, instead of this workflow, would it be possible to take the following path to migrate the flow version, which I have seen others use: [Env A] NiFi >> [Env A] NiFi Registry >> [Env B] NiFi Registry >> [Env B] NiFi In this FDLC workflow, templates are never used, because in Env B the Registry is populated first and then imported into NiFi using the import from Registry feature. If this sounds like it could work for your use case, and you would be able to avoid templates entirely (i.e., there is no other reason you need them other than migrating flows across environments), then there are lots of great write ups by others on how to achieve this workflow. The trick is in the Registry A to Registry B step, and for that there are a few approaches and tools that others have documented. A great write up about this “one Registry per environment” FDLC approach is included in a blog post by Pierre Villard, “Automate workflow deployment in Apache NiFi with the NiFi Registry” [1]. As Pierre describes, you can use the NiFi CLI [1], included in the NiFi Toolkit as a separate executable since NiFi 1.6.0 [2] as a tool to migrate flow version snapshots from one NiFi Registry to another. Specifically, the CLI tools “registry export-flow-version” and “registry import-flow-version” commands. An alternative to using the NiFi CLI would be NiPyApi [4], a NiFi client SDK that makes the REST API for NiFi and NiFi Registry easier to work with from Python automation scripts. There’s a good write up on using this to interact with flows in NiFi Registry instances from Timothy Spann [5]. I hope this helps you workaround the issue you are running into. It certainly does not solve the specific error you have documented, but perhaps it is a viable alternative that avoids that exception. Best, Kevin [1] https://pierrevillard.com/2018/04/09/automate-workflow-deployment-in-apache-nifi-with-the-nifi-registry/ [2] https://github.com/apache/nifi/tree/master/nifi-toolkit/nifi-toolkit-cli [3] https://nifi.apache.org/download.html [4] https://github.com/Chaffelson/nipyapi [5] https://community.hortonworks.com/articles/177301/big-data-devops-apache-nifi-flow-versioning-and-au.html From: Ryan H <[email protected]> Reply-To: <[email protected]> Date: Sunday, May 6, 2018 at 21:05 To: <[email protected]> Subject: Error Reference When Creating Template From PG Under NiFi Registry Version Control Hi All, We have found what seems to be an issue when creating a template of a Process Group that is under version control with NiFi Registry. What seems to happen is that there is a hard reference to the Registry Instance from the Env that the template was created from within the template xml file. When importing the template into a new Env the reference to the registry goes with the template and looks to stay with the new process group, is written to the flow.xml.gz, and causes an issue when clustered (the nodes end up not being able to sync after a while because of the error and goes into a disconnected state). I'll say upfront that this somewhat defeats the purpose of having the registry and I'm guessing that there will be suggestions that we should either go the template FLDC route or go the registry FLDC route. For this purpose we cannot share the registry instance between the two environments and may be the case that we should not use the registry at this time. Should this be an error? Is a better way other than going in a searching/destroying all the old registry references out of the template and flow xml files? Should we manually change the registry instances in each of the environments to be the same guid id? As always any help or direction is greatly appreciated. Cheers, Ryan Howell template.xml snippet (created template to download, which will then be uploaded into new env) (seems that the error is automatically created knowing that it is going to be a problem importing): ... <versionControlInformation> <bucketId>1f26b7fd-8987-491d-a2ac-4f87292ab29b</bucketId> <bucketName>Dev-Test</bucketName> <flowDescription></flowDescription> <flowId>c8e54d77-ce35-4be7-aff6-339a96b58e7d</flowId> <flowName>MySuperCoolFlow</flowName> <registryId>4ee39816-0161-1000-ffff-ffffc4085349</registryId> <registryName>4ee39816-0161-1000-ffff-ffffc4085349</registryName> <state>SYNC_FAILURE</state> <stateExplanation>Unable to synchronize Process Group with Flow Registry because Process Group was placed under Version Control using Flow Registry with identifier 4ee39816-0161-1000-ffff-ffffc4085349 but cannot find any Flow Registry with this identifier</stateExplanation> <version>1</version> </versionControlInformation> </processGroups> <processGroups> <id>9d90f6ea-f2a1-3d65-0000-000000000000</id> <parentGroupId>d8cf7c44-48c7-3c7d-0000-000000000000</parentGroupId> <position> <x>422.23612851041094</x> <y>-1527.1966721061617</y> </position> <versionedComponentId>322cb8f0-cec6-30f9-9565-6948120c96d6</versionedComponentId> <comments></comments> <contents> <connections> <id>80c3a35a-27ec-33a2-0000-000000000000</id> <parentGroupId>9d90f6ea-f2a1-3d65-0000-000000000000</parentGroupId> <versionedComponentId>d77ff912-6127-39f2-9c7a-2eb3c7dde817</versionedComponentId> <backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold> <backPressureObjectThreshold>10000</backPressureObjectThreshold> <destination> <groupId>9d90f6ea-f2a1-3d65-0000-000000000000</groupId> <id>14b964e1-0775-3e6f-0000-000000000000</id> <type>PROCESSOR</type> <versionedComponentId>66873d14-b329-3322-b845-d144571ac93e</versionedComponentId> </destination> <flowFileExpiration>0 sec</flowFileExpiration> <labelIndex>1</labelIndex> <name></name> <selectedRelationships>success</selectedRelationships> <source> <groupId>9d90f6ea-f2a1-3d65-0000-000000000000</groupId> <id>f26999d5-adbe-33ea-0000-000000000000</id> <type>PROCESSOR</type> <versionedComponentId>c928e355-f4b0-3673-ba8e-9d7f0cf57d5f</versionedComponentId> </source>
