The 1.6 toolkit should work fine with 1.5.0, even the latest CLI from master should work with 1.5 or 1.6.
Even though I do agree that we should be moving away from using templates for deployment, I do think we should be clearing the version control DTO when creating a template, it was not left in there intentionally. On Mon, May 7, 2018 at 9:04 AM, Kevin Doran <[email protected]> wrote: >> It seems that the version control information (registry references) >> shouldn't be included when creating templates or there should be an option >> to include/exclude when creating the template. I would guess that there are >> a lot of use cases where a template is being created to be used in >> environments other than the one which it was created in. > > > > Ryan, I agree with you on this point. New features (NiFi Registry > integration) should not break old features (NiFi flow templates), unless the > community explicitly deprecates those features, in which case that should be > documented, and, ideally, migration tools and guides should be provided. > This case you’ve encountered is definitely one where we could improve the > documentation, and possibly even NiFi’s handling of the Registry metadata > when exporting or importing templates if it is not a huge effort. > > > > Thanks, > > Kevin > > > > From: Ryan H <[email protected]> > Reply-To: <[email protected]> > Date: Monday, May 7, 2018 at 08:30 > To: <[email protected]> > Subject: Re: Error Reference When Creating Template From PG Under NiFi > Registry Version Control > > > > Hi Kevin, > > Thanks for the multiple options as it relates to FDLC; good articles. The > workaround for our current error was to manually search/destroy all > references to the registry in the created template xml file > (<versionControlInformation> tag in the template). Then when the template is > imported into a the new environment, the issue was resolved as all of the > bad references were not ported into the new environment. > > It seems that the version control information (registry references) > shouldn't be included when creating templates or there should be an option > to include/exclude when creating the template. I would guess that there are > a lot of use cases where a template is being created to be used in > environments other than the one which it was created in. While it may not > fully align with best practices/suggested methods for FDLC, this does cause > issues if done this way. It may just be the case that this is documented as > being and issue and the recommendation is to follow one of the suggested > practices to avoid issues such as this one. > > Cheers, > > Ryan > > > > On Mon, May 7, 2018 at 7:10 AM, Kevin Doran <[email protected]> wrote: > > Hi Ryan, > > > > I’ve never tried creating a template from a process group that was saved to > a NiFi Registry, so I haven’t run into this exact error. However, there are > users that cannot connect multiple environments (e.g., dev and production) > to the same NiFi Registry and therefore have the need you’ve described of > having to move flow definitions without a shared NiFi Registry instance. > > > > From what you’ve described and the error you’ve encountered, I’ve gathered > that the approach you are using to migrate flows across environments is: > > > > 1. [Env A] Design / test a flow in NiFi, saving/loading from environment’s > NiFi Registry as needed > > 2. [Env A] Export to template > > 3. [Env B] Import template into NiFI > > 4. [Env B] Save to environment’s NiFi Registry > > > > Is that more or less correct? If so, instead of this workflow, would it be > possible to take the following path to migrate the flow version, which I > have seen others use: > > > > [Env A] NiFi >> [Env A] NiFi Registry >> [Env B] NiFi Registry >> [Env B] > NiFi > > > > In this FDLC workflow, templates are never used, because in Env B the > Registry is populated first and then imported into NiFi using the import > from Registry feature. > > > > If this sounds like it could work for your use case, and you would be able > to avoid templates entirely (i.e., there is no other reason you need them > other than migrating flows across environments), then there are lots of > great write ups by others on how to achieve this workflow. The trick is in > the Registry A to Registry B step, and for that there are a few approaches > and tools that others have documented. > > > > A great write up about this “one Registry per environment” FDLC approach is > included in a blog post by Pierre Villard, “Automate workflow deployment in > Apache NiFi with the NiFi Registry” [1]. As Pierre describes, you can use > the NiFi CLI [1], included in the NiFi Toolkit as a separate executable > since NiFi 1.6.0 [2] as a tool to migrate flow version snapshots from one > NiFi Registry to another. Specifically, the CLI tools “registry > export-flow-version” and “registry import-flow-version” commands. > > > > An alternative to using the NiFi CLI would be NiPyApi [4], a NiFi client SDK > that makes the REST API for NiFi and NiFi Registry easier to work with from > Python automation scripts. There’s a good write up on using this to interact > with flows in NiFi Registry instances from Timothy Spann [5]. > > > > I hope this helps you workaround the issue you are running into. It > certainly does not solve the specific error you have documented, but perhaps > it is a viable alternative that avoids that exception. > > > > Best, > Kevin > > > > [1] > https://pierrevillard.com/2018/04/09/automate-workflow-deployment-in-apache-nifi-with-the-nifi-registry/ > > [2] https://github.com/apache/nifi/tree/master/nifi-toolkit/nifi-toolkit-cli > > [3] https://nifi.apache.org/download.html > > [4] https://github.com/Chaffelson/nipyapi > > [5] > https://community.hortonworks.com/articles/177301/big-data-devops-apache-nifi-flow-versioning-and-au.html > > > > > > From: Ryan H <[email protected]> > Reply-To: <[email protected]> > Date: Sunday, May 6, 2018 at 21:05 > To: <[email protected]> > Subject: Error Reference When Creating Template From PG Under NiFi Registry > Version Control > > > > Hi All, > > We have found what seems to be an issue when creating a template of a > Process Group that is under version control with NiFi Registry. What seems > to happen is that there is a hard reference to the Registry Instance from > the Env that the template was created from within the template xml file. > When importing the template into a new Env the reference to the registry > goes with the template and looks to stay with the new process group, is > written to the flow.xml.gz, and causes an issue when clustered (the nodes > end up not being able to sync after a while because of the error and goes > into a disconnected state). > > I'll say upfront that this somewhat defeats the purpose of having the > registry and I'm guessing that there will be suggestions that we should > either go the template FLDC route or go the registry FLDC route. For this > purpose we cannot share the registry instance between the two environments > and may be the case that we should not use the registry at this time. > > Should this be an error? Is a better way other than going in a > searching/destroying all the old registry references out of the template and > flow xml files? Should we manually change the registry instances in each of > the environments to be the same guid id? > > > > As always any help or direction is greatly appreciated. > > Cheers, > > Ryan Howell > > template.xml snippet (created template to download, which will then be > uploaded into new env) (seems that the error is automatically created > knowing that it is going to be a problem importing): > ... > <versionControlInformation> > > <bucketId>1f26b7fd-8987-491d-a2ac-4f87292ab29b</bucketId> > <bucketName>Dev-Test</bucketName> > <flowDescription></flowDescription> > > <flowId>c8e54d77-ce35-4be7-aff6-339a96b58e7d</flowId> > <flowName>MySuperCoolFlow</flowName> > > <registryId>4ee39816-0161-1000-ffff-ffffc4085349</registryId> > > <registryName>4ee39816-0161-1000-ffff-ffffc4085349</registryName> > <state>SYNC_FAILURE</state> > <stateExplanation>Unable to synchronize Process > Group with Flow Registry because Process Group was placed under Version > Control using Flow Registry with identifier > 4ee39816-0161-1000-ffff-ffffc4085349 but cannot find any Flow Registry with > this identifier</stateExplanation> > <version>1</version> > </versionControlInformation> > </processGroups> > <processGroups> > <id>9d90f6ea-f2a1-3d65-0000-000000000000</id> > > <parentGroupId>d8cf7c44-48c7-3c7d-0000-000000000000</parentGroupId> > <position> > <x>422.23612851041094</x> > <y>-1527.1966721061617</y> > </position> > > <versionedComponentId>322cb8f0-cec6-30f9-9565-6948120c96d6</versionedComponentId> > <comments></comments> > <contents> > <connections> > <id>80c3a35a-27ec-33a2-0000-000000000000</id> > > <parentGroupId>9d90f6ea-f2a1-3d65-0000-000000000000</parentGroupId> > > <versionedComponentId>d77ff912-6127-39f2-9c7a-2eb3c7dde817</versionedComponentId> > <backPressureDataSizeThreshold>1 > GB</backPressureDataSizeThreshold> > > <backPressureObjectThreshold>10000</backPressureObjectThreshold> > <destination> > <groupId>9d90f6ea-f2a1-3d65-0000-000000000000</groupId> > <id>14b964e1-0775-3e6f-0000-000000000000</id> > <type>PROCESSOR</type> > <versionedComponentId>66873d14-b329-3322-b845-d144571ac93e</versionedComponentId> > </destination> > <flowFileExpiration>0 sec</flowFileExpiration> > <labelIndex>1</labelIndex> > <name></name> > > <selectedRelationships>success</selectedRelationships> > <source> > <groupId>9d90f6ea-f2a1-3d65-0000-000000000000</groupId> > <id>f26999d5-adbe-33ea-0000-000000000000</id> > <type>PROCESSOR</type> > <versionedComponentId>c928e355-f4b0-3673-ba8e-9d7f0cf57d5f</versionedComponentId> > </source> > >
