Thanks a lot, Bryan. PR merged.
From: Ryan H <[email protected]> Reply-To: <[email protected]> Date: Monday, May 7, 2018 at 15:21 To: <[email protected]> Subject: Re: Error Reference When Creating Template From PG Under NiFi Registry Version Control Awesome, thanks for sending out. Great turn around! -Ryan On Mon, May 7, 2018 at 2:58 PM, Bryan Bende <[email protected]> wrote: FYI, created a JIRA and PR here: https://issues.apache.org/jira/browse/NIFI-5163 On Mon, May 7, 2018 at 10:03 AM, Bryan Bende <[email protected]> wrote: > The 1.6 toolkit should work fine with 1.5.0, even the latest CLI from > master should work with 1.5 or 1.6. > > Even though I do agree that we should be moving away from using > templates for deployment, I do think we should be clearing the version > control DTO when creating a template, it was not left in there > intentionally. > > On Mon, May 7, 2018 at 9:04 AM, Kevin Doran <[email protected]> wrote: >>> It seems that the version control information (registry references) >>> shouldn't be included when creating templates or there should be an option >>> to include/exclude when creating the template. I would guess that there are >>> a lot of use cases where a template is being created to be used in >>> environments other than the one which it was created in. >> >> >> >> Ryan, I agree with you on this point. New features (NiFi Registry >> integration) should not break old features (NiFi flow templates), unless the >> community explicitly deprecates those features, in which case that should be >> documented, and, ideally, migration tools and guides should be provided. >> This case you’ve encountered is definitely one where we could improve the >> documentation, and possibly even NiFi’s handling of the Registry metadata >> when exporting or importing templates if it is not a huge effort. >> >> >> >> Thanks, >> >> Kevin >> >> >> >> From: Ryan H <[email protected]> >> Reply-To: <[email protected]> >> Date: Monday, May 7, 2018 at 08:30 >> To: <[email protected]> >> Subject: Re: Error Reference When Creating Template From PG Under NiFi >> Registry Version Control >> >> >> >> Hi Kevin, >> >> Thanks for the multiple options as it relates to FDLC; good articles. The >> workaround for our current error was to manually search/destroy all >> references to the registry in the created template xml file >> (<versionControlInformation> tag in the template). Then when the template is >> imported into a the new environment, the issue was resolved as all of the >> bad references were not ported into the new environment. >> >> It seems that the version control information (registry references) >> shouldn't be included when creating templates or there should be an option >> to include/exclude when creating the template. I would guess that there are >> a lot of use cases where a template is being created to be used in >> environments other than the one which it was created in. While it may not >> fully align with best practices/suggested methods for FDLC, this does cause >> issues if done this way. It may just be the case that this is documented as >> being and issue and the recommendation is to follow one of the suggested >> practices to avoid issues such as this one. >> >> Cheers, >> >> Ryan >> >> >> >> On Mon, May 7, 2018 at 7:10 AM, Kevin Doran <[email protected]> wrote: >> >> Hi Ryan, >> >> >> >> I’ve never tried creating a template from a process group that was saved to >> a NiFi Registry, so I haven’t run into this exact error. However, there are >> users that cannot connect multiple environments (e.g., dev and production) >> to the same NiFi Registry and therefore have the need you’ve described of >> having to move flow definitions without a shared NiFi Registry instance. >> >> >> >> From what you’ve described and the error you’ve encountered, I’ve gathered >> that the approach you are using to migrate flows across environments is: >> >> >> >> 1. [Env A] Design / test a flow in NiFi, saving/loading from environment’s >> NiFi Registry as needed >> >> 2. [Env A] Export to template >> >> 3. [Env B] Import template into NiFI >> >> 4. [Env B] Save to environment’s NiFi Registry >> >> >> >> Is that more or less correct? If so, instead of this workflow, would it be >> possible to take the following path to migrate the flow version, which I >> have seen others use: >> >> >> >> [Env A] NiFi >> [Env A] NiFi Registry >> [Env B] NiFi Registry >> [Env B] >> NiFi >> >> >> >> In this FDLC workflow, templates are never used, because in Env B the >> Registry is populated first and then imported into NiFi using the import >> from Registry feature. >> >> >> >> If this sounds like it could work for your use case, and you would be able >> to avoid templates entirely (i.e., there is no other reason you need them >> other than migrating flows across environments), then there are lots of >> great write ups by others on how to achieve this workflow. The trick is in >> the Registry A to Registry B step, and for that there are a few approaches >> and tools that others have documented. >> >> >> >> A great write up about this “one Registry per environment” FDLC approach is >> included in a blog post by Pierre Villard, “Automate workflow deployment in >> Apache NiFi with the NiFi Registry” [1]. As Pierre describes, you can use >> the NiFi CLI [1], included in the NiFi Toolkit as a separate executable >> since NiFi 1.6.0 [2] as a tool to migrate flow version snapshots from one >> NiFi Registry to another. Specifically, the CLI tools “registry >> export-flow-version” and “registry import-flow-version” commands. >> >> >> >> An alternative to using the NiFi CLI would be NiPyApi [4], a NiFi client SDK >> that makes the REST API for NiFi and NiFi Registry easier to work with from >> Python automation scripts. There’s a good write up on using this to interact >> with flows in NiFi Registry instances from Timothy Spann [5]. >> >> >> >> I hope this helps you workaround the issue you are running into. It >> certainly does not solve the specific error you have documented, but perhaps >> it is a viable alternative that avoids that exception. >> >> >> >> Best, >> Kevin >> >> >> >> [1] >> https://pierrevillard.com/2018/04/09/automate-workflow-deployment-in-apache-nifi-with-the-nifi-registry/ >> >> [2] https://github.com/apache/nifi/tree/master/nifi-toolkit/nifi-toolkit-cli >> >> [3] https://nifi.apache.org/download.html >> >> [4] https://github.com/Chaffelson/nipyapi >> >> [5] >> https://community.hortonworks.com/articles/177301/big-data-devops-apache-nifi-flow-versioning-and-au.html >> >> >> >> >> >> From: Ryan H <[email protected]> >> Reply-To: <[email protected]> >> Date: Sunday, May 6, 2018 at 21:05 >> To: <[email protected]> >> Subject: Error Reference When Creating Template From PG Under NiFi Registry >> Version Control >> >> >> >> Hi All, >> >> We have found what seems to be an issue when creating a template of a >> Process Group that is under version control with NiFi Registry. What seems >> to happen is that there is a hard reference to the Registry Instance from >> the Env that the template was created from within the template xml file. >> When importing the template into a new Env the reference to the registry >> goes with the template and looks to stay with the new process group, is >> written to the flow.xml.gz, and causes an issue when clustered (the nodes >> end up not being able to sync after a while because of the error and goes >> into a disconnected state). >> >> I'll say upfront that this somewhat defeats the purpose of having the >> registry and I'm guessing that there will be suggestions that we should >> either go the template FLDC route or go the registry FLDC route. For this >> purpose we cannot share the registry instance between the two environments >> and may be the case that we should not use the registry at this time. >> >> Should this be an error? Is a better way other than going in a >> searching/destroying all the old registry references out of the template and >> flow xml files? Should we manually change the registry instances in each of >> the environments to be the same guid id? >> >> >> >> As always any help or direction is greatly appreciated. >> >> Cheers, >> >> Ryan Howell >> >> template.xml snippet (created template to download, which will then be >> uploaded into new env) (seems that the error is automatically created >> knowing that it is going to be a problem importing): >> ... >> <versionControlInformation> >> >> <bucketId>1f26b7fd-8987-491d-a2ac-4f87292ab29b</bucketId> >> <bucketName>Dev-Test</bucketName> >> <flowDescription></flowDescription> >> >> <flowId>c8e54d77-ce35-4be7-aff6-339a96b58e7d</flowId> >> <flowName>MySuperCoolFlow</flowName> >> >> <registryId>4ee39816-0161-1000-ffff-ffffc4085349</registryId> >> >> <registryName>4ee39816-0161-1000-ffff-ffffc4085349</registryName> >> <state>SYNC_FAILURE</state> >> <stateExplanation>Unable to synchronize Process >> Group with Flow Registry because Process Group was placed under Version >> Control using Flow Registry with identifier >> 4ee39816-0161-1000-ffff-ffffc4085349 but cannot find any Flow Registry with >> this identifier</stateExplanation> >> <version>1</version> >> </versionControlInformation> >> </processGroups> >> <processGroups> >> <id>9d90f6ea-f2a1-3d65-0000-000000000000</id> >> >> <parentGroupId>d8cf7c44-48c7-3c7d-0000-000000000000</parentGroupId> >> <position> >> <x>422.23612851041094</x> >> <y>-1527.1966721061617</y> >> </position> >> >> <versionedComponentId>322cb8f0-cec6-30f9-9565-6948120c96d6</versionedComponentId> >> <comments></comments> >> <contents> >> <connections> >> <id>80c3a35a-27ec-33a2-0000-000000000000</id> >> >> <parentGroupId>9d90f6ea-f2a1-3d65-0000-000000000000</parentGroupId> >> >> <versionedComponentId>d77ff912-6127-39f2-9c7a-2eb3c7dde817</versionedComponentId> >> <backPressureDataSizeThreshold>1 >> GB</backPressureDataSizeThreshold> >> >> <backPressureObjectThreshold>10000</backPressureObjectThreshold> >> <destination> >> <groupId>9d90f6ea-f2a1-3d65-0000-000000000000</groupId> >> <id>14b964e1-0775-3e6f-0000-000000000000</id> >> <type>PROCESSOR</type> >> <versionedComponentId>66873d14-b329-3322-b845-d144571ac93e</versionedComponentId> >> </destination> >> <flowFileExpiration>0 sec</flowFileExpiration> >> <labelIndex>1</labelIndex> >> <name></name> >> >> <selectedRelationships>success</selectedRelationships> >> <source> >> <groupId>9d90f6ea-f2a1-3d65-0000-000000000000</groupId> >> <id>f26999d5-adbe-33ea-0000-000000000000</id> >> <type>PROCESSOR</type> >> <versionedComponentId>c928e355-f4b0-3673-ba8e-9d7f0cf57d5f</versionedComponentId> >> </source> >> >>
