Awesome, thanks for sending out. Great turn around! -Ryan
On Mon, May 7, 2018 at 2:58 PM, Bryan Bende <[email protected]> wrote: > FYI, created a JIRA and PR here: > > https://issues.apache.org/jira/browse/NIFI-5163 > > On Mon, May 7, 2018 at 10:03 AM, Bryan Bende <[email protected]> wrote: > > The 1.6 toolkit should work fine with 1.5.0, even the latest CLI from > > master should work with 1.5 or 1.6. > > > > Even though I do agree that we should be moving away from using > > templates for deployment, I do think we should be clearing the version > > control DTO when creating a template, it was not left in there > > intentionally. > > > > On Mon, May 7, 2018 at 9:04 AM, Kevin Doran <[email protected]> wrote: > >>> It seems that the version control information (registry references) > >>> shouldn't be included when creating templates or there should be an > option > >>> to include/exclude when creating the template. I would guess that > there are > >>> a lot of use cases where a template is being created to be used in > >>> environments other than the one which it was created in. > >> > >> > >> > >> Ryan, I agree with you on this point. New features (NiFi Registry > >> integration) should not break old features (NiFi flow templates), > unless the > >> community explicitly deprecates those features, in which case that > should be > >> documented, and, ideally, migration tools and guides should be provided. > >> This case you’ve encountered is definitely one where we could improve > the > >> documentation, and possibly even NiFi’s handling of the Registry > metadata > >> when exporting or importing templates if it is not a huge effort. > >> > >> > >> > >> Thanks, > >> > >> Kevin > >> > >> > >> > >> From: Ryan H <[email protected]> > >> Reply-To: <[email protected]> > >> Date: Monday, May 7, 2018 at 08:30 > >> To: <[email protected]> > >> Subject: Re: Error Reference When Creating Template From PG Under NiFi > >> Registry Version Control > >> > >> > >> > >> Hi Kevin, > >> > >> Thanks for the multiple options as it relates to FDLC; good articles. > The > >> workaround for our current error was to manually search/destroy all > >> references to the registry in the created template xml file > >> (<versionControlInformation> tag in the template). Then when the > template is > >> imported into a the new environment, the issue was resolved as all of > the > >> bad references were not ported into the new environment. > >> > >> It seems that the version control information (registry references) > >> shouldn't be included when creating templates or there should be an > option > >> to include/exclude when creating the template. I would guess that there > are > >> a lot of use cases where a template is being created to be used in > >> environments other than the one which it was created in. While it may > not > >> fully align with best practices/suggested methods for FDLC, this does > cause > >> issues if done this way. It may just be the case that this is > documented as > >> being and issue and the recommendation is to follow one of the suggested > >> practices to avoid issues such as this one. > >> > >> Cheers, > >> > >> Ryan > >> > >> > >> > >> On Mon, May 7, 2018 at 7:10 AM, Kevin Doran <[email protected]> wrote: > >> > >> Hi Ryan, > >> > >> > >> > >> I’ve never tried creating a template from a process group that was > saved to > >> a NiFi Registry, so I haven’t run into this exact error. However, there > are > >> users that cannot connect multiple environments (e.g., dev and > production) > >> to the same NiFi Registry and therefore have the need you’ve described > of > >> having to move flow definitions without a shared NiFi Registry instance. > >> > >> > >> > >> From what you’ve described and the error you’ve encountered, I’ve > gathered > >> that the approach you are using to migrate flows across environments is: > >> > >> > >> > >> 1. [Env A] Design / test a flow in NiFi, saving/loading from > environment’s > >> NiFi Registry as needed > >> > >> 2. [Env A] Export to template > >> > >> 3. [Env B] Import template into NiFI > >> > >> 4. [Env B] Save to environment’s NiFi Registry > >> > >> > >> > >> Is that more or less correct? If so, instead of this workflow, would it > be > >> possible to take the following path to migrate the flow version, which I > >> have seen others use: > >> > >> > >> > >> [Env A] NiFi >> [Env A] NiFi Registry >> [Env B] NiFi Registry >> [Env > B] > >> NiFi > >> > >> > >> > >> In this FDLC workflow, templates are never used, because in Env B the > >> Registry is populated first and then imported into NiFi using the import > >> from Registry feature. > >> > >> > >> > >> If this sounds like it could work for your use case, and you would be > able > >> to avoid templates entirely (i.e., there is no other reason you need > them > >> other than migrating flows across environments), then there are lots of > >> great write ups by others on how to achieve this workflow. The trick is > in > >> the Registry A to Registry B step, and for that there are a few > approaches > >> and tools that others have documented. > >> > >> > >> > >> A great write up about this “one Registry per environment” FDLC > approach is > >> included in a blog post by Pierre Villard, “Automate workflow > deployment in > >> Apache NiFi with the NiFi Registry” [1]. As Pierre describes, you can > use > >> the NiFi CLI [1], included in the NiFi Toolkit as a separate executable > >> since NiFi 1.6.0 [2] as a tool to migrate flow version snapshots from > one > >> NiFi Registry to another. Specifically, the CLI tools “registry > >> export-flow-version” and “registry import-flow-version” commands. > >> > >> > >> > >> An alternative to using the NiFi CLI would be NiPyApi [4], a NiFi > client SDK > >> that makes the REST API for NiFi and NiFi Registry easier to work with > from > >> Python automation scripts. There’s a good write up on using this to > interact > >> with flows in NiFi Registry instances from Timothy Spann [5]. > >> > >> > >> > >> I hope this helps you workaround the issue you are running into. It > >> certainly does not solve the specific error you have documented, but > perhaps > >> it is a viable alternative that avoids that exception. > >> > >> > >> > >> Best, > >> Kevin > >> > >> > >> > >> [1] > >> https://pierrevillard.com/2018/04/09/automate-workflow- > deployment-in-apache-nifi-with-the-nifi-registry/ > >> > >> [2] https://github.com/apache/nifi/tree/master/nifi-toolkit/ > nifi-toolkit-cli > >> > >> [3] https://nifi.apache.org/download.html > >> > >> [4] https://github.com/Chaffelson/nipyapi > >> > >> [5] > >> https://community.hortonworks.com/articles/177301/big-data- > devops-apache-nifi-flow-versioning-and-au.html > >> > >> > >> > >> > >> > >> From: Ryan H <[email protected]> > >> Reply-To: <[email protected]> > >> Date: Sunday, May 6, 2018 at 21:05 > >> To: <[email protected]> > >> Subject: Error Reference When Creating Template From PG Under NiFi > Registry > >> Version Control > >> > >> > >> > >> Hi All, > >> > >> We have found what seems to be an issue when creating a template of a > >> Process Group that is under version control with NiFi Registry. What > seems > >> to happen is that there is a hard reference to the Registry Instance > from > >> the Env that the template was created from within the template xml file. > >> When importing the template into a new Env the reference to the registry > >> goes with the template and looks to stay with the new process group, is > >> written to the flow.xml.gz, and causes an issue when clustered (the > nodes > >> end up not being able to sync after a while because of the error and > goes > >> into a disconnected state). > >> > >> I'll say upfront that this somewhat defeats the purpose of having the > >> registry and I'm guessing that there will be suggestions that we should > >> either go the template FLDC route or go the registry FLDC route. For > this > >> purpose we cannot share the registry instance between the two > environments > >> and may be the case that we should not use the registry at this time. > >> > >> Should this be an error? Is a better way other than going in a > >> searching/destroying all the old registry references out of the > template and > >> flow xml files? Should we manually change the registry instances in > each of > >> the environments to be the same guid id? > >> > >> > >> > >> As always any help or direction is greatly appreciated. > >> > >> Cheers, > >> > >> Ryan Howell > >> > >> template.xml snippet (created template to download, which will then be > >> uploaded into new env) (seems that the error is automatically created > >> knowing that it is going to be a problem importing): > >> ... > >> <versionControlInformation> > >> > >> <bucketId>1f26b7fd-8987-491d-a2ac-4f87292ab29b</bucketId> > >> <bucketName>Dev-Test</bucketName> > >> <flowDescription></flowDescription> > >> > >> <flowId>c8e54d77-ce35-4be7-aff6-339a96b58e7d</flowId> > >> <flowName>MySuperCoolFlow</flowName> > >> > >> <registryId>4ee39816-0161-1000-ffff-ffffc4085349</registryId> > >> > >> <registryName>4ee39816-0161-1000-ffff-ffffc4085349</registryName> > >> <state>SYNC_FAILURE</state> > >> <stateExplanation>Unable to synchronize Process > >> Group with Flow Registry because Process Group was placed under Version > >> Control using Flow Registry with identifier > >> 4ee39816-0161-1000-ffff-ffffc4085349 but cannot find any Flow Registry > with > >> this identifier</stateExplanation> > >> <version>1</version> > >> </versionControlInformation> > >> </processGroups> > >> <processGroups> > >> <id>9d90f6ea-f2a1-3d65-0000-000000000000</id> > >> > >> <parentGroupId>d8cf7c44-48c7-3c7d-0000-000000000000</parentGroupId> > >> <position> > >> <x>422.23612851041094</x> > >> <y>-1527.1966721061617</y> > >> </position> > >> > >> <versionedComponentId>322cb8f0-cec6-30f9-9565-6948120c96d6</ > versionedComponentId> > >> <comments></comments> > >> <contents> > >> <connections> > >> <id>80c3a35a-27ec-33a2-0000- > 000000000000</id> > >> > >> <parentGroupId>9d90f6ea-f2a1-3d65-0000-000000000000</parentGroupId> > >> > >> <versionedComponentId>d77ff912-6127-39f2-9c7a-2eb3c7dde817</ > versionedComponentId> > >> <backPressureDataSizeThreshold>1 > >> GB</backPressureDataSizeThreshold> > >> > >> <backPressureObjectThreshold>10000</backPressureObjectThreshold> > >> <destination> > >> <groupId>9d90f6ea-f2a1-3d65-0000-000000000000</groupId> > >> <id>14b964e1-0775-3e6f-0000-000000000000</id> > >> <type>PROCESSOR</type> > >> <versionedComponentId>66873d14-b329-3322-b845-d144571ac93e</ > versionedComponentId> > >> </destination> > >> <flowFileExpiration>0 > sec</flowFileExpiration> > >> <labelIndex>1</labelIndex> > >> <name></name> > >> > >> <selectedRelationships>success</selectedRelationships> > >> <source> > >> <groupId>9d90f6ea-f2a1-3d65-0000-000000000000</groupId> > >> <id>f26999d5-adbe-33ea-0000-000000000000</id> > >> <type>PROCESSOR</type> > >> <versionedComponentId>c928e355-f4b0-3673-ba8e-9d7f0cf57d5f</ > versionedComponentId> > >> </source> > >> > >> >
