Joe, No more to it other than for us to use the tools. We are on NiFi 1.5 right now. Can we use the 1.6 toolkit with a 1.5 NiFi Cluster? If not it sounds like we would have to version bump.
-Ryan On Mon, May 7, 2018 at 8:54 AM, Joe Witt <[email protected]> wrote: > Andrew, > > I do agree with you point. I think Ryan's saying we need to do a > better job of calling that out in the docs/guiding folks to the best > path so they don't try combining older practices with ones that are > meant to totally replace it. > > Is there more to it Ryan? > > Thanks > > On Mon, May 7, 2018 at 8:50 AM, Andrew Grande <[email protected]> wrote: > > The NiFi CLI has a dedicated command to move a flow version snapshot > between > > registry instances. > > > > What I find out of line in Ryan's description is they are using templates > > and registry for the version control. Templates were created with no > > registry in sight, and, frankly, should not be used anymore now. > > > > If registries are setup, use the tools meant for the registry (and come > for > > any help here). > > > > Cheers! > > Andrew > > > > > > On Mon, May 7, 2018, 8:30 AM Ryan H <[email protected]> > > wrote: > >> > >> Hi Kevin, > >> > >> Thanks for the multiple options as it relates to FDLC; good articles. > The > >> workaround for our current error was to manually search/destroy all > >> references to the registry in the created template xml file > >> (<versionControlInformation> tag in the template). Then when the > template is > >> imported into a the new environment, the issue was resolved as all of > the > >> bad references were not ported into the new environment. > >> > >> It seems that the version control information (registry references) > >> shouldn't be included when creating templates or there should be an > option > >> to include/exclude when creating the template. I would guess that there > are > >> a lot of use cases where a template is being created to be used in > >> environments other than the one which it was created in. While it may > not > >> fully align with best practices/suggested methods for FDLC, this does > cause > >> issues if done this way. It may just be the case that this is > documented as > >> being and issue and the recommendation is to follow one of the suggested > >> practices to avoid issues such as this one. > >> > >> Cheers, > >> > >> Ryan > >> > >> On Mon, May 7, 2018 at 7:10 AM, Kevin Doran <[email protected]> wrote: > >>> > >>> Hi Ryan, > >>> > >>> > >>> > >>> I’ve never tried creating a template from a process group that was > saved > >>> to a NiFi Registry, so I haven’t run into this exact error. However, > there > >>> are users that cannot connect multiple environments (e.g., dev and > >>> production) to the same NiFi Registry and therefore have the need > you’ve > >>> described of having to move flow definitions without a shared NiFi > Registry > >>> instance. > >>> > >>> > >>> > >>> From what you’ve described and the error you’ve encountered, I’ve > >>> gathered that the approach you are using to migrate flows across > >>> environments is: > >>> > >>> > >>> > >>> 1. [Env A] Design / test a flow in NiFi, saving/loading from > >>> environment’s NiFi Registry as needed > >>> > >>> 2. [Env A] Export to template > >>> > >>> 3. [Env B] Import template into NiFI > >>> > >>> 4. [Env B] Save to environment’s NiFi Registry > >>> > >>> > >>> > >>> Is that more or less correct? If so, instead of this workflow, would it > >>> be possible to take the following path to migrate the flow version, > which I > >>> have seen others use: > >>> > >>> > >>> > >>> [Env A] NiFi >> [Env A] NiFi Registry >> [Env B] NiFi Registry >> [Env > >>> B] NiFi > >>> > >>> > >>> > >>> In this FDLC workflow, templates are never used, because in Env B the > >>> Registry is populated first and then imported into NiFi using the > import > >>> from Registry feature. > >>> > >>> > >>> > >>> If this sounds like it could work for your use case, and you would be > >>> able to avoid templates entirely (i.e., there is no other reason you > need > >>> them other than migrating flows across environments), then there are > lots of > >>> great write ups by others on how to achieve this workflow. The trick > is in > >>> the Registry A to Registry B step, and for that there are a few > approaches > >>> and tools that others have documented. > >>> > >>> > >>> > >>> A great write up about this “one Registry per environment” FDLC > approach > >>> is included in a blog post by Pierre Villard, “Automate workflow > deployment > >>> in Apache NiFi with the NiFi Registry” [1]. As Pierre describes, you > can use > >>> the NiFi CLI [1], included in the NiFi Toolkit as a separate executable > >>> since NiFi 1.6.0 [2] as a tool to migrate flow version snapshots from > one > >>> NiFi Registry to another. Specifically, the CLI tools “registry > >>> export-flow-version” and “registry import-flow-version” commands. > >>> > >>> > >>> > >>> An alternative to using the NiFi CLI would be NiPyApi [4], a NiFi > client > >>> SDK that makes the REST API for NiFi and NiFi Registry easier to work > with > >>> from Python automation scripts. There’s a good write up on using this > to > >>> interact with flows in NiFi Registry instances from Timothy Spann [5]. > >>> > >>> > >>> > >>> I hope this helps you workaround the issue you are running into. It > >>> certainly does not solve the specific error you have documented, but > perhaps > >>> it is a viable alternative that avoids that exception. > >>> > >>> > >>> > >>> Best, > >>> Kevin > >>> > >>> > >>> > >>> [1] > >>> https://pierrevillard.com/2018/04/09/automate-workflow- > deployment-in-apache-nifi-with-the-nifi-registry/ > >>> > >>> [2] > >>> https://github.com/apache/nifi/tree/master/nifi-toolkit/ > nifi-toolkit-cli > >>> > >>> [3] https://nifi.apache.org/download.html > >>> > >>> [4] https://github.com/Chaffelson/nipyapi > >>> > >>> [5] > >>> https://community.hortonworks.com/articles/177301/big-data- > devops-apache-nifi-flow-versioning-and-au.html > >>> > >>> > >>> > >>> > >>> > >>> From: Ryan H <[email protected]> > >>> Reply-To: <[email protected]> > >>> Date: Sunday, May 6, 2018 at 21:05 > >>> To: <[email protected]> > >>> Subject: Error Reference When Creating Template From PG Under NiFi > >>> Registry Version Control > >>> > >>> > >>> > >>> Hi All, > >>> > >>> We have found what seems to be an issue when creating a template of a > >>> Process Group that is under version control with NiFi Registry. What > seems > >>> to happen is that there is a hard reference to the Registry Instance > from > >>> the Env that the template was created from within the template xml > file. > >>> When importing the template into a new Env the reference to the > registry > >>> goes with the template and looks to stay with the new process group, is > >>> written to the flow.xml.gz, and causes an issue when clustered (the > nodes > >>> end up not being able to sync after a while because of the error and > goes > >>> into a disconnected state). > >>> > >>> I'll say upfront that this somewhat defeats the purpose of having the > >>> registry and I'm guessing that there will be suggestions that we should > >>> either go the template FLDC route or go the registry FLDC route. For > this > >>> purpose we cannot share the registry instance between the two > environments > >>> and may be the case that we should not use the registry at this time. > >>> > >>> Should this be an error? Is a better way other than going in a > >>> searching/destroying all the old registry references out of the > template and > >>> flow xml files? Should we manually change the registry instances in > each of > >>> the environments to be the same guid id? > >>> > >>> > >>> > >>> As always any help or direction is greatly appreciated. > >>> > >>> Cheers, > >>> > >>> Ryan Howell > >>> > >>> > >>> template.xml snippet (created template to download, which will then be > >>> uploaded into new env) (seems that the error is automatically created > >>> knowing that it is going to be a problem importing): > >>> ... > >>> <versionControlInformation> > >>> > >>> <bucketId>1f26b7fd-8987-491d-a2ac-4f87292ab29b</bucketId> > >>> <bucketName>Dev-Test</bucketName> > >>> <flowDescription></flowDescription> > >>> > >>> <flowId>c8e54d77-ce35-4be7-aff6-339a96b58e7d</flowId> > >>> <flowName>MySuperCoolFlow</flowName> > >>> > >>> <registryId>4ee39816-0161-1000-ffff-ffffc4085349</registryId> > >>> > >>> <registryName>4ee39816-0161-1000-ffff-ffffc4085349</registryName> > >>> <state>SYNC_FAILURE</state> > >>> <stateExplanation>Unable to synchronize Process > >>> Group with Flow Registry because Process Group was placed under Version > >>> Control using Flow Registry with identifier > >>> 4ee39816-0161-1000-ffff-ffffc4085349 but cannot find any Flow > Registry with > >>> this identifier</stateExplanation> > >>> <version>1</version> > >>> </versionControlInformation> > >>> </processGroups> > >>> <processGroups> > >>> <id>9d90f6ea-f2a1-3d65-0000-000000000000</id> > >>> > >>> <parentGroupId>d8cf7c44-48c7-3c7d-0000-000000000000</parentGroupId> > >>> <position> > >>> <x>422.23612851041094</x> > >>> <y>-1527.1966721061617</y> > >>> </position> > >>> > >>> <versionedComponentId>322cb8f0-cec6-30f9-9565-6948120c96d6</ > versionedComponentId> > >>> <comments></comments> > >>> <contents> > >>> <connections> > >>> <id>80c3a35a-27ec-33a2-0000- > 000000000000</id> > >>> > >>> <parentGroupId>9d90f6ea-f2a1-3d65-0000-000000000000</parentGroupId> > >>> > >>> <versionedComponentId>d77ff912-6127-39f2-9c7a-2eb3c7dde817</ > versionedComponentId> > >>> <backPressureDataSizeThreshold>1 > >>> GB</backPressureDataSizeThreshold> > >>> > >>> <backPressureObjectThreshold>10000</backPressureObjectThreshold> > >>> <destination> > >>> <groupId>9d90f6ea-f2a1-3d65-0000-000000000000</groupId> > >>> <id>14b964e1-0775-3e6f-0000-000000000000</id> > >>> <type>PROCESSOR</type> > >>> > >>> <versionedComponentId>66873d14-b329-3322-b845-d144571ac93e</ > versionedComponentId> > >>> </destination> > >>> <flowFileExpiration>0 > >>> sec</flowFileExpiration> > >>> <labelIndex>1</labelIndex> > >>> <name></name> > >>> > >>> <selectedRelationships>success</selectedRelationships> > >>> <source> > >>> <groupId>9d90f6ea-f2a1-3d65-0000-000000000000</groupId> > >>> <id>f26999d5-adbe-33ea-0000-000000000000</id> > >>> <type>PROCESSOR</type> > >>> > >>> <versionedComponentId>c928e355-f4b0-3673-ba8e-9d7f0cf57d5f</ > versionedComponentId> > >>> </source> > >>> > >> > > >
