I am actually working on this same problem, Deploying versioned flows via NiF API's,
Here is a good guide https://pierrevillard.com/?s=registry but I am having authentication issues with a secured instance of NiFi NiFi Registry is Docker'ized and deployed without authentication enabled (IMO, dont need security on the registry) NiFi nodes are docker'ized and deployed secured with nifi.security.user.login.identity.provider=ldap-provider I have nifi-toolkit/bin/cli.sh registry * commands are working fine. Quite easy because the registry is not authenticated. I can interact with the registry just fine. Now I need to get the nifi-toolkit/bin/cli.sh nifi * Commands working but I cant talk with the nifi instances as they are all secured. NiFi instances are all configured using LDAP like so nifi.security.user.login.identity.provider=ldap-provider I assume I cant get the nifi-toolkit to authenticate using LDAP. It appears I must use SSL certificates only. 1) How do I get nifi-toolkit to authenticate using an LDAP? or 2) How do I convert the NiFi nodes over to a mixed mode, LDAP + Certificates? (Cant find articles or setups showing mixed authentication mechanisms) maybe 3) .... There might be a 3rd option, NiPyAPI, but I dont know much about using NiPyAPI in a secured LDAP setup. Erik Anderson Bloomberg On Tue, Feb 5, 2019, at 12:49 AM, Tim Dean wrote: > Bryan, > > I’ve been working through this and I have it working - mostly. My flows > have been registered successfully, and I’ve been able to instantiate > the parent flow via the NiFi API to create a new process group. > > The final piece - I think - is figuring out how to properly configure > the controller services referenced by my flows using the APIs. I know I > can use the API to create a new controller service and generate an ID > for each new controller service. But how do I update all of my NiFi > components that reference the controller service by ID. Is there a > straightforward way to do this? > > Thanks > > -Tim > > > > On Feb 1, 2019, at 11:09 AM, Bryan Bende <[email protected]> wrote: > > > > Tim, > > > > For moving between registries the approach you described sounds > > correct, admittedly it would be nice if there was an easier way. > > > > In your case it is only two levels, but in general you'd have to start > > at the lowest level, and work your way up the levels, applying the > > correct ids from the level below. > > > > -Bryan > > > > > > On Fri, Feb 1, 2019 at 11:44 AM Tim Dean <[email protected]> wrote: > >> > >> Thanks Bryan - > >> > >> If I use a nested versioned process group, it appears that the parent > >> group will reference its child process groups by ID. If I am populating my > >> registry in a new environment using the API, those IDs will be dynamically > >> generated as I make the API calls, correct? > >> > >> In that case, do I need to POST the child process groups first, get their > >> bucket and flow IDs back from the registry API, and then manipulate the > >> JSON for the parent process group to replace the contents of the > >> “versionedFlowCoordinates” JSON object’s identifiers with the new IDs? > >> > >> Or is there a better way to insert parent and child process groups via my > >> scripts? > >> > >> -Tim > >> > >> On Jan 31, 2019, at 6:25 PM, Bryan Bende <[email protected]> wrote: > >> > >> Hi Tim, > >> > >> I think the second option is the correct approach. The higher level > >> versioned PG is the way of saying that the lower level PGs work together > >> as a cohesive unit. > >> > >> -Bryan > >> > >> On Thu, Jan 31, 2019 at 7:00 PM Tim Dean <[email protected]> wrote: > >>> > >>> I am trying to automate deployment of a NiFi flow with several versioned > >>> process groups using the NiFi APIs. The basic setup I have is this: > >>> > >>> I have a dozen or so process groups, each of which has been versioned > >>> within a NiFi registry > >>> My root process group contains each of those process groups, with various > >>> connections between their ports as well as a few variable definitions and > >>> controller service instances. > >>> > >>> > >>> My goal is to deploy this flow, including the root process group that > >>> links the versioned PGs as well as the versioned PGs themselves. So far, > >>> I’ve managed to use the registry API to create a bucket and to add the > >>> versioned flows into the registry. Now I’m trying to use the NiFi APIs to > >>> instantiate the root PG and link together all the versioned PGs that I > >>> have just inserted into the registry. > >>> > >>> The approach I have been trying is to capture my root PG as a template, > >>> and then use the NiFi APIs to import and then instantiate that template. > >>> I have gotten this much working, but unfortunately that leaves the PGs > >>> disconnected from the versioned flows in the registry. I was hoping there > >>> was a way to transform the template to insert the appropriate bucket and > >>> flow IDs but I have been unable to figure out if this is possible. > >>> > >>> Alternatively, I suspect I could create an intermediate process group to > >>> contain all my “real” PGs, and then version that intermediate PG. I could > >>> then use the APIs to instantiate a new PG at the root level that is > >>> imported from the intermediate PG. I suspect that this could work, but it > >>> is less than ideal because I’m creating an artificial intermediate PG to > >>> contain all of my real contents, which will be a distraction for users > >>> who come into the NiFi data flow manager to monitor this process. > >>> > >>> Am I looking at this approach correctly? Are there other options I should > >>> be considering? > >>> > >>> Thanks in advance, > >>> > >>> -Tim > >> > >> -- > >> Sent from Gmail Mobile > >> > >> > >
