Thanks again Bryan For my current use case I think I can get away with duplicating controller services so that each versioned PG only references its own controller services.
Unfortunately my initial changes to try this out haven’t worked completely as you are describing. I do find that any processors that reference a controller service are correctly configured after instantiating the versioned PG. However, I have a couple of cases where one controller service depends on another. And it appears that references from one service controller to another (all within the same process group) are not being correctly maintained when I instantiate via the API. I will dig deeper to find more specific details on what is happening, but at a glance it doesn’t appear to be working for me. Has this use case been tested before or am I running into a new issue? -Tim Sent from my iPad > On Feb 5, 2019, at 8:41 AM, Bryan Bende <[email protected]> wrote: > > Tim, > > Unfortunately there isn't really a straight forward way to do this. > There are probably two things you can consider... > > 1) Any services that are part of the versioned process group will be > imported to the next environment and the components will have the > correct references to them. It is only when the versioned process > group references a service in a higher level that it is not included > and needs to be reset in the next environment. So there may be cases > where it is just easier to duplicate some services and make the > versioned process groups more portable. > > 2) For stuff where you do need to reset the reference, you might be > able to come up with a convention. For example, you go through the > newly imported PG and find all the components that reference service > ids that don't exist. For each one of these service ids you know the > type of service required based on the property, so you can then check > the parent PG for a service of the same type, and if only one exists, > then take that id. > > A lot of this stuff was originally meant to be done through the UI > which is why it is a bit tricky to script, but I would really like to > get all of this stuff built into the CLI. > > -Bryan > >> On Tue, Feb 5, 2019 at 12:49 AM Tim Dean <[email protected]> wrote: >> >> Bryan, >> >> I’ve been working through this and I have it working - mostly. My flows have >> been registered successfully, and I’ve been able to instantiate the parent >> flow via the NiFi API to create a new process group. >> >> The final piece - I think - is figuring out how to properly configure the >> controller services referenced by my flows using the APIs. I know I can use >> the API to create a new controller service and generate an ID for each new >> controller service. But how do I update all of my NiFi components that >> reference the controller service by ID. Is there a straightforward way to do >> this? >> >> Thanks >> >> -Tim >> >> >>> On Feb 1, 2019, at 11:09 AM, Bryan Bende <[email protected]> wrote: >>> >>> Tim, >>> >>> For moving between registries the approach you described sounds >>> correct, admittedly it would be nice if there was an easier way. >>> >>> In your case it is only two levels, but in general you'd have to start >>> at the lowest level, and work your way up the levels, applying the >>> correct ids from the level below. >>> >>> -Bryan >>> >>> >>>> On Fri, Feb 1, 2019 at 11:44 AM Tim Dean <[email protected]> wrote: >>>> >>>> Thanks Bryan - >>>> >>>> If I use a nested versioned process group, it appears that the parent >>>> group will reference its child process groups by ID. If I am populating my >>>> registry in a new environment using the API, those IDs will be dynamically >>>> generated as I make the API calls, correct? >>>> >>>> In that case, do I need to POST the child process groups first, get their >>>> bucket and flow IDs back from the registry API, and then manipulate the >>>> JSON for the parent process group to replace the contents of the >>>> “versionedFlowCoordinates” JSON object’s identifiers with the new IDs? >>>> >>>> Or is there a better way to insert parent and child process groups via my >>>> scripts? >>>> >>>> -Tim >>>> >>>> On Jan 31, 2019, at 6:25 PM, Bryan Bende <[email protected]> wrote: >>>> >>>> Hi Tim, >>>> >>>> I think the second option is the correct approach. The higher level >>>> versioned PG is the way of saying that the lower level PGs work together >>>> as a cohesive unit. >>>> >>>> -Bryan >>>> >>>>> On Thu, Jan 31, 2019 at 7:00 PM Tim Dean <[email protected]> wrote: >>>>> >>>>> I am trying to automate deployment of a NiFi flow with several versioned >>>>> process groups using the NiFi APIs. The basic setup I have is this: >>>>> >>>>> I have a dozen or so process groups, each of which has been versioned >>>>> within a NiFi registry >>>>> My root process group contains each of those process groups, with various >>>>> connections between their ports as well as a few variable definitions and >>>>> controller service instances. >>>>> >>>>> >>>>> My goal is to deploy this flow, including the root process group that >>>>> links the versioned PGs as well as the versioned PGs themselves. So far, >>>>> I’ve managed to use the registry API to create a bucket and to add the >>>>> versioned flows into the registry. Now I’m trying to use the NiFi APIs to >>>>> instantiate the root PG and link together all the versioned PGs that I >>>>> have just inserted into the registry. >>>>> >>>>> The approach I have been trying is to capture my root PG as a template, >>>>> and then use the NiFi APIs to import and then instantiate that template. >>>>> I have gotten this much working, but unfortunately that leaves the PGs >>>>> disconnected from the versioned flows in the registry. I was hoping there >>>>> was a way to transform the template to insert the appropriate bucket and >>>>> flow IDs but I have been unable to figure out if this is possible. >>>>> >>>>> Alternatively, I suspect I could create an intermediate process group to >>>>> contain all my “real” PGs, and then version that intermediate PG. I could >>>>> then use the APIs to instantiate a new PG at the root level that is >>>>> imported from the intermediate PG. I suspect that this could work, but it >>>>> is less than ideal because I’m creating an artificial intermediate PG to >>>>> contain all of my real contents, which will be a distraction for users >>>>> who come into the NiFi data flow manager to monitor this process. >>>>> >>>>> Am I looking at this approach correctly? Are there other options I should >>>>> be considering? >>>>> >>>>> Thanks in advance, >>>>> >>>>> -Tim >>>> >>>> -- >>>> Sent from Gmail Mobile >>>> >>>> >>
