I have continued to try isolating this problem. I’m not there yet, but I have made some observations:
I currently have a PG called ‘Project’ and, embedded within it, another PG called `Subproject’. Project has nothing in it other than Subproject. No processors, no controller services, nothing. Subproject has two controller services defined within it, one of which references the other Both Project and Subproject have been versioned in a local NiFi registry If I import Subproject directly into a NiFi flow, its 2 controller services are correctly instantiated. But if I import Project and then navigate down into Subproject, I find that the depending service controller no longer has a valid reference to the dependent service controller Looking at the detailed IDs of controller services using the NiFi and NiFi registry APIs directly, I have found that a successful import of Subproject results in new controller service IDs being assigned to replace the IDs for the service controllers within the NiFi registry, Those new controller service IDs are correctly referenced in the appropriate component property values. Looking at the same structures using the NiFi and NiFi registry APIs, I have found that an unsuccessful import of Project does result in new controller service IDs being assigned, but that the references to those service controller IDs from component property values have not been updated. Those properties still reference the controller service using the ID that is present in the NiFi registry’s copy of the flow. Doing this with a couple of out-of-the-box controller services (ElasticSearch client and SSL context) in a fresh flow did not exhibit this problem, at least not in an obvious way. The problem is encountered when using our own custom-built controller services. So either the problem is something unique with our particular controller services, or something unique has happened with our flows to put them in a bad state that causes this problem. Not sure if all of this helps anyone else to help with a workaround, but figure more information might be helpful. -Tim > On Feb 5, 2019, at 11:28 AM, Bryan Bende <[email protected]> wrote: > > I just did a quick test where I created a PG with Processor -> CS1 -> > CS2, saved that to registry and then imported another instance of it, > and it looks like CS1 is still referencing CS2. > > That doesn't mean there isn't an issue though. Let us know if you > figure out a specific scenario that creates a problem. > > On Tue, Feb 5, 2019 at 11:57 AM Tim Dean <[email protected]> wrote: >> >> Thanks again Bryan >> >> For my current use case I think I can get away with duplicating controller >> services so that each versioned PG only references its own controller >> services. >> >> Unfortunately my initial changes to try this out haven’t worked completely >> as you are describing. I do find that any processors that reference a >> controller service are correctly configured after instantiating the >> versioned PG. However, I have a couple of cases where one controller service >> depends on another. And it appears that references from one service >> controller to another (all within the same process group) are not being >> correctly maintained when I instantiate via the API. >> >> I will dig deeper to find more specific details on what is happening, but at >> a glance it doesn’t appear to be working for me. Has this use case been >> tested before or am I running into a new issue? >> >> -Tim >> >> Sent from my iPad >> >>> On Feb 5, 2019, at 8:41 AM, Bryan Bende <[email protected]> wrote: >>> >>> Tim, >>> >>> Unfortunately there isn't really a straight forward way to do this. >>> There are probably two things you can consider... >>> >>> 1) Any services that are part of the versioned process group will be >>> imported to the next environment and the components will have the >>> correct references to them. It is only when the versioned process >>> group references a service in a higher level that it is not included >>> and needs to be reset in the next environment. So there may be cases >>> where it is just easier to duplicate some services and make the >>> versioned process groups more portable. >>> >>> 2) For stuff where you do need to reset the reference, you might be >>> able to come up with a convention. For example, you go through the >>> newly imported PG and find all the components that reference service >>> ids that don't exist. For each one of these service ids you know the >>> type of service required based on the property, so you can then check >>> the parent PG for a service of the same type, and if only one exists, >>> then take that id. >>> >>> A lot of this stuff was originally meant to be done through the UI >>> which is why it is a bit tricky to script, but I would really like to >>> get all of this stuff built into the CLI. >>> >>> -Bryan >>> >>>> On Tue, Feb 5, 2019 at 12:49 AM Tim Dean <[email protected]> wrote: >>>> >>>> Bryan, >>>> >>>> I’ve been working through this and I have it working - mostly. My flows >>>> have been registered successfully, and I’ve been able to instantiate the >>>> parent flow via the NiFi API to create a new process group. >>>> >>>> The final piece - I think - is figuring out how to properly configure the >>>> controller services referenced by my flows using the APIs. I know I can >>>> use the API to create a new controller service and generate an ID for each >>>> new controller service. But how do I update all of my NiFi components that >>>> reference the controller service by ID. Is there a straightforward way to >>>> do this? >>>> >>>> Thanks >>>> >>>> -Tim >>>> >>>> >>>>> On Feb 1, 2019, at 11:09 AM, Bryan Bende <[email protected]> wrote: >>>>> >>>>> Tim, >>>>> >>>>> For moving between registries the approach you described sounds >>>>> correct, admittedly it would be nice if there was an easier way. >>>>> >>>>> In your case it is only two levels, but in general you'd have to start >>>>> at the lowest level, and work your way up the levels, applying the >>>>> correct ids from the level below. >>>>> >>>>> -Bryan >>>>> >>>>> >>>>>> On Fri, Feb 1, 2019 at 11:44 AM Tim Dean <[email protected]> wrote: >>>>>> >>>>>> Thanks Bryan - >>>>>> >>>>>> If I use a nested versioned process group, it appears that the parent >>>>>> group will reference its child process groups by ID. If I am populating >>>>>> my registry in a new environment using the API, those IDs will be >>>>>> dynamically generated as I make the API calls, correct? >>>>>> >>>>>> In that case, do I need to POST the child process groups first, get >>>>>> their bucket and flow IDs back from the registry API, and then >>>>>> manipulate the JSON for the parent process group to replace the contents >>>>>> of the “versionedFlowCoordinates” JSON object’s identifiers with the new >>>>>> IDs? >>>>>> >>>>>> Or is there a better way to insert parent and child process groups via >>>>>> my scripts? >>>>>> >>>>>> -Tim >>>>>> >>>>>> On Jan 31, 2019, at 6:25 PM, Bryan Bende <[email protected]> wrote: >>>>>> >>>>>> Hi Tim, >>>>>> >>>>>> I think the second option is the correct approach. The higher level >>>>>> versioned PG is the way of saying that the lower level PGs work together >>>>>> as a cohesive unit. >>>>>> >>>>>> -Bryan >>>>>> >>>>>>> On Thu, Jan 31, 2019 at 7:00 PM Tim Dean <[email protected]> wrote: >>>>>>> >>>>>>> I am trying to automate deployment of a NiFi flow with several >>>>>>> versioned process groups using the NiFi APIs. The basic setup I have is >>>>>>> this: >>>>>>> >>>>>>> I have a dozen or so process groups, each of which has been versioned >>>>>>> within a NiFi registry >>>>>>> My root process group contains each of those process groups, with >>>>>>> various connections between their ports as well as a few variable >>>>>>> definitions and controller service instances. >>>>>>> >>>>>>> >>>>>>> My goal is to deploy this flow, including the root process group that >>>>>>> links the versioned PGs as well as the versioned PGs themselves. So >>>>>>> far, I’ve managed to use the registry API to create a bucket and to add >>>>>>> the versioned flows into the registry. Now I’m trying to use the NiFi >>>>>>> APIs to instantiate the root PG and link together all the versioned PGs >>>>>>> that I have just inserted into the registry. >>>>>>> >>>>>>> The approach I have been trying is to capture my root PG as a template, >>>>>>> and then use the NiFi APIs to import and then instantiate that >>>>>>> template. I have gotten this much working, but unfortunately that >>>>>>> leaves the PGs disconnected from the versioned flows in the registry. I >>>>>>> was hoping there was a way to transform the template to insert the >>>>>>> appropriate bucket and flow IDs but I have been unable to figure out if >>>>>>> this is possible. >>>>>>> >>>>>>> Alternatively, I suspect I could create an intermediate process group >>>>>>> to contain all my “real” PGs, and then version that intermediate PG. I >>>>>>> could then use the APIs to instantiate a new PG at the root level that >>>>>>> is imported from the intermediate PG. I suspect that this could work, >>>>>>> but it is less than ideal because I’m creating an artificial >>>>>>> intermediate PG to contain all of my real contents, which will be a >>>>>>> distraction for users who come into the NiFi data flow manager to >>>>>>> monitor this process. >>>>>>> >>>>>>> Am I looking at this approach correctly? Are there other options I >>>>>>> should be considering? >>>>>>> >>>>>>> Thanks in advance, >>>>>>> >>>>>>> -Tim >>>>>> >>>>>> -- >>>>>> Sent from Gmail Mobile >>>>>> >>>>>> >>>>
