I was recently also working through issues with non-processor components in NiPyAPI and prepared some controller service management methods. You can take a look here as it may be a shortcut to testing the automation you are looking for. https://github.com/Chaffelson/nipyapi/blob/dac6601b06e1675e434fbe4428225717068f7b88/nipyapi/canvas.py#L1034
On Tue, Feb 5, 2019 at 2:42 PM Bryan Bende <[email protected]> wrote: > Tim, > > Unfortunately there isn't really a straight forward way to do this. > There are probably two things you can consider... > > 1) Any services that are part of the versioned process group will be > imported to the next environment and the components will have the > correct references to them. It is only when the versioned process > group references a service in a higher level that it is not included > and needs to be reset in the next environment. So there may be cases > where it is just easier to duplicate some services and make the > versioned process groups more portable. > > 2) For stuff where you do need to reset the reference, you might be > able to come up with a convention. For example, you go through the > newly imported PG and find all the components that reference service > ids that don't exist. For each one of these service ids you know the > type of service required based on the property, so you can then check > the parent PG for a service of the same type, and if only one exists, > then take that id. > > A lot of this stuff was originally meant to be done through the UI > which is why it is a bit tricky to script, but I would really like to > get all of this stuff built into the CLI. > > -Bryan > > On Tue, Feb 5, 2019 at 12:49 AM Tim Dean <[email protected]> wrote: > > > > Bryan, > > > > I’ve been working through this and I have it working - mostly. My flows > have been registered successfully, and I’ve been able to instantiate the > parent flow via the NiFi API to create a new process group. > > > > The final piece - I think - is figuring out how to properly configure > the controller services referenced by my flows using the APIs. I know I can > use the API to create a new controller service and generate an ID for each > new controller service. But how do I update all of my NiFi components that > reference the controller service by ID. Is there a straightforward way to > do this? > > > > Thanks > > > > -Tim > > > > > > > On Feb 1, 2019, at 11:09 AM, Bryan Bende <[email protected]> wrote: > > > > > > Tim, > > > > > > For moving between registries the approach you described sounds > > > correct, admittedly it would be nice if there was an easier way. > > > > > > In your case it is only two levels, but in general you'd have to start > > > at the lowest level, and work your way up the levels, applying the > > > correct ids from the level below. > > > > > > -Bryan > > > > > > > > > On Fri, Feb 1, 2019 at 11:44 AM Tim Dean <[email protected]> wrote: > > >> > > >> Thanks Bryan - > > >> > > >> If I use a nested versioned process group, it appears that the parent > group will reference its child process groups by ID. If I am populating my > registry in a new environment using the API, those IDs will be dynamically > generated as I make the API calls, correct? > > >> > > >> In that case, do I need to POST the child process groups first, get > their bucket and flow IDs back from the registry API, and then manipulate > the JSON for the parent process group to replace the contents of the > “versionedFlowCoordinates” JSON object’s identifiers with the new IDs? > > >> > > >> Or is there a better way to insert parent and child process groups > via my scripts? > > >> > > >> -Tim > > >> > > >> On Jan 31, 2019, at 6:25 PM, Bryan Bende <[email protected]> wrote: > > >> > > >> Hi Tim, > > >> > > >> I think the second option is the correct approach. The higher level > versioned PG is the way of saying that the lower level PGs work together as > a cohesive unit. > > >> > > >> -Bryan > > >> > > >> On Thu, Jan 31, 2019 at 7:00 PM Tim Dean <[email protected]> wrote: > > >>> > > >>> I am trying to automate deployment of a NiFi flow with several > versioned process groups using the NiFi APIs. The basic setup I have is > this: > > >>> > > >>> I have a dozen or so process groups, each of which has been > versioned within a NiFi registry > > >>> My root process group contains each of those process groups, with > various connections between their ports as well as a few variable > definitions and controller service instances. > > >>> > > >>> > > >>> My goal is to deploy this flow, including the root process group > that links the versioned PGs as well as the versioned PGs themselves. So > far, I’ve managed to use the registry API to create a bucket and to add the > versioned flows into the registry. Now I’m trying to use the NiFi APIs to > instantiate the root PG and link together all the versioned PGs that I have > just inserted into the registry. > > >>> > > >>> The approach I have been trying is to capture my root PG as a > template, and then use the NiFi APIs to import and then instantiate that > template. I have gotten this much working, but unfortunately that leaves > the PGs disconnected from the versioned flows in the registry. I was hoping > there was a way to transform the template to insert the appropriate bucket > and flow IDs but I have been unable to figure out if this is possible. > > >>> > > >>> Alternatively, I suspect I could create an intermediate process > group to contain all my “real” PGs, and then version that intermediate PG. > I could then use the APIs to instantiate a new PG at the root level that is > imported from the intermediate PG. I suspect that this could work, but it > is less than ideal because I’m creating an artificial intermediate PG to > contain all of my real contents, which will be a distraction for users who > come into the NiFi data flow manager to monitor this process. > > >>> > > >>> Am I looking at this approach correctly? Are there other options I > should be considering? > > >>> > > >>> Thanks in advance, > > >>> > > >>> -Tim > > >> > > >> -- > > >> Sent from Gmail Mobile > > >> > > >> > > >
