Thanks Bryan - I checked this out in the UI and, sure enough, Controller Service B doesn’t show up as a referencing component when enabling controller service B. So the UI is acting exactly as I would expect given what I am seeing via the API. For some reason the references API is not returning the referencing controller service.
I am wondering if this failure to return the referencing controller service is related to the workaround I have in place for https://issues.apache.org/jira/browse/NIFI-6001 <https://issues.apache.org/jira/browse/NIFI-6001>. To work around that problem, my script is doing the following: Go through each of the controller services in the PG and creating a map of APIs they implement via the `component.controllerServiceApis.type` value. With that map in hand, go through each the controller services again and look at their property descriptors for properties that implement this service API via the `component.descriptors.{name}.identifiesControllerService` value. For each descriptor that identifies the controller service API it expects, retrieve the ID of its implementation from the map created earlier, and set the `component.properties.{name}` value to that ID Save the modified controller service using `PUT /nifi-api/controller-services/{id}` Given that workaround that is currently in place, can you think of any reason why the references retrieval API would miss these references? Perhaps there is something specific that needs to be done when modifying a controller service’s property values to allow references to be correctly recomputed? This scripting is currently in Python, and we are considering moving it over to use NiPy API on the not-too-distant future. However, I suspect that we would still need the low-level direct access to the API for doing things like the workaround described above. I also see that the exact feature we’re discussing here seems to be in there as a feature request (https://github.com/Chaffelson/nipyapi/issues/113 <https://github.com/Chaffelson/nipyapi/issues/113>) so perhaps that isn’t implemented as part of the released version at this time? I can convert my logic to emulate what you are doing in the CLI, but if there’s a simple fix to the problem with the approach I’ve already got that would be preferable. -Tim >> > On Feb 22, 2019, at 8:44 AM, Bryan Bende <[email protected]> wrote: > > Tim, > > Hard to say what is going on with the references end-point, I would > have to play around with it, but I believe calling > "controller-services/<id>/references" should return other services > that reference the service with the given id. It is used in the UI > when you go to enable/disable a service, and you have the option to > enable all referencing services as well. So you could probably open > chrome dev tools and play around with it through the UI and see what > calls are being made. > > I'm not sure what language you are scripting all of this in, but the > CLI already has a command to enable all services in a process group: > > https://github.com/apache/nifi/blob/master/nifi-toolkit/nifi-toolkit-cli/src/main/java/org/apache/nifi/toolkit/cli/impl/command/nifi/pg/PGEnableControllerServices.java#L76-L154 > > Instead of trying to implement the logic you have, we just go in a > loop trying to enable valid services until no more enable. So in your > example, first time through it would enable service A because it is > valid and standalone, then second time through service B would now be > valid so it would enabled, and third time through there are no more > services to enable. > > If you are working Python then I believe NiPyApi also has a similar command. > > Thanks, > > Bryan > > On Thu, Feb 21, 2019 at 5:14 PM Tim Dean <[email protected]> wrote: >> >> I’ve run into another problem. I’m not sure if it’s related to the bug >> you’ve created and referenced below, but it may be caused by my workaround >> to that bug so I’m replying to the existing thread to provide full context. >> >> The problem I’ve got now is an issue with programmatically enabling all of >> my controller services once I’ve instantiated the nested PG’s from the NiFi >> registry. In the case where one controller depends on another, I must >> enable the controller service that is depended on before I enable the >> controller service that depends on it. My logic for doing this is the >> following: >> >> For each of the process groups I’ve instantiated, get the list of all >> controller services in that group using >> /nifi-api/flow/process-groups/{id}/controller-services >> For each of those controller services, get the list of references to that >> service using /nifi-api/controller-services/{id}/references >> Using this information, create a map for each controller service to a list >> of controller services it depends on >> When I go to start all my controller services, use that map to look up each >> controller service’s dependencies and make sure those are started first. >> >> >> I have now run into circumstances where this does not work. I initially >> assumed that my algorithm was wrong or there was a bug in its >> implementation, but I have now confirmed this problem by looking directly in >> the NiFi API at my controller services, and here is what I have found: >> >> Controller Service A is a simple standalone service >> Controller Service B has a property whose value is set to the ID of >> controller service A. This value was set by a workaround I implemented for >> the bug described earlier in this thread, so perhaps something about how >> this value got set is a factor. >> When I make a GET request to >> /nifi-api/controller-services/<id-for-A>/references, it returns an empty >> list and does not controller service B as I would have expected it to >> >> >> >> So this leads me to 2 questions: >> >> If I have a controller service B that has a property value referencing >> controller service B, should I expect to see controller service A in the >> list of controller service A’s references returned by the API? If so I’d >> like to understand what triggers this so I can understand if this is a new >> bug or if I’ve somehow created a problem by working around the previous bug. >> Is there a more straightforward way to enable all my controller services for >> a process group that automatically handles the dependency issues I’m dealing >> with? >> >> >> Thanks for any advice you can provide >> >> -Tim >> >> >> >> On Feb 6, 2019, at 9:56 AM, Bryan Bende <[email protected]> wrote: >> >> Hi Tim, >> >> Thanks for the additional information. I was able to reproduce the >> issue with the services inside a nested versioned process group and it >> is definitely a bug, although I don't know where yet. >> >> I created this JIRA - https://issues.apache.org/jira/browse/NIFI-6001 >> >> Thanks, >> >> Bryan >> >> On Wed, Feb 6, 2019 at 2:04 AM Tim Dean <[email protected]> wrote: >> >> >> I have continued to try isolating this problem. I’m not there yet, but I >> have made some observations: >> >> I currently have a PG called ‘Project’ and, embedded within it, another PG >> called `Subproject’. >> Project has nothing in it other than Subproject. No processors, no >> controller services, nothing. >> Subproject has two controller services defined within it, one of which >> references the other >> Both Project and Subproject have been versioned in a local NiFi registry >> If I import Subproject directly into a NiFi flow, its 2 controller services >> are correctly instantiated. But if I import Project and then navigate down >> into Subproject, I find that the depending service controller no longer has >> a valid reference to the dependent service controller >> Looking at the detailed IDs of controller services using the NiFi and NiFi >> registry APIs directly, I have found that a successful import of Subproject >> results in new controller service IDs being assigned to replace the IDs for >> the service controllers within the NiFi registry, Those new controller >> service IDs are correctly referenced in the appropriate component property >> values. >> Looking at the same structures using the NiFi and NiFi registry APIs, I have >> found that an unsuccessful import of Project does result in new controller >> service IDs being assigned, but that the references to those service >> controller IDs from component property values have not been updated. Those >> properties still reference the controller service using the ID that is >> present in the NiFi registry’s copy of the flow. >> Doing this with a couple of out-of-the-box controller services >> (ElasticSearch client and SSL context) in a fresh flow did not exhibit this >> problem, at least not in an obvious way. The problem is encountered when >> using our own custom-built controller services. So either the problem is >> something unique with our particular controller services, or something >> unique has happened with our flows to put them in a bad state that causes >> this problem. >> >> >> Not sure if all of this helps anyone else to help with a workaround, but >> figure more information might be helpful. >> >> -Tim >> >> On Feb 5, 2019, at 11:28 AM, Bryan Bende <[email protected]> wrote: >> >> I just did a quick test where I created a PG with Processor -> CS1 -> >> CS2, saved that to registry and then imported another instance of it, >> and it looks like CS1 is still referencing CS2. >> >> That doesn't mean there isn't an issue though. Let us know if you >> figure out a specific scenario that creates a problem. >> >> On Tue, Feb 5, 2019 at 11:57 AM Tim Dean <[email protected]> wrote: >> >> >> Thanks again Bryan >> >> For my current use case I think I can get away with duplicating controller >> services so that each versioned PG only references its own controller >> services. >> >> Unfortunately my initial changes to try this out haven’t worked completely >> as you are describing. I do find that any processors that reference a >> controller service are correctly configured after instantiating the >> versioned PG. However, I have a couple of cases where one controller service >> depends on another. And it appears that references from one service >> controller to another (all within the same process group) are not being >> correctly maintained when I instantiate via the API. >> >> I will dig deeper to find more specific details on what is happening, but at >> a glance it doesn’t appear to be working for me. Has this use case been >> tested before or am I running into a new issue? >> >> -Tim >> >> Sent from my iPad >> >> On Feb 5, 2019, at 8:41 AM, Bryan Bende <[email protected]> wrote: >> >> Tim, >> >> Unfortunately there isn't really a straight forward way to do this. >> There are probably two things you can consider... >> >> 1) Any services that are part of the versioned process group will be >> imported to the next environment and the components will have the >> correct references to them. It is only when the versioned process >> group references a service in a higher level that it is not included >> and needs to be reset in the next environment. So there may be cases >> where it is just easier to duplicate some services and make the >> versioned process groups more portable. >> >> 2) For stuff where you do need to reset the reference, you might be >> able to come up with a convention. For example, you go through the >> newly imported PG and find all the components that reference service >> ids that don't exist. For each one of these service ids you know the >> type of service required based on the property, so you can then check >> the parent PG for a service of the same type, and if only one exists, >> then take that id. >> >> A lot of this stuff was originally meant to be done through the UI >> which is why it is a bit tricky to script, but I would really like to >> get all of this stuff built into the CLI. >> >> -Bryan >> >> On Tue, Feb 5, 2019 at 12:49 AM Tim Dean <[email protected]> wrote: >> >> Bryan, >> >> I’ve been working through this and I have it working - mostly. My flows have >> been registered successfully, and I’ve been able to instantiate the parent >> flow via the NiFi API to create a new process group. >> >> The final piece - I think - is figuring out how to properly configure the >> controller services referenced by my flows using the APIs. I know I can use >> the API to create a new controller service and generate an ID for each new >> controller service. But how do I update all of my NiFi components that >> reference the controller service by ID. Is there a straightforward way to do >> this? >> >> Thanks >> >> -Tim >> >> >> On Feb 1, 2019, at 11:09 AM, Bryan Bende <[email protected]> wrote: >> >> Tim, >> >> For moving between registries the approach you described sounds >> correct, admittedly it would be nice if there was an easier way. >> >> In your case it is only two levels, but in general you'd have to start >> at the lowest level, and work your way up the levels, applying the >> correct ids from the level below. >> >> -Bryan >> >> >> On Fri, Feb 1, 2019 at 11:44 AM Tim Dean <[email protected]> wrote: >> >> Thanks Bryan - >> >> If I use a nested versioned process group, it appears that the parent group >> will reference its child process groups by ID. If I am populating my >> registry in a new environment using the API, those IDs will be dynamically >> generated as I make the API calls, correct? >> >> In that case, do I need to POST the child process groups first, get their >> bucket and flow IDs back from the registry API, and then manipulate the JSON >> for the parent process group to replace the contents of the >> “versionedFlowCoordinates” JSON object’s identifiers with the new IDs? >> >> Or is there a better way to insert parent and child process groups via my >> scripts? >> >> -Tim >> >> On Jan 31, 2019, at 6:25 PM, Bryan Bende <[email protected]> wrote: >> >> Hi Tim, >> >> I think the second option is the correct approach. The higher level >> versioned PG is the way of saying that the lower level PGs work together as >> a cohesive unit. >> >> -Bryan >> >> On Thu, Jan 31, 2019 at 7:00 PM Tim Dean <[email protected]> wrote: >> >> I am trying to automate deployment of a NiFi flow with several versioned >> process groups using the NiFi APIs. The basic setup I have is this: >> >> I have a dozen or so process groups, each of which has been versioned within >> a NiFi registry >> My root process group contains each of those process groups, with various >> connections between their ports as well as a few variable definitions and >> controller service instances. >> >> >> My goal is to deploy this flow, including the root process group that links >> the versioned PGs as well as the versioned PGs themselves. So far, I’ve >> managed to use the registry API to create a bucket and to add the versioned >> flows into the registry. Now I’m trying to use the NiFi APIs to instantiate >> the root PG and link together all the versioned PGs that I have just >> inserted into the registry. >> >> The approach I have been trying is to capture my root PG as a template, and >> then use the NiFi APIs to import and then instantiate that template. I have >> gotten this much working, but unfortunately that leaves the PGs disconnected >> from the versioned flows in the registry. I was hoping there was a way to >> transform the template to insert the appropriate bucket and flow IDs but I >> have been unable to figure out if this is possible. >> >> Alternatively, I suspect I could create an intermediate process group to >> contain all my “real” PGs, and then version that intermediate PG. I could >> then use the APIs to instantiate a new PG at the root level that is imported >> from the intermediate PG. I suspect that this could work, but it is less >> than ideal because I’m creating an artificial intermediate PG to contain all >> of my real contents, which will be a distraction for users who come into the >> NiFi data flow manager to monitor this process. >> >> Am I looking at this approach correctly? Are there other options I should be >> considering? >> >> Thanks in advance, >> >> -Tim >> >> >> -- >> Sent from Gmail Mobile >> >> >> >> >>
