I’ve run into another problem. I’m not sure if it’s related to the bug you’ve
created and referenced below, but it may be caused by my workaround to that bug
so I’m replying to the existing thread to provide full context.
The problem I’ve got now is an issue with programmatically enabling all of my
controller services once I’ve instantiated the nested PG’s from the NiFi
registry. In the case where one controller depends on another, I must enable
the controller service that is depended on before I enable the controller
service that depends on it. My logic for doing this is the following:
For each of the process groups I’ve instantiated, get the list of all
controller services in that group using
/nifi-api/flow/process-groups/{id}/controller-services
For each of those controller services, get the list of references to that
service using /nifi-api/controller-services/{id}/references
Using this information, create a map for each controller service to a list of
controller services it depends on
When I go to start all my controller services, use that map to look up each
controller service’s dependencies and make sure those are started first.
I have now run into circumstances where this does not work. I initially assumed
that my algorithm was wrong or there was a bug in its implementation, but I
have now confirmed this problem by looking directly in the NiFi API at my
controller services, and here is what I have found:
Controller Service A is a simple standalone service
Controller Service B has a property whose value is set to the ID of controller
service A. This value was set by a workaround I implemented for the bug
described earlier in this thread, so perhaps something about how this value got
set is a factor.
When I make a GET request to
/nifi-api/controller-services/<id-for-A>/references, it returns an empty list
and does not controller service B as I would have expected it to
So this leads me to 2 questions:
If I have a controller service B that has a property value referencing
controller service B, should I expect to see controller service A in the list
of controller service A’s references returned by the API? If so I’d like to
understand what triggers this so I can understand if this is a new bug or if
I’ve somehow created a problem by working around the previous bug.
Is there a more straightforward way to enable all my controller services for a
process group that automatically handles the dependency issues I’m dealing with?
Thanks for any advice you can provide
-Tim
> On Feb 6, 2019, at 9:56 AM, Bryan Bende <[email protected]> wrote:
>
> Hi Tim,
>
> Thanks for the additional information. I was able to reproduce the
> issue with the services inside a nested versioned process group and it
> is definitely a bug, although I don't know where yet.
>
> I created this JIRA - https://issues.apache.org/jira/browse/NIFI-6001
>
> Thanks,
>
> Bryan
>
> On Wed, Feb 6, 2019 at 2:04 AM Tim Dean <[email protected]> wrote:
>>
>> I have continued to try isolating this problem. I’m not there yet, but I
>> have made some observations:
>>
>> I currently have a PG called ‘Project’ and, embedded within it, another PG
>> called `Subproject’.
>> Project has nothing in it other than Subproject. No processors, no
>> controller services, nothing.
>> Subproject has two controller services defined within it, one of which
>> references the other
>> Both Project and Subproject have been versioned in a local NiFi registry
>> If I import Subproject directly into a NiFi flow, its 2 controller services
>> are correctly instantiated. But if I import Project and then navigate down
>> into Subproject, I find that the depending service controller no longer has
>> a valid reference to the dependent service controller
>> Looking at the detailed IDs of controller services using the NiFi and NiFi
>> registry APIs directly, I have found that a successful import of Subproject
>> results in new controller service IDs being assigned to replace the IDs for
>> the service controllers within the NiFi registry, Those new controller
>> service IDs are correctly referenced in the appropriate component property
>> values.
>> Looking at the same structures using the NiFi and NiFi registry APIs, I have
>> found that an unsuccessful import of Project does result in new controller
>> service IDs being assigned, but that the references to those service
>> controller IDs from component property values have not been updated. Those
>> properties still reference the controller service using the ID that is
>> present in the NiFi registry’s copy of the flow.
>> Doing this with a couple of out-of-the-box controller services
>> (ElasticSearch client and SSL context) in a fresh flow did not exhibit this
>> problem, at least not in an obvious way. The problem is encountered when
>> using our own custom-built controller services. So either the problem is
>> something unique with our particular controller services, or something
>> unique has happened with our flows to put them in a bad state that causes
>> this problem.
>>
>>
>> Not sure if all of this helps anyone else to help with a workaround, but
>> figure more information might be helpful.
>>
>> -Tim
>>
>> On Feb 5, 2019, at 11:28 AM, Bryan Bende <[email protected]> wrote:
>>
>> I just did a quick test where I created a PG with Processor -> CS1 ->
>> CS2, saved that to registry and then imported another instance of it,
>> and it looks like CS1 is still referencing CS2.
>>
>> That doesn't mean there isn't an issue though. Let us know if you
>> figure out a specific scenario that creates a problem.
>>
>> On Tue, Feb 5, 2019 at 11:57 AM Tim Dean <[email protected]> wrote:
>>
>>
>> Thanks again Bryan
>>
>> For my current use case I think I can get away with duplicating controller
>> services so that each versioned PG only references its own controller
>> services.
>>
>> Unfortunately my initial changes to try this out haven’t worked completely
>> as you are describing. I do find that any processors that reference a
>> controller service are correctly configured after instantiating the
>> versioned PG. However, I have a couple of cases where one controller service
>> depends on another. And it appears that references from one service
>> controller to another (all within the same process group) are not being
>> correctly maintained when I instantiate via the API.
>>
>> I will dig deeper to find more specific details on what is happening, but at
>> a glance it doesn’t appear to be working for me. Has this use case been
>> tested before or am I running into a new issue?
>>
>> -Tim
>>
>> Sent from my iPad
>>
>> On Feb 5, 2019, at 8:41 AM, Bryan Bende <[email protected]> wrote:
>>
>> Tim,
>>
>> Unfortunately there isn't really a straight forward way to do this.
>> There are probably two things you can consider...
>>
>> 1) Any services that are part of the versioned process group will be
>> imported to the next environment and the components will have the
>> correct references to them. It is only when the versioned process
>> group references a service in a higher level that it is not included
>> and needs to be reset in the next environment. So there may be cases
>> where it is just easier to duplicate some services and make the
>> versioned process groups more portable.
>>
>> 2) For stuff where you do need to reset the reference, you might be
>> able to come up with a convention. For example, you go through the
>> newly imported PG and find all the components that reference service
>> ids that don't exist. For each one of these service ids you know the
>> type of service required based on the property, so you can then check
>> the parent PG for a service of the same type, and if only one exists,
>> then take that id.
>>
>> A lot of this stuff was originally meant to be done through the UI
>> which is why it is a bit tricky to script, but I would really like to
>> get all of this stuff built into the CLI.
>>
>> -Bryan
>>
>> On Tue, Feb 5, 2019 at 12:49 AM Tim Dean <[email protected]> wrote:
>>
>> Bryan,
>>
>> I’ve been working through this and I have it working - mostly. My flows have
>> been registered successfully, and I’ve been able to instantiate the parent
>> flow via the NiFi API to create a new process group.
>>
>> The final piece - I think - is figuring out how to properly configure the
>> controller services referenced by my flows using the APIs. I know I can use
>> the API to create a new controller service and generate an ID for each new
>> controller service. But how do I update all of my NiFi components that
>> reference the controller service by ID. Is there a straightforward way to do
>> this?
>>
>> Thanks
>>
>> -Tim
>>
>>
>> On Feb 1, 2019, at 11:09 AM, Bryan Bende <[email protected]> wrote:
>>
>> Tim,
>>
>> For moving between registries the approach you described sounds
>> correct, admittedly it would be nice if there was an easier way.
>>
>> In your case it is only two levels, but in general you'd have to start
>> at the lowest level, and work your way up the levels, applying the
>> correct ids from the level below.
>>
>> -Bryan
>>
>>
>> On Fri, Feb 1, 2019 at 11:44 AM Tim Dean <[email protected]> wrote:
>>
>> Thanks Bryan -
>>
>> If I use a nested versioned process group, it appears that the parent group
>> will reference its child process groups by ID. If I am populating my
>> registry in a new environment using the API, those IDs will be dynamically
>> generated as I make the API calls, correct?
>>
>> In that case, do I need to POST the child process groups first, get their
>> bucket and flow IDs back from the registry API, and then manipulate the JSON
>> for the parent process group to replace the contents of the
>> “versionedFlowCoordinates” JSON object’s identifiers with the new IDs?
>>
>> Or is there a better way to insert parent and child process groups via my
>> scripts?
>>
>> -Tim
>>
>> On Jan 31, 2019, at 6:25 PM, Bryan Bende <[email protected]> wrote:
>>
>> Hi Tim,
>>
>> I think the second option is the correct approach. The higher level
>> versioned PG is the way of saying that the lower level PGs work together as
>> a cohesive unit.
>>
>> -Bryan
>>
>> On Thu, Jan 31, 2019 at 7:00 PM Tim Dean <[email protected]> wrote:
>>
>> I am trying to automate deployment of a NiFi flow with several versioned
>> process groups using the NiFi APIs. The basic setup I have is this:
>>
>> I have a dozen or so process groups, each of which has been versioned within
>> a NiFi registry
>> My root process group contains each of those process groups, with various
>> connections between their ports as well as a few variable definitions and
>> controller service instances.
>>
>>
>> My goal is to deploy this flow, including the root process group that links
>> the versioned PGs as well as the versioned PGs themselves. So far, I’ve
>> managed to use the registry API to create a bucket and to add the versioned
>> flows into the registry. Now I’m trying to use the NiFi APIs to instantiate
>> the root PG and link together all the versioned PGs that I have just
>> inserted into the registry.
>>
>> The approach I have been trying is to capture my root PG as a template, and
>> then use the NiFi APIs to import and then instantiate that template. I have
>> gotten this much working, but unfortunately that leaves the PGs disconnected
>> from the versioned flows in the registry. I was hoping there was a way to
>> transform the template to insert the appropriate bucket and flow IDs but I
>> have been unable to figure out if this is possible.
>>
>> Alternatively, I suspect I could create an intermediate process group to
>> contain all my “real” PGs, and then version that intermediate PG. I could
>> then use the APIs to instantiate a new PG at the root level that is imported
>> from the intermediate PG. I suspect that this could work, but it is less
>> than ideal because I’m creating an artificial intermediate PG to contain all
>> of my real contents, which will be a distraction for users who come into the
>> NiFi data flow manager to monitor this process.
>>
>> Am I looking at this approach correctly? Are there other options I should be
>> considering?
>>
>> Thanks in advance,
>>
>> -Tim
>>
>>
>> --
>> Sent from Gmail Mobile
>>
>>
>>
>>
