Devs, For some reason NIFI is not working with some custom controller services I have written. I wrote new implementations of AWSCredentialsProviderService, that aim to work with session tokens. I am hoping to run NIFI from my local machine and to be able to connect with AWS using session tokens.
For both implementations I tried, it fails when I try to create them from the web UI. I created a PutS3Object processor, and configured the “AWS Credentials Provider Service” property. From that property I tried to create a new service and selected my custom implementation. When I click “create” the value for the credentials provider service is the ID of the controller service, not its name. While my controller services require several properties to be set, the web UI is not letting me set them. Usually I see an arrow next to the property, which allows me to configure a controller service, but I am not getting that now. I looked in the nifi-app logs, and I do not see any exception, I have even set the logging to TRACE for all things, and still don’t see any problem in the logs. The PutS3Object processor is not validating because the controller service is found to be invalid. I tried creating a unit test, it seems to work for me in tests, but I can’t use TestRunners because that is processor oriented, not meant for controller services. I have a suspicion that spring’s aspect oriented programming is somehow fuddling with my service. Does anybody know what I am doing wrong here? Other unrelated comments: 1. The first time you unpack NIFI it takes super long for it to start for me, like a half hour or more. I think you should make it easy for people to scale back their NIFI implementation. Really I would like to start it with just the minimum NAR files for it to start, and I can add others that I need. Maybe a sub-directory in lib for the essential nars could help people separate the essential stuff from the optional nars. The first time I tried installing it, I thought it was broken when really it just was taking forever (over 30 minutes). I think that new users will probably abandon NIFI if they can’t get it to start quickly out of the box. Maybe split the optional nars into an “extra-lib”, and people can move those into lib as necessary for their goals. 2. Building NIFI from source takes over an hour for me, really I just want to build the bare minimum things to get it to start. I tried creating maven profiles to have it build just the minimum pieces, but this proved to be non-trivial as maven does not seem to respect the “modules” tag in profiles, and the nifi-assembly project requires all of the optional nars to also be built. Creating this might be too complicated for me. Has anybody thought about supporting a quick/minimal build? 3. The “nifi-aws-processors” is challenging to use because in one project they have defined the interfaces for controller services (AWSCredentialsProviderService) and also included the services. I tried creating my own nar with an implementation of AWSCredentialsProviderService, but since it depended on “nifi-aws-processors”, my nar was also re-hosting all of the AWS processors. I was facing a lot of classpath issues because of this. I worked around this by using maven shade to shade in the “nifi-aws-processors” into my own jar, but excluding the services it provided. Then in my nar project I had to exclude the dependency on “nifi-aws-processors”. This was a lot of work on my part when all they needed to do was split that project into api, api-nar, impl, and impl-nar. 4. I think it is very confusing how there is a “Controller Services” for the entire NIFI canvas, and separate ones for individual processor groups. It seems that processors cannot use global controller services, and I am still uncertain about why I would ever create a global one. From Nifi settings, I would like to also see controller services in processor groups, and vice versa. From a processor, I would like to assign controller services that are global in scope, not limited to a processor group. I think this is something that will confuse and frustrate a lot of new developers, driving them to consider competing products. 5. I think the developer guide needs some clarification on what jars are provided and not. New developers will be unsure if they should include “nifi-api” as a provided or compile dependency, and same goes for nifi-framework-core. 6. Perhaps the maven-nar-plugin could let people tell NIFI to only use services listed under a certain set of bundled-dependencies. For example, my code depends on “nifi-aws-processors”, but I don’t want my nar to also host the services in that jar. From the MOJO you are able to exclude entire artifacts, but you can’t exclude the services within certain artifacts. This might be a problem to fix on the classloader side instead of from the maven plugin, I don’t know which is easier. Please help, Michael Knapp Capital One ________________________________________________________ The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates and may only be used solely in performance of work or services for Capital One. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
