Brian, So...
nifi-foo-service-impl-nar + nifi-foo-processors-nar ---depend on---> nifi-foo-service-api-nar ---depends on---> nifi-standard-services-api-nar That's what I should look for in the ES NARs? On Mon, Mar 26, 2018 at 10:31 PM, Bryan Bende <[email protected]> wrote: > Also, regarding the elastic search issue Mike mentioned… > > Usually when a processor can’t select the controller service at runtime it > is because theres an issue with the way the dependencies are setup between > NARs. There should generally be dependency paths like the following… > > nifi-foo-service-impl-nar ------| > |--> > nifi-foo-service-api-nar -> nifi-standard-services-api-nar > nifi-foo-processors-nar -------| > > All these links would be dependencies of <type>nar</type> in the poms. > > The above is for the case where processors and service implementation are > packaged separately, which is slightly different than what I was suggesting > for the Mongo case. > > > On Mar 26, 2018, at 10:06 PM, Bryan Bende <[email protected]> wrote: > > > > I’m a +1 for moving the Mongo stuff out of standard services. > > > > Controller service APIs should always be broken out into their own NAR, > but the processors and implementation can usually be bundled together. > > > > So something like the following would work: > > > > nifi-mongo-bundle > > - nifi-mongodb-services-api > > - nifi-mongodb-services-api-nar (would have NAR dependency on standard > services API NAR) > > - nifi-mongodb-processors > > - nifi-mongodb-nar (would have NAR dependency on > nifi-mongodb-services-api-nar) > > > > The processors module would have the processors and the client service > implementation. > > > > > >> On Mar 26, 2018, at 9:13 PM, Brian Ghigiarelli <[email protected]> > wrote: > >> > >> Thanks for the clarification, Mike. > >> > >> My primary motivation for this upgrade is to support an aggregation > >> pipeline with the $out stage. Support for this in the Java driver was > >> introduced in 3.4 via https://jira.mongodb.org/browse/JAVA-2254. Prior > to > >> this, I believe you had to use the cursor to query all of the results, > >> which suffers from both query performance issues and concurrency issues > for > >> the data flow if we're trying to overwrite an entire collection at once. > >> That said, if I'm missing something, I'm happy to learn! > >> > >> Thanks again, > >> Brian > >> > >> On Mon, Mar 26, 2018 at 7:57 PM Mike Thomsen <[email protected]> > wrote: > >> > >>> I was just following a convention at the time. So unless something > breaks > >>> because of the move, I don't see any reason why it would be an issue to > >>> move it. > >>> > >>> In fact, with NIFI-4325 the only reason I put the new ElasticSearch > >>> service/api in the standard-services segment of the code base was I ran > >>> into a bizarre issue at runtime where I could define the controller > >>> service, but the processor had no idea it was there. So that too might > >>> warrant some extra eyes now that #4325 was added to master today. > >>> > >>> FWIW, I can't think of any reason why the MongoDB client service needs > 3.4 > >>> or 3.6 over 3.2. > >>> > >>> Brian, feel free to ping me on the review. > >>> > >>> Thanks, > >>> > >>> Mike > >>> > >>> On Mon, Mar 26, 2018 at 6:59 PM, Matt Burgess <[email protected]> > >>> wrote: > >>> > >>>> Brian (this one's all you ;), > >>>> > >>>> I think that PR is quite welcome in my opinion, but I'd like to get > >>>> Mike Thomsen's and others' opinions on the subject too, I think Mike > >>>> wrote the original service (or at the least is very knowledgable about > >>>> Mongo and NiFi), he might have run into issues that led him to put it > >>>> there for a reason. If not, let's do it, if you're willing to submit > >>>> the contribution I/we'd be more than happy to review/incorporate it :) > >>>> > >>>> Regards, > >>>> Matt > >>>> > >>>> > >>>> On Mon, Mar 26, 2018 at 6:20 PM, Brian Ghigiarelli < > [email protected]> > >>>> wrote: > >>>>> Matt, > >>>>> > >>>>> +1 [non-binding] for the idea to move the Mongo dependencies into > that > >>>>> bundle. I think it will likely need the NAR dependency on > >>>>> nifi-standard-services-api in order to provide a link to the SSL > >>> Context > >>>>> Service. Is that ticket / PR worthy as a one-off in the short term? > No > >>>>> doubt the Extension Registry will be a powerful capability. > >>>>> > >>>>> Also, +2 for the greeting, though Bryan may choose to put the "y" > >>> first. > >>>> :-) > >>>>> > >>>>> Thanks, > >>>>> Brian > >>>>> > >>>>> On Mon, Mar 26, 2018 at 5:13 PM Matt Burgess <[email protected]> > >>>> wrote: > >>>>> > >>>>>> Bri/yan, > >>>>>> > >>>>>> IMO I think we'd be better off with moving the > >>>>>> nifi-mongodb-client-service-api and nifi-mongodb-services bundles > into > >>>>>> the nifi-mongodb-bundle. That's something I missed while reviewing > the > >>>>>> PR that put them in standard services. I don't see a direct > dependency > >>>>>> on nifi-standard-services, and if the nifi-mongodb-client-service- > api > >>>>>> needs the nifi-standard-services-api as a dependency, it can make > it a > >>>>>> NAR dependency. > >>>>>> > >>>>>> We might want to visit this as a much broader case, since I believe > we > >>>>>> could run into this with other services APIs (Elasticsearch, HBase, > >>>>>> etc.)? Certainly when the Extension Registry becomes a thing, we > might > >>>>>> not want "external" service APIs to be part of > >>>>>> nifi-standard-services-api. > >>>>>> > >>>>>> Regards, > >>>>>> Matt > >>>>>> > >>>>>> > >>>>>> On Mon, Mar 26, 2018 at 5:03 PM, Brian Ghigiarelli < > >>> [email protected] > >>>>> > >>>>>> wrote: > >>>>>>> Thanks, Bryan! Sure enough, we have nifi-standard-services-api-nar > >>> as > >>>> a > >>>>>>> parent to use the standard SSLContextService. Sounds like upgrading > >>>> the > >>>>>>> mongo-java-driver for the base build is the way to go for us. > >>>>>>> > >>>>>>> Thanks again, > >>>>>>> Brian > >>>>>>> On Mon, Mar 26, 2018 at 15:29 Bryan Bende <[email protected]> > wrote: > >>>>>>> > >>>>>>>> Brian, > >>>>>>>> > >>>>>>>> Is your custom processor using the MongoDBClientService provided > by > >>>>>>>> NiFI's standard services API? or does your NAR have a parent of > >>>>>>>> nifi-standard-services-api-nar to use other services? > >>>>>>>> > >>>>>>>> Looking at where the Mongo JARs are from a build of master... > >>>>>>>> > >>>>>>>> find work/nar/ -name *mongo-java*.jar > >>>>>>>> > >>>>>>>> > >>>>>> work/nar//extensions/nifi-standard-services-api-nar-1.6. > >>>> 0-SNAPSHOT.nar-unpacked/META-INF/bundled-dependencies/ > >>>> mongo-java-driver-3.2.2.jar > >>>>>>>> > >>>>>>>> > >>>>>> work/nar//extensions/nifi-mongodb-services-nar-1.6.0- > >>>> SNAPSHOT.nar-unpacked/META-INF/bundled-dependencies/ > >>>> mongo-java-driver-3.2.2.jar > >>>>>>>> > >>>>>>>> > >>>>>> work/nar//extensions/nifi-mongodb-nar-1.6.0-SNAPSHOT. > >>>> nar-unpacked/META-INF/bundled-dependencies/mongo-java- > driver-3.2.2.jar > >>>>>>>> > >>>>>>>> I think the issue is that if your NAR has > >>>>>>>> nifi-standard-services-api-nar as a parent NAR (which it probably > >>>> does > >>>>>>>> to use SSLContext service, or any other standard service) then you > >>>> are > >>>>>>>> getting mongo-java-driver-3.2.2 from MongoDBClientService since we > >>>>>>>> have parent first class loading. > >>>>>>>> > >>>>>>>> Given this situation I think there are only two options... not > >>> having > >>>>>>>> nifi-standard-services-api-nar as a parent (which stops you from > >>>> using > >>>>>>>> all standard services), or upgrading the version of > >>> mongo-java-driver > >>>>>>>> used by MongoDBClientService. > >>>>>>>> > >>>>>>>> -Bryan > >>>>>>>> > >>>>>>>> > >>>>>>>> On Mon, Mar 26, 2018 at 3:16 PM, Brian Ghigiarelli < > >>>> [email protected] > >>>>>>> > >>>>>>>> wrote: > >>>>>>>>> Hey all, > >>>>>>>>> > >>>>>>>>> Is there a means of troubleshooting a custom NAR that seems to be > >>>>>> having > >>>>>>>>> runtime conflicts by picking up an older JAR that's provided by > >>> the > >>>>>> NiFi > >>>>>>>>> standard bundle? > >>>>>>>>> > >>>>>>>>> In this particular case, I'm using some custom processors that > >>> are > >>>>>> built > >>>>>>>>> against NiFi 1.4.0 and have a dependency on MongoDB 3.6.3. At > >>>> runtime, > >>>>>>>> I'm > >>>>>>>>> seeing the processor use classes from MongoDB 3.2.2 that's > >>>> provided by > >>>>>>>>> NiFi. Both NiFi and the custom NAR compile successfully on their > >>>> own, > >>>>>> but > >>>>>>>>> using the custom NAR in NiFi causes NoSuchMethodError's due to a > >>>> new > >>>>>>>> method > >>>>>>>>> only available in MongoDB 3.4+. > >>>>>>>>> > >>>>>>>>> Thanks, > >>>>>>>>> Brian > >>>>>>>> > >>>>>> > >>>> > >>> > > > >
