Re: Class Loading Conflicts - Different JAR Versions
Mike, Here is a PR that should setup the ES stuff appropriately: https://github.com/apache/nifi/pull/2586 If we merge this for 1.7 then we should mention something in the migration notes, not that there is anything someone needs to do, but just for knowledge in case someone tries to do some interesting setup running 1.7 stuff in 1.6. -Bryan On Tue, Mar 27, 2018 at 9:37 AM, Otto Fowler wrote: > You can look at the aws nar for a sample of what I think Brian means. > > > > On March 27, 2018 at 07:31:54, Mike Thomsen (mikerthom...@gmail.com) wrote: > > Brian, > > So... > > nifi-foo-service-impl-nar + nifi-foo-processors-nar ---depend on---> > nifi-foo-service-api-nar > ---depends on---> nifi-standard-services-api-nar > > That's what I should look for in the ES NARs? > > On Mon, Mar 26, 2018 at 10:31 PM, Bryan Bende wrote: > >> Also, regarding the elastic search issue Mike mentioned… >> >> Usually when a processor can’t select the controller service at runtime > it >> is because theres an issue with the way the dependencies are setup > between >> NARs. There should generally be dependency paths like the following… >> >> nifi-foo-service-impl-nar --| >> |--> >> nifi-foo-service-api-nar -> nifi-standard-services-api-nar >> nifi-foo-processors-nar ---| >> >> All these links would be dependencies of nar in the poms. >> >> The above is for the case where processors and service implementation are >> packaged separately, which is slightly different than what I was > suggesting >> for the Mongo case. >> >> > On Mar 26, 2018, at 10:06 PM, Bryan Bende wrote: >> > >> > I’m a +1 for moving the Mongo stuff out of standard services. >> > >> > Controller service APIs should always be broken out into their own NAR, >> but the processors and implementation can usually be bundled together. >> > >> > So something like the following would work: >> > >> > nifi-mongo-bundle >> > - nifi-mongodb-services-api >> > - nifi-mongodb-services-api-nar (would have NAR dependency on standard >> services API NAR) >> > - nifi-mongodb-processors >> > - nifi-mongodb-nar (would have NAR dependency on >> nifi-mongodb-services-api-nar) >> > >> > The processors module would have the processors and the client service >> implementation. >> > >> > >> >> On Mar 26, 2018, at 9:13 PM, Brian Ghigiarelli >> wrote: >> >> >> >> Thanks for the clarification, Mike. >> >> >> >> My primary motivation for this upgrade is to support an aggregation >> >> pipeline with the $out stage. Support for this in the Java driver was >> >> introduced in 3.4 via https://jira.mongodb.org/browse/JAVA-2254. Prior >> to >> >> this, I believe you had to use the cursor to query all of the results, >> >> which suffers from both query performance issues and concurrency > issues >> for >> >> the data flow if we're trying to overwrite an entire collection at > once. >> >> That said, if I'm missing something, I'm happy to learn! >> >> >> >> Thanks again, >> >> Brian >> >> >> >> On Mon, Mar 26, 2018 at 7:57 PM Mike Thomsen >> wrote: >> >> >> >>> I was just following a convention at the time. So unless something >> breaks >> >>> because of the move, I don't see any reason why it would be an issue > to >> >>> move it. >> >>> >> >>> In fact, with NIFI-4325 the only reason I put the new ElasticSearch >> >>> service/api in the standard-services segment of the code base was I > ran >> >>> into a bizarre issue at runtime where I could define the controller >> >>> service, but the processor had no idea it was there. So that too > might >> >>> warrant some extra eyes now that #4325 was added to master today. >> >>> >> >>> FWIW, I can't think of any reason why the MongoDB client service > needs >> 3.4 >> >>> or 3.6 over 3.2. >> >>> >> >>> Brian, feel free to ping me on the review. >> >>> >> >>> Thanks, >> >>> >> >>> Mike >> >>> >> >>> On Mon, Mar 26, 2018 at 6:59 PM, Matt Burgess >> >>> wrote: >> >>> >> Brian (this one's all you ;), >> >> I think that PR is quite welcome in my opinion, but I'd like to get >> Mike Thomsen's and others' opinions on the subject too, I think Mike >> wrote the original service (or at the least is very knowledgable > about >> Mongo and NiFi), he might have run into issues that led him to put > it >> there for a reason. If not, let's do it, if you're willing to submit >> the contribution I/we'd be more than happy to review/incorporate it > :) >> >> Regards, >> Matt >> >> >> On Mon, Mar 26, 2018 at 6:20 PM, Brian Ghigiarelli < >> briang...@gmail.com> >> wrote: >> > Matt, >> > >> > +1 [non-binding] for the idea to move the Mongo dependencies into >> that >> > bundle. I think it will likely need the NAR dependency on >> > nifi-standard-services-api in order to provide a link to the SSL >> >>> Context >> > Service. Is that ticket / PR worthy as a one-off in the short term? >> No >> > doubt the Extension Registry will be a powerful capability. >> > >> >>>
Re: Class Loading Conflicts - Different JAR Versions
You can look at the aws nar for a sample of what I think Brian means. On March 27, 2018 at 07:31:54, Mike Thomsen (mikerthom...@gmail.com) wrote: Brian, So... nifi-foo-service-impl-nar + nifi-foo-processors-nar ---depend on---> nifi-foo-service-api-nar ---depends on---> nifi-standard-services-api-nar That's what I should look for in the ES NARs? On Mon, Mar 26, 2018 at 10:31 PM, Bryan Bende wrote: > Also, regarding the elastic search issue Mike mentioned… > > Usually when a processor can’t select the controller service at runtime it > is because theres an issue with the way the dependencies are setup between > NARs. There should generally be dependency paths like the following… > > nifi-foo-service-impl-nar --| > |--> > nifi-foo-service-api-nar -> nifi-standard-services-api-nar > nifi-foo-processors-nar ---| > > All these links would be dependencies of nar in the poms. > > The above is for the case where processors and service implementation are > packaged separately, which is slightly different than what I was suggesting > for the Mongo case. > > > On Mar 26, 2018, at 10:06 PM, Bryan Bende wrote: > > > > I’m a +1 for moving the Mongo stuff out of standard services. > > > > Controller service APIs should always be broken out into their own NAR, > but the processors and implementation can usually be bundled together. > > > > So something like the following would work: > > > > nifi-mongo-bundle > > - nifi-mongodb-services-api > > - nifi-mongodb-services-api-nar (would have NAR dependency on standard > services API NAR) > > - nifi-mongodb-processors > > - nifi-mongodb-nar (would have NAR dependency on > nifi-mongodb-services-api-nar) > > > > The processors module would have the processors and the client service > implementation. > > > > > >> On Mar 26, 2018, at 9:13 PM, Brian Ghigiarelli > wrote: > >> > >> Thanks for the clarification, Mike. > >> > >> My primary motivation for this upgrade is to support an aggregation > >> pipeline with the $out stage. Support for this in the Java driver was > >> introduced in 3.4 via https://jira.mongodb.org/browse/JAVA-2254. Prior > to > >> this, I believe you had to use the cursor to query all of the results, > >> which suffers from both query performance issues and concurrency issues > for > >> the data flow if we're trying to overwrite an entire collection at once. > >> That said, if I'm missing something, I'm happy to learn! > >> > >> Thanks again, > >> Brian > >> > >> On Mon, Mar 26, 2018 at 7:57 PM Mike Thomsen > wrote: > >> > >>> I was just following a convention at the time. So unless something > breaks > >>> because of the move, I don't see any reason why it would be an issue to > >>> move it. > >>> > >>> In fact, with NIFI-4325 the only reason I put the new ElasticSearch > >>> service/api in the standard-services segment of the code base was I ran > >>> into a bizarre issue at runtime where I could define the controller > >>> service, but the processor had no idea it was there. So that too might > >>> warrant some extra eyes now that #4325 was added to master today. > >>> > >>> FWIW, I can't think of any reason why the MongoDB client service needs > 3.4 > >>> or 3.6 over 3.2. > >>> > >>> Brian, feel free to ping me on the review. > >>> > >>> Thanks, > >>> > >>> Mike > >>> > >>> On Mon, Mar 26, 2018 at 6:59 PM, Matt Burgess > >>> wrote: > >>> > Brian (this one's all you ;), > > I think that PR is quite welcome in my opinion, but I'd like to get > Mike Thomsen's and others' opinions on the subject too, I think Mike > wrote the original service (or at the least is very knowledgable about > Mongo and NiFi), he might have run into issues that led him to put it > there for a reason. If not, let's do it, if you're willing to submit > the contribution I/we'd be more than happy to review/incorporate it :) > > Regards, > Matt > > > On Mon, Mar 26, 2018 at 6:20 PM, Brian Ghigiarelli < > briang...@gmail.com> > wrote: > > Matt, > > > > +1 [non-binding] for the idea to move the Mongo dependencies into > that > > bundle. I think it will likely need the NAR dependency on > > nifi-standard-services-api in order to provide a link to the SSL > >>> Context > > Service. Is that ticket / PR worthy as a one-off in the short term? > No > > doubt the Extension Registry will be a powerful capability. > > > > Also, +2 for the greeting, though Bryan may choose to put the "y" > >>> first. > :-) > > > > Thanks, > > Brian > > > > On Mon, Mar 26, 2018 at 5:13 PM Matt Burgess > wrote: > > > >> Bri/yan, > >> > >> IMO I think we'd be better off with moving the > >> nifi-mongodb-client-service-api and nifi-mongodb-services bundles > into > >> the nifi-mongodb-bundle. That's something I missed while reviewing > the > >> PR that put them in standard services. I don't see a direct > dependency > >> on nifi-sta
Re: Class Loading Conflicts - Different JAR Versions
Brian, So... nifi-foo-service-impl-nar + nifi-foo-processors-nar ---depend on---> nifi-foo-service-api-nar ---depends on---> nifi-standard-services-api-nar That's what I should look for in the ES NARs? On Mon, Mar 26, 2018 at 10:31 PM, Bryan Bende wrote: > Also, regarding the elastic search issue Mike mentioned… > > Usually when a processor can’t select the controller service at runtime it > is because theres an issue with the way the dependencies are setup between > NARs. There should generally be dependency paths like the following… > > nifi-foo-service-impl-nar --| > |--> > nifi-foo-service-api-nar -> nifi-standard-services-api-nar > nifi-foo-processors-nar ---| > > All these links would be dependencies of nar in the poms. > > The above is for the case where processors and service implementation are > packaged separately, which is slightly different than what I was suggesting > for the Mongo case. > > > On Mar 26, 2018, at 10:06 PM, Bryan Bende wrote: > > > > I’m a +1 for moving the Mongo stuff out of standard services. > > > > Controller service APIs should always be broken out into their own NAR, > but the processors and implementation can usually be bundled together. > > > > So something like the following would work: > > > > nifi-mongo-bundle > > - nifi-mongodb-services-api > > - nifi-mongodb-services-api-nar (would have NAR dependency on standard > services API NAR) > > - nifi-mongodb-processors > > - nifi-mongodb-nar (would have NAR dependency on > nifi-mongodb-services-api-nar) > > > > The processors module would have the processors and the client service > implementation. > > > > > >> On Mar 26, 2018, at 9:13 PM, Brian Ghigiarelli > wrote: > >> > >> Thanks for the clarification, Mike. > >> > >> My primary motivation for this upgrade is to support an aggregation > >> pipeline with the $out stage. Support for this in the Java driver was > >> introduced in 3.4 via https://jira.mongodb.org/browse/JAVA-2254. Prior > to > >> this, I believe you had to use the cursor to query all of the results, > >> which suffers from both query performance issues and concurrency issues > for > >> the data flow if we're trying to overwrite an entire collection at once. > >> That said, if I'm missing something, I'm happy to learn! > >> > >> Thanks again, > >> Brian > >> > >> On Mon, Mar 26, 2018 at 7:57 PM Mike Thomsen > wrote: > >> > >>> I was just following a convention at the time. So unless something > breaks > >>> because of the move, I don't see any reason why it would be an issue to > >>> move it. > >>> > >>> In fact, with NIFI-4325 the only reason I put the new ElasticSearch > >>> service/api in the standard-services segment of the code base was I ran > >>> into a bizarre issue at runtime where I could define the controller > >>> service, but the processor had no idea it was there. So that too might > >>> warrant some extra eyes now that #4325 was added to master today. > >>> > >>> FWIW, I can't think of any reason why the MongoDB client service needs > 3.4 > >>> or 3.6 over 3.2. > >>> > >>> Brian, feel free to ping me on the review. > >>> > >>> Thanks, > >>> > >>> Mike > >>> > >>> On Mon, Mar 26, 2018 at 6:59 PM, Matt Burgess > >>> wrote: > >>> > Brian (this one's all you ;), > > I think that PR is quite welcome in my opinion, but I'd like to get > Mike Thomsen's and others' opinions on the subject too, I think Mike > wrote the original service (or at the least is very knowledgable about > Mongo and NiFi), he might have run into issues that led him to put it > there for a reason. If not, let's do it, if you're willing to submit > the contribution I/we'd be more than happy to review/incorporate it :) > > Regards, > Matt > > > On Mon, Mar 26, 2018 at 6:20 PM, Brian Ghigiarelli < > briang...@gmail.com> > wrote: > > Matt, > > > > +1 [non-binding] for the idea to move the Mongo dependencies into > that > > bundle. I think it will likely need the NAR dependency on > > nifi-standard-services-api in order to provide a link to the SSL > >>> Context > > Service. Is that ticket / PR worthy as a one-off in the short term? > No > > doubt the Extension Registry will be a powerful capability. > > > > Also, +2 for the greeting, though Bryan may choose to put the "y" > >>> first. > :-) > > > > Thanks, > > Brian > > > > On Mon, Mar 26, 2018 at 5:13 PM Matt Burgess > wrote: > > > >> Bri/yan, > >> > >> IMO I think we'd be better off with moving the > >> nifi-mongodb-client-service-api and nifi-mongodb-services bundles > into > >> the nifi-mongodb-bundle. That's something I missed while reviewing > the > >> PR that put them in standard services. I don't see a direct > dependency > >> on nifi-standard-services, and if the nifi-mongodb-client-service- > api > >> needs the nifi-standard-services-
Re: Class Loading Conflicts - Different JAR Versions
Also, regarding the elastic search issue Mike mentioned… Usually when a processor can’t select the controller service at runtime it is because theres an issue with the way the dependencies are setup between NARs. There should generally be dependency paths like the following… nifi-foo-service-impl-nar --| |--> nifi-foo-service-api-nar -> nifi-standard-services-api-nar nifi-foo-processors-nar ---| All these links would be dependencies of nar in the poms. The above is for the case where processors and service implementation are packaged separately, which is slightly different than what I was suggesting for the Mongo case. > On Mar 26, 2018, at 10:06 PM, Bryan Bende wrote: > > I’m a +1 for moving the Mongo stuff out of standard services. > > Controller service APIs should always be broken out into their own NAR, but > the processors and implementation can usually be bundled together. > > So something like the following would work: > > nifi-mongo-bundle > - nifi-mongodb-services-api > - nifi-mongodb-services-api-nar (would have NAR dependency on standard > services API NAR) > - nifi-mongodb-processors > - nifi-mongodb-nar (would have NAR dependency on > nifi-mongodb-services-api-nar) > > The processors module would have the processors and the client service > implementation. > > >> On Mar 26, 2018, at 9:13 PM, Brian Ghigiarelli wrote: >> >> Thanks for the clarification, Mike. >> >> My primary motivation for this upgrade is to support an aggregation >> pipeline with the $out stage. Support for this in the Java driver was >> introduced in 3.4 via https://jira.mongodb.org/browse/JAVA-2254. Prior to >> this, I believe you had to use the cursor to query all of the results, >> which suffers from both query performance issues and concurrency issues for >> the data flow if we're trying to overwrite an entire collection at once. >> That said, if I'm missing something, I'm happy to learn! >> >> Thanks again, >> Brian >> >> On Mon, Mar 26, 2018 at 7:57 PM Mike Thomsen wrote: >> >>> I was just following a convention at the time. So unless something breaks >>> because of the move, I don't see any reason why it would be an issue to >>> move it. >>> >>> In fact, with NIFI-4325 the only reason I put the new ElasticSearch >>> service/api in the standard-services segment of the code base was I ran >>> into a bizarre issue at runtime where I could define the controller >>> service, but the processor had no idea it was there. So that too might >>> warrant some extra eyes now that #4325 was added to master today. >>> >>> FWIW, I can't think of any reason why the MongoDB client service needs 3.4 >>> or 3.6 over 3.2. >>> >>> Brian, feel free to ping me on the review. >>> >>> Thanks, >>> >>> Mike >>> >>> On Mon, Mar 26, 2018 at 6:59 PM, Matt Burgess >>> wrote: >>> Brian (this one's all you ;), I think that PR is quite welcome in my opinion, but I'd like to get Mike Thomsen's and others' opinions on the subject too, I think Mike wrote the original service (or at the least is very knowledgable about Mongo and NiFi), he might have run into issues that led him to put it there for a reason. If not, let's do it, if you're willing to submit the contribution I/we'd be more than happy to review/incorporate it :) Regards, Matt On Mon, Mar 26, 2018 at 6:20 PM, Brian Ghigiarelli wrote: > Matt, > > +1 [non-binding] for the idea to move the Mongo dependencies into that > bundle. I think it will likely need the NAR dependency on > nifi-standard-services-api in order to provide a link to the SSL >>> Context > Service. Is that ticket / PR worthy as a one-off in the short term? No > doubt the Extension Registry will be a powerful capability. > > Also, +2 for the greeting, though Bryan may choose to put the "y" >>> first. :-) > > Thanks, > Brian > > On Mon, Mar 26, 2018 at 5:13 PM Matt Burgess wrote: > >> Bri/yan, >> >> IMO I think we'd be better off with moving the >> nifi-mongodb-client-service-api and nifi-mongodb-services bundles into >> the nifi-mongodb-bundle. That's something I missed while reviewing the >> PR that put them in standard services. I don't see a direct dependency >> on nifi-standard-services, and if the nifi-mongodb-client-service-api >> needs the nifi-standard-services-api as a dependency, it can make it a >> NAR dependency. >> >> We might want to visit this as a much broader case, since I believe we >> could run into this with other services APIs (Elasticsearch, HBase, >> etc.)? Certainly when the Extension Registry becomes a thing, we might >> not want "external" service APIs to be part of >> nifi-standard-services-api. >> >> Regards, >> Matt >> >> >> On Mon, Mar 26, 2018 at 5:03 PM, Brian Ghigiare
Re: Class Loading Conflicts - Different JAR Versions
I’m a +1 for moving the Mongo stuff out of standard services. Controller service APIs should always be broken out into their own NAR, but the processors and implementation can usually be bundled together. So something like the following would work: nifi-mongo-bundle - nifi-mongodb-services-api - nifi-mongodb-services-api-nar (would have NAR dependency on standard services API NAR) - nifi-mongodb-processors - nifi-mongodb-nar (would have NAR dependency on nifi-mongodb-services-api-nar) The processors module would have the processors and the client service implementation. > On Mar 26, 2018, at 9:13 PM, Brian Ghigiarelli wrote: > > Thanks for the clarification, Mike. > > My primary motivation for this upgrade is to support an aggregation > pipeline with the $out stage. Support for this in the Java driver was > introduced in 3.4 via https://jira.mongodb.org/browse/JAVA-2254. Prior to > this, I believe you had to use the cursor to query all of the results, > which suffers from both query performance issues and concurrency issues for > the data flow if we're trying to overwrite an entire collection at once. > That said, if I'm missing something, I'm happy to learn! > > Thanks again, > Brian > > On Mon, Mar 26, 2018 at 7:57 PM Mike Thomsen wrote: > >> I was just following a convention at the time. So unless something breaks >> because of the move, I don't see any reason why it would be an issue to >> move it. >> >> In fact, with NIFI-4325 the only reason I put the new ElasticSearch >> service/api in the standard-services segment of the code base was I ran >> into a bizarre issue at runtime where I could define the controller >> service, but the processor had no idea it was there. So that too might >> warrant some extra eyes now that #4325 was added to master today. >> >> FWIW, I can't think of any reason why the MongoDB client service needs 3.4 >> or 3.6 over 3.2. >> >> Brian, feel free to ping me on the review. >> >> Thanks, >> >> Mike >> >> On Mon, Mar 26, 2018 at 6:59 PM, Matt Burgess >> wrote: >> >>> Brian (this one's all you ;), >>> >>> I think that PR is quite welcome in my opinion, but I'd like to get >>> Mike Thomsen's and others' opinions on the subject too, I think Mike >>> wrote the original service (or at the least is very knowledgable about >>> Mongo and NiFi), he might have run into issues that led him to put it >>> there for a reason. If not, let's do it, if you're willing to submit >>> the contribution I/we'd be more than happy to review/incorporate it :) >>> >>> Regards, >>> Matt >>> >>> >>> On Mon, Mar 26, 2018 at 6:20 PM, Brian Ghigiarelli >>> wrote: Matt, +1 [non-binding] for the idea to move the Mongo dependencies into that bundle. I think it will likely need the NAR dependency on nifi-standard-services-api in order to provide a link to the SSL >> Context Service. Is that ticket / PR worthy as a one-off in the short term? No doubt the Extension Registry will be a powerful capability. Also, +2 for the greeting, though Bryan may choose to put the "y" >> first. >>> :-) Thanks, Brian On Mon, Mar 26, 2018 at 5:13 PM Matt Burgess >>> wrote: > Bri/yan, > > IMO I think we'd be better off with moving the > nifi-mongodb-client-service-api and nifi-mongodb-services bundles into > the nifi-mongodb-bundle. That's something I missed while reviewing the > PR that put them in standard services. I don't see a direct dependency > on nifi-standard-services, and if the nifi-mongodb-client-service-api > needs the nifi-standard-services-api as a dependency, it can make it a > NAR dependency. > > We might want to visit this as a much broader case, since I believe we > could run into this with other services APIs (Elasticsearch, HBase, > etc.)? Certainly when the Extension Registry becomes a thing, we might > not want "external" service APIs to be part of > nifi-standard-services-api. > > Regards, > Matt > > > On Mon, Mar 26, 2018 at 5:03 PM, Brian Ghigiarelli < >> briang...@gmail.com > wrote: >> Thanks, Bryan! Sure enough, we have nifi-standard-services-api-nar >> as >>> a >> parent to use the standard SSLContextService. Sounds like upgrading >>> the >> mongo-java-driver for the base build is the way to go for us. >> >> Thanks again, >> Brian >> On Mon, Mar 26, 2018 at 15:29 Bryan Bende wrote: >> >>> Brian, >>> >>> Is your custom processor using the MongoDBClientService provided by >>> NiFI's standard services API? or does your NAR have a parent of >>> nifi-standard-services-api-nar to use other services? >>> >>> Looking at where the Mongo JARs are from a build of master... >>> >>> find work/nar/ -name *mongo-java*.jar >>> >>> > work/nar//extensions/nifi-standard-services-api-nar-1.6. >>> 0-SNAPSHOT.nar-unpacked/META-
Re: Class Loading Conflicts - Different JAR Versions
Thanks for the clarification, Mike. My primary motivation for this upgrade is to support an aggregation pipeline with the $out stage. Support for this in the Java driver was introduced in 3.4 via https://jira.mongodb.org/browse/JAVA-2254. Prior to this, I believe you had to use the cursor to query all of the results, which suffers from both query performance issues and concurrency issues for the data flow if we're trying to overwrite an entire collection at once. That said, if I'm missing something, I'm happy to learn! Thanks again, Brian On Mon, Mar 26, 2018 at 7:57 PM Mike Thomsen wrote: > I was just following a convention at the time. So unless something breaks > because of the move, I don't see any reason why it would be an issue to > move it. > > In fact, with NIFI-4325 the only reason I put the new ElasticSearch > service/api in the standard-services segment of the code base was I ran > into a bizarre issue at runtime where I could define the controller > service, but the processor had no idea it was there. So that too might > warrant some extra eyes now that #4325 was added to master today. > > FWIW, I can't think of any reason why the MongoDB client service needs 3.4 > or 3.6 over 3.2. > > Brian, feel free to ping me on the review. > > Thanks, > > Mike > > On Mon, Mar 26, 2018 at 6:59 PM, Matt Burgess > wrote: > > > Brian (this one's all you ;), > > > > I think that PR is quite welcome in my opinion, but I'd like to get > > Mike Thomsen's and others' opinions on the subject too, I think Mike > > wrote the original service (or at the least is very knowledgable about > > Mongo and NiFi), he might have run into issues that led him to put it > > there for a reason. If not, let's do it, if you're willing to submit > > the contribution I/we'd be more than happy to review/incorporate it :) > > > > Regards, > > Matt > > > > > > On Mon, Mar 26, 2018 at 6:20 PM, Brian Ghigiarelli > > wrote: > > > Matt, > > > > > > +1 [non-binding] for the idea to move the Mongo dependencies into that > > > bundle. I think it will likely need the NAR dependency on > > > nifi-standard-services-api in order to provide a link to the SSL > Context > > > Service. Is that ticket / PR worthy as a one-off in the short term? No > > > doubt the Extension Registry will be a powerful capability. > > > > > > Also, +2 for the greeting, though Bryan may choose to put the "y" > first. > > :-) > > > > > > Thanks, > > > Brian > > > > > > On Mon, Mar 26, 2018 at 5:13 PM Matt Burgess > > wrote: > > > > > >> Bri/yan, > > >> > > >> IMO I think we'd be better off with moving the > > >> nifi-mongodb-client-service-api and nifi-mongodb-services bundles into > > >> the nifi-mongodb-bundle. That's something I missed while reviewing the > > >> PR that put them in standard services. I don't see a direct dependency > > >> on nifi-standard-services, and if the nifi-mongodb-client-service-api > > >> needs the nifi-standard-services-api as a dependency, it can make it a > > >> NAR dependency. > > >> > > >> We might want to visit this as a much broader case, since I believe we > > >> could run into this with other services APIs (Elasticsearch, HBase, > > >> etc.)? Certainly when the Extension Registry becomes a thing, we might > > >> not want "external" service APIs to be part of > > >> nifi-standard-services-api. > > >> > > >> Regards, > > >> Matt > > >> > > >> > > >> On Mon, Mar 26, 2018 at 5:03 PM, Brian Ghigiarelli < > briang...@gmail.com > > > > > >> wrote: > > >> > Thanks, Bryan! Sure enough, we have nifi-standard-services-api-nar > as > > a > > >> > parent to use the standard SSLContextService. Sounds like upgrading > > the > > >> > mongo-java-driver for the base build is the way to go for us. > > >> > > > >> > Thanks again, > > >> > Brian > > >> > On Mon, Mar 26, 2018 at 15:29 Bryan Bende wrote: > > >> > > > >> >> Brian, > > >> >> > > >> >> Is your custom processor using the MongoDBClientService provided by > > >> >> NiFI's standard services API? or does your NAR have a parent of > > >> >> nifi-standard-services-api-nar to use other services? > > >> >> > > >> >> Looking at where the Mongo JARs are from a build of master... > > >> >> > > >> >> find work/nar/ -name *mongo-java*.jar > > >> >> > > >> >> > > >> work/nar//extensions/nifi-standard-services-api-nar-1.6. > > 0-SNAPSHOT.nar-unpacked/META-INF/bundled-dependencies/ > > mongo-java-driver-3.2.2.jar > > >> >> > > >> >> > > >> work/nar//extensions/nifi-mongodb-services-nar-1.6.0- > > SNAPSHOT.nar-unpacked/META-INF/bundled-dependencies/ > > mongo-java-driver-3.2.2.jar > > >> >> > > >> >> > > >> work/nar//extensions/nifi-mongodb-nar-1.6.0-SNAPSHOT. > > nar-unpacked/META-INF/bundled-dependencies/mongo-java-driver-3.2.2.jar > > >> >> > > >> >> I think the issue is that if your NAR has > > >> >> nifi-standard-services-api-nar as a parent NAR (which it probably > > does > > >> >> to use SSLContext service, or any other standard service) then you > > are > > >> >> getting mongo-java-driver-3.2.2 fr
Re: Class Loading Conflicts - Different JAR Versions
I was just following a convention at the time. So unless something breaks because of the move, I don't see any reason why it would be an issue to move it. In fact, with NIFI-4325 the only reason I put the new ElasticSearch service/api in the standard-services segment of the code base was I ran into a bizarre issue at runtime where I could define the controller service, but the processor had no idea it was there. So that too might warrant some extra eyes now that #4325 was added to master today. FWIW, I can't think of any reason why the MongoDB client service needs 3.4 or 3.6 over 3.2. Brian, feel free to ping me on the review. Thanks, Mike On Mon, Mar 26, 2018 at 6:59 PM, Matt Burgess wrote: > Brian (this one's all you ;), > > I think that PR is quite welcome in my opinion, but I'd like to get > Mike Thomsen's and others' opinions on the subject too, I think Mike > wrote the original service (or at the least is very knowledgable about > Mongo and NiFi), he might have run into issues that led him to put it > there for a reason. If not, let's do it, if you're willing to submit > the contribution I/we'd be more than happy to review/incorporate it :) > > Regards, > Matt > > > On Mon, Mar 26, 2018 at 6:20 PM, Brian Ghigiarelli > wrote: > > Matt, > > > > +1 [non-binding] for the idea to move the Mongo dependencies into that > > bundle. I think it will likely need the NAR dependency on > > nifi-standard-services-api in order to provide a link to the SSL Context > > Service. Is that ticket / PR worthy as a one-off in the short term? No > > doubt the Extension Registry will be a powerful capability. > > > > Also, +2 for the greeting, though Bryan may choose to put the "y" first. > :-) > > > > Thanks, > > Brian > > > > On Mon, Mar 26, 2018 at 5:13 PM Matt Burgess > wrote: > > > >> Bri/yan, > >> > >> IMO I think we'd be better off with moving the > >> nifi-mongodb-client-service-api and nifi-mongodb-services bundles into > >> the nifi-mongodb-bundle. That's something I missed while reviewing the > >> PR that put them in standard services. I don't see a direct dependency > >> on nifi-standard-services, and if the nifi-mongodb-client-service-api > >> needs the nifi-standard-services-api as a dependency, it can make it a > >> NAR dependency. > >> > >> We might want to visit this as a much broader case, since I believe we > >> could run into this with other services APIs (Elasticsearch, HBase, > >> etc.)? Certainly when the Extension Registry becomes a thing, we might > >> not want "external" service APIs to be part of > >> nifi-standard-services-api. > >> > >> Regards, > >> Matt > >> > >> > >> On Mon, Mar 26, 2018 at 5:03 PM, Brian Ghigiarelli > > >> wrote: > >> > Thanks, Bryan! Sure enough, we have nifi-standard-services-api-nar as > a > >> > parent to use the standard SSLContextService. Sounds like upgrading > the > >> > mongo-java-driver for the base build is the way to go for us. > >> > > >> > Thanks again, > >> > Brian > >> > On Mon, Mar 26, 2018 at 15:29 Bryan Bende wrote: > >> > > >> >> Brian, > >> >> > >> >> Is your custom processor using the MongoDBClientService provided by > >> >> NiFI's standard services API? or does your NAR have a parent of > >> >> nifi-standard-services-api-nar to use other services? > >> >> > >> >> Looking at where the Mongo JARs are from a build of master... > >> >> > >> >> find work/nar/ -name *mongo-java*.jar > >> >> > >> >> > >> work/nar//extensions/nifi-standard-services-api-nar-1.6. > 0-SNAPSHOT.nar-unpacked/META-INF/bundled-dependencies/ > mongo-java-driver-3.2.2.jar > >> >> > >> >> > >> work/nar//extensions/nifi-mongodb-services-nar-1.6.0- > SNAPSHOT.nar-unpacked/META-INF/bundled-dependencies/ > mongo-java-driver-3.2.2.jar > >> >> > >> >> > >> work/nar//extensions/nifi-mongodb-nar-1.6.0-SNAPSHOT. > nar-unpacked/META-INF/bundled-dependencies/mongo-java-driver-3.2.2.jar > >> >> > >> >> I think the issue is that if your NAR has > >> >> nifi-standard-services-api-nar as a parent NAR (which it probably > does > >> >> to use SSLContext service, or any other standard service) then you > are > >> >> getting mongo-java-driver-3.2.2 from MongoDBClientService since we > >> >> have parent first class loading. > >> >> > >> >> Given this situation I think there are only two options... not having > >> >> nifi-standard-services-api-nar as a parent (which stops you from > using > >> >> all standard services), or upgrading the version of mongo-java-driver > >> >> used by MongoDBClientService. > >> >> > >> >> -Bryan > >> >> > >> >> > >> >> On Mon, Mar 26, 2018 at 3:16 PM, Brian Ghigiarelli < > briang...@gmail.com > >> > > >> >> wrote: > >> >> > Hey all, > >> >> > > >> >> > Is there a means of troubleshooting a custom NAR that seems to be > >> having > >> >> > runtime conflicts by picking up an older JAR that's provided by the > >> NiFi > >> >> > standard bundle? > >> >> > > >> >> > In this particular case, I'm using some custom processors that are > >> built > >> >> > against NiFi 1.4.0
Re: Class Loading Conflicts - Different JAR Versions
Brian (this one's all you ;), I think that PR is quite welcome in my opinion, but I'd like to get Mike Thomsen's and others' opinions on the subject too, I think Mike wrote the original service (or at the least is very knowledgable about Mongo and NiFi), he might have run into issues that led him to put it there for a reason. If not, let's do it, if you're willing to submit the contribution I/we'd be more than happy to review/incorporate it :) Regards, Matt On Mon, Mar 26, 2018 at 6:20 PM, Brian Ghigiarelli wrote: > Matt, > > +1 [non-binding] for the idea to move the Mongo dependencies into that > bundle. I think it will likely need the NAR dependency on > nifi-standard-services-api in order to provide a link to the SSL Context > Service. Is that ticket / PR worthy as a one-off in the short term? No > doubt the Extension Registry will be a powerful capability. > > Also, +2 for the greeting, though Bryan may choose to put the "y" first. :-) > > Thanks, > Brian > > On Mon, Mar 26, 2018 at 5:13 PM Matt Burgess wrote: > >> Bri/yan, >> >> IMO I think we'd be better off with moving the >> nifi-mongodb-client-service-api and nifi-mongodb-services bundles into >> the nifi-mongodb-bundle. That's something I missed while reviewing the >> PR that put them in standard services. I don't see a direct dependency >> on nifi-standard-services, and if the nifi-mongodb-client-service-api >> needs the nifi-standard-services-api as a dependency, it can make it a >> NAR dependency. >> >> We might want to visit this as a much broader case, since I believe we >> could run into this with other services APIs (Elasticsearch, HBase, >> etc.)? Certainly when the Extension Registry becomes a thing, we might >> not want "external" service APIs to be part of >> nifi-standard-services-api. >> >> Regards, >> Matt >> >> >> On Mon, Mar 26, 2018 at 5:03 PM, Brian Ghigiarelli >> wrote: >> > Thanks, Bryan! Sure enough, we have nifi-standard-services-api-nar as a >> > parent to use the standard SSLContextService. Sounds like upgrading the >> > mongo-java-driver for the base build is the way to go for us. >> > >> > Thanks again, >> > Brian >> > On Mon, Mar 26, 2018 at 15:29 Bryan Bende wrote: >> > >> >> Brian, >> >> >> >> Is your custom processor using the MongoDBClientService provided by >> >> NiFI's standard services API? or does your NAR have a parent of >> >> nifi-standard-services-api-nar to use other services? >> >> >> >> Looking at where the Mongo JARs are from a build of master... >> >> >> >> find work/nar/ -name *mongo-java*.jar >> >> >> >> >> work/nar//extensions/nifi-standard-services-api-nar-1.6.0-SNAPSHOT.nar-unpacked/META-INF/bundled-dependencies/mongo-java-driver-3.2.2.jar >> >> >> >> >> work/nar//extensions/nifi-mongodb-services-nar-1.6.0-SNAPSHOT.nar-unpacked/META-INF/bundled-dependencies/mongo-java-driver-3.2.2.jar >> >> >> >> >> work/nar//extensions/nifi-mongodb-nar-1.6.0-SNAPSHOT.nar-unpacked/META-INF/bundled-dependencies/mongo-java-driver-3.2.2.jar >> >> >> >> I think the issue is that if your NAR has >> >> nifi-standard-services-api-nar as a parent NAR (which it probably does >> >> to use SSLContext service, or any other standard service) then you are >> >> getting mongo-java-driver-3.2.2 from MongoDBClientService since we >> >> have parent first class loading. >> >> >> >> Given this situation I think there are only two options... not having >> >> nifi-standard-services-api-nar as a parent (which stops you from using >> >> all standard services), or upgrading the version of mongo-java-driver >> >> used by MongoDBClientService. >> >> >> >> -Bryan >> >> >> >> >> >> On Mon, Mar 26, 2018 at 3:16 PM, Brian Ghigiarelli > > >> >> wrote: >> >> > Hey all, >> >> > >> >> > Is there a means of troubleshooting a custom NAR that seems to be >> having >> >> > runtime conflicts by picking up an older JAR that's provided by the >> NiFi >> >> > standard bundle? >> >> > >> >> > In this particular case, I'm using some custom processors that are >> built >> >> > against NiFi 1.4.0 and have a dependency on MongoDB 3.6.3. At runtime, >> >> I'm >> >> > seeing the processor use classes from MongoDB 3.2.2 that's provided by >> >> > NiFi. Both NiFi and the custom NAR compile successfully on their own, >> but >> >> > using the custom NAR in NiFi causes NoSuchMethodError's due to a new >> >> method >> >> > only available in MongoDB 3.4+. >> >> > >> >> > Thanks, >> >> > Brian >> >> >>
Re: Class Loading Conflicts - Different JAR Versions
Matt, +1 [non-binding] for the idea to move the Mongo dependencies into that bundle. I think it will likely need the NAR dependency on nifi-standard-services-api in order to provide a link to the SSL Context Service. Is that ticket / PR worthy as a one-off in the short term? No doubt the Extension Registry will be a powerful capability. Also, +2 for the greeting, though Bryan may choose to put the "y" first. :-) Thanks, Brian On Mon, Mar 26, 2018 at 5:13 PM Matt Burgess wrote: > Bri/yan, > > IMO I think we'd be better off with moving the > nifi-mongodb-client-service-api and nifi-mongodb-services bundles into > the nifi-mongodb-bundle. That's something I missed while reviewing the > PR that put them in standard services. I don't see a direct dependency > on nifi-standard-services, and if the nifi-mongodb-client-service-api > needs the nifi-standard-services-api as a dependency, it can make it a > NAR dependency. > > We might want to visit this as a much broader case, since I believe we > could run into this with other services APIs (Elasticsearch, HBase, > etc.)? Certainly when the Extension Registry becomes a thing, we might > not want "external" service APIs to be part of > nifi-standard-services-api. > > Regards, > Matt > > > On Mon, Mar 26, 2018 at 5:03 PM, Brian Ghigiarelli > wrote: > > Thanks, Bryan! Sure enough, we have nifi-standard-services-api-nar as a > > parent to use the standard SSLContextService. Sounds like upgrading the > > mongo-java-driver for the base build is the way to go for us. > > > > Thanks again, > > Brian > > On Mon, Mar 26, 2018 at 15:29 Bryan Bende wrote: > > > >> Brian, > >> > >> Is your custom processor using the MongoDBClientService provided by > >> NiFI's standard services API? or does your NAR have a parent of > >> nifi-standard-services-api-nar to use other services? > >> > >> Looking at where the Mongo JARs are from a build of master... > >> > >> find work/nar/ -name *mongo-java*.jar > >> > >> > work/nar//extensions/nifi-standard-services-api-nar-1.6.0-SNAPSHOT.nar-unpacked/META-INF/bundled-dependencies/mongo-java-driver-3.2.2.jar > >> > >> > work/nar//extensions/nifi-mongodb-services-nar-1.6.0-SNAPSHOT.nar-unpacked/META-INF/bundled-dependencies/mongo-java-driver-3.2.2.jar > >> > >> > work/nar//extensions/nifi-mongodb-nar-1.6.0-SNAPSHOT.nar-unpacked/META-INF/bundled-dependencies/mongo-java-driver-3.2.2.jar > >> > >> I think the issue is that if your NAR has > >> nifi-standard-services-api-nar as a parent NAR (which it probably does > >> to use SSLContext service, or any other standard service) then you are > >> getting mongo-java-driver-3.2.2 from MongoDBClientService since we > >> have parent first class loading. > >> > >> Given this situation I think there are only two options... not having > >> nifi-standard-services-api-nar as a parent (which stops you from using > >> all standard services), or upgrading the version of mongo-java-driver > >> used by MongoDBClientService. > >> > >> -Bryan > >> > >> > >> On Mon, Mar 26, 2018 at 3:16 PM, Brian Ghigiarelli > > >> wrote: > >> > Hey all, > >> > > >> > Is there a means of troubleshooting a custom NAR that seems to be > having > >> > runtime conflicts by picking up an older JAR that's provided by the > NiFi > >> > standard bundle? > >> > > >> > In this particular case, I'm using some custom processors that are > built > >> > against NiFi 1.4.0 and have a dependency on MongoDB 3.6.3. At runtime, > >> I'm > >> > seeing the processor use classes from MongoDB 3.2.2 that's provided by > >> > NiFi. Both NiFi and the custom NAR compile successfully on their own, > but > >> > using the custom NAR in NiFi causes NoSuchMethodError's due to a new > >> method > >> > only available in MongoDB 3.4+. > >> > > >> > Thanks, > >> > Brian > >> >
Re: Class Loading Conflicts - Different JAR Versions
Bri/yan, IMO I think we'd be better off with moving the nifi-mongodb-client-service-api and nifi-mongodb-services bundles into the nifi-mongodb-bundle. That's something I missed while reviewing the PR that put them in standard services. I don't see a direct dependency on nifi-standard-services, and if the nifi-mongodb-client-service-api needs the nifi-standard-services-api as a dependency, it can make it a NAR dependency. We might want to visit this as a much broader case, since I believe we could run into this with other services APIs (Elasticsearch, HBase, etc.)? Certainly when the Extension Registry becomes a thing, we might not want "external" service APIs to be part of nifi-standard-services-api. Regards, Matt On Mon, Mar 26, 2018 at 5:03 PM, Brian Ghigiarelli wrote: > Thanks, Bryan! Sure enough, we have nifi-standard-services-api-nar as a > parent to use the standard SSLContextService. Sounds like upgrading the > mongo-java-driver for the base build is the way to go for us. > > Thanks again, > Brian > On Mon, Mar 26, 2018 at 15:29 Bryan Bende wrote: > >> Brian, >> >> Is your custom processor using the MongoDBClientService provided by >> NiFI's standard services API? or does your NAR have a parent of >> nifi-standard-services-api-nar to use other services? >> >> Looking at where the Mongo JARs are from a build of master... >> >> find work/nar/ -name *mongo-java*.jar >> >> work/nar//extensions/nifi-standard-services-api-nar-1.6.0-SNAPSHOT.nar-unpacked/META-INF/bundled-dependencies/mongo-java-driver-3.2.2.jar >> >> work/nar//extensions/nifi-mongodb-services-nar-1.6.0-SNAPSHOT.nar-unpacked/META-INF/bundled-dependencies/mongo-java-driver-3.2.2.jar >> >> work/nar//extensions/nifi-mongodb-nar-1.6.0-SNAPSHOT.nar-unpacked/META-INF/bundled-dependencies/mongo-java-driver-3.2.2.jar >> >> I think the issue is that if your NAR has >> nifi-standard-services-api-nar as a parent NAR (which it probably does >> to use SSLContext service, or any other standard service) then you are >> getting mongo-java-driver-3.2.2 from MongoDBClientService since we >> have parent first class loading. >> >> Given this situation I think there are only two options... not having >> nifi-standard-services-api-nar as a parent (which stops you from using >> all standard services), or upgrading the version of mongo-java-driver >> used by MongoDBClientService. >> >> -Bryan >> >> >> On Mon, Mar 26, 2018 at 3:16 PM, Brian Ghigiarelli >> wrote: >> > Hey all, >> > >> > Is there a means of troubleshooting a custom NAR that seems to be having >> > runtime conflicts by picking up an older JAR that's provided by the NiFi >> > standard bundle? >> > >> > In this particular case, I'm using some custom processors that are built >> > against NiFi 1.4.0 and have a dependency on MongoDB 3.6.3. At runtime, >> I'm >> > seeing the processor use classes from MongoDB 3.2.2 that's provided by >> > NiFi. Both NiFi and the custom NAR compile successfully on their own, but >> > using the custom NAR in NiFi causes NoSuchMethodError's due to a new >> method >> > only available in MongoDB 3.4+. >> > >> > Thanks, >> > Brian >>
Re: Class Loading Conflicts - Different JAR Versions
Thanks, Bryan! Sure enough, we have nifi-standard-services-api-nar as a parent to use the standard SSLContextService. Sounds like upgrading the mongo-java-driver for the base build is the way to go for us. Thanks again, Brian On Mon, Mar 26, 2018 at 15:29 Bryan Bende wrote: > Brian, > > Is your custom processor using the MongoDBClientService provided by > NiFI's standard services API? or does your NAR have a parent of > nifi-standard-services-api-nar to use other services? > > Looking at where the Mongo JARs are from a build of master... > > find work/nar/ -name *mongo-java*.jar > > work/nar//extensions/nifi-standard-services-api-nar-1.6.0-SNAPSHOT.nar-unpacked/META-INF/bundled-dependencies/mongo-java-driver-3.2.2.jar > > work/nar//extensions/nifi-mongodb-services-nar-1.6.0-SNAPSHOT.nar-unpacked/META-INF/bundled-dependencies/mongo-java-driver-3.2.2.jar > > work/nar//extensions/nifi-mongodb-nar-1.6.0-SNAPSHOT.nar-unpacked/META-INF/bundled-dependencies/mongo-java-driver-3.2.2.jar > > I think the issue is that if your NAR has > nifi-standard-services-api-nar as a parent NAR (which it probably does > to use SSLContext service, or any other standard service) then you are > getting mongo-java-driver-3.2.2 from MongoDBClientService since we > have parent first class loading. > > Given this situation I think there are only two options... not having > nifi-standard-services-api-nar as a parent (which stops you from using > all standard services), or upgrading the version of mongo-java-driver > used by MongoDBClientService. > > -Bryan > > > On Mon, Mar 26, 2018 at 3:16 PM, Brian Ghigiarelli > wrote: > > Hey all, > > > > Is there a means of troubleshooting a custom NAR that seems to be having > > runtime conflicts by picking up an older JAR that's provided by the NiFi > > standard bundle? > > > > In this particular case, I'm using some custom processors that are built > > against NiFi 1.4.0 and have a dependency on MongoDB 3.6.3. At runtime, > I'm > > seeing the processor use classes from MongoDB 3.2.2 that's provided by > > NiFi. Both NiFi and the custom NAR compile successfully on their own, but > > using the custom NAR in NiFi causes NoSuchMethodError's due to a new > method > > only available in MongoDB 3.4+. > > > > Thanks, > > Brian >
Re: Class Loading Conflicts - Different JAR Versions
Brian, Is your custom processor using the MongoDBClientService provided by NiFI's standard services API? or does your NAR have a parent of nifi-standard-services-api-nar to use other services? Looking at where the Mongo JARs are from a build of master... find work/nar/ -name *mongo-java*.jar work/nar//extensions/nifi-standard-services-api-nar-1.6.0-SNAPSHOT.nar-unpacked/META-INF/bundled-dependencies/mongo-java-driver-3.2.2.jar work/nar//extensions/nifi-mongodb-services-nar-1.6.0-SNAPSHOT.nar-unpacked/META-INF/bundled-dependencies/mongo-java-driver-3.2.2.jar work/nar//extensions/nifi-mongodb-nar-1.6.0-SNAPSHOT.nar-unpacked/META-INF/bundled-dependencies/mongo-java-driver-3.2.2.jar I think the issue is that if your NAR has nifi-standard-services-api-nar as a parent NAR (which it probably does to use SSLContext service, or any other standard service) then you are getting mongo-java-driver-3.2.2 from MongoDBClientService since we have parent first class loading. Given this situation I think there are only two options... not having nifi-standard-services-api-nar as a parent (which stops you from using all standard services), or upgrading the version of mongo-java-driver used by MongoDBClientService. -Bryan On Mon, Mar 26, 2018 at 3:16 PM, Brian Ghigiarelli wrote: > Hey all, > > Is there a means of troubleshooting a custom NAR that seems to be having > runtime conflicts by picking up an older JAR that's provided by the NiFi > standard bundle? > > In this particular case, I'm using some custom processors that are built > against NiFi 1.4.0 and have a dependency on MongoDB 3.6.3. At runtime, I'm > seeing the processor use classes from MongoDB 3.2.2 that's provided by > NiFi. Both NiFi and the custom NAR compile successfully on their own, but > using the custom NAR in NiFi causes NoSuchMethodError's due to a new method > only available in MongoDB 3.4+. > > Thanks, > Brian
