Also, regarding the elastic search issue Mike mentioned…
Usually when a processor can’t select the controller service at runtime it is
because theres an issue with the way the dependencies are setup between NARs.
There should generally be dependency paths like the following…
nifi-foo-service-impl-nar ------|
|--> nifi-foo-service-api-nar ->
nifi-standard-services-api-nar
nifi-foo-processors-nar -------|
All these links would be dependencies of <type>nar</type> in the poms.
The above is for the case where processors and service implementation are
packaged separately, which is slightly different than what I was suggesting for
the Mongo case.
> On Mar 26, 2018, at 10:06 PM, Bryan Bende <bbe...@gmail.com> wrote:
>
> I’m a +1 for moving the Mongo stuff out of standard services.
>
> Controller service APIs should always be broken out into their own NAR, but
> the processors and implementation can usually be bundled together.
>
> So something like the following would work:
>
> nifi-mongo-bundle
> - nifi-mongodb-services-api
> - nifi-mongodb-services-api-nar (would have NAR dependency on standard
> services API NAR)
> - nifi-mongodb-processors
> - nifi-mongodb-nar (would have NAR dependency on
> nifi-mongodb-services-api-nar)
>
> The processors module would have the processors and the client service
> implementation.
>
>
>> On Mar 26, 2018, at 9:13 PM, Brian Ghigiarelli <briang...@gmail.com> wrote:
>>
>> Thanks for the clarification, Mike.
>>
>> My primary motivation for this upgrade is to support an aggregation
>> pipeline with the $out stage. Support for this in the Java driver was
>> introduced in 3.4 via https://jira.mongodb.org/browse/JAVA-2254. Prior to
>> this, I believe you had to use the cursor to query all of the results,
>> which suffers from both query performance issues and concurrency issues for
>> the data flow if we're trying to overwrite an entire collection at once.
>> That said, if I'm missing something, I'm happy to learn!
>>
>> Thanks again,
>> Brian
>>
>> On Mon, Mar 26, 2018 at 7:57 PM Mike Thomsen <mikerthom...@gmail.com> wrote:
>>
>>> I was just following a convention at the time. So unless something breaks
>>> because of the move, I don't see any reason why it would be an issue to
>>> move it.
>>>
>>> In fact, with NIFI-4325 the only reason I put the new ElasticSearch
>>> service/api in the standard-services segment of the code base was I ran
>>> into a bizarre issue at runtime where I could define the controller
>>> service, but the processor had no idea it was there. So that too might
>>> warrant some extra eyes now that #4325 was added to master today.
>>>
>>> FWIW, I can't think of any reason why the MongoDB client service needs 3.4
>>> or 3.6 over 3.2.
>>>
>>> Brian, feel free to ping me on the review.
>>>
>>> Thanks,
>>>
>>> Mike
>>>
>>> On Mon, Mar 26, 2018 at 6:59 PM, Matt Burgess <mattyb...@apache.org>
>>> wrote:
>>>
>>>> Brian (this one's all you ;),
>>>>
>>>> I think that PR is quite welcome in my opinion, but I'd like to get
>>>> Mike Thomsen's and others' opinions on the subject too, I think Mike
>>>> wrote the original service (or at the least is very knowledgable about
>>>> Mongo and NiFi), he might have run into issues that led him to put it
>>>> there for a reason. If not, let's do it, if you're willing to submit
>>>> the contribution I/we'd be more than happy to review/incorporate it :)
>>>>
>>>> Regards,
>>>> Matt
>>>>
>>>>
>>>> On Mon, Mar 26, 2018 at 6:20 PM, Brian Ghigiarelli <briang...@gmail.com>
>>>> wrote:
>>>>> Matt,
>>>>>
>>>>> +1 [non-binding] for the idea to move the Mongo dependencies into that
>>>>> bundle. I think it will likely need the NAR dependency on
>>>>> nifi-standard-services-api in order to provide a link to the SSL
>>> Context
>>>>> Service. Is that ticket / PR worthy as a one-off in the short term? No
>>>>> doubt the Extension Registry will be a powerful capability.
>>>>>
>>>>> Also, +2 for the greeting, though Bryan may choose to put the "y"
>>> first.
>>>> :-)
>>>>>
>>>>> Thanks,
>>>>> Brian
>>>>>
>>>>> On Mon, Mar 26, 2018 at 5:13 PM Matt Burgess <mattyb...@apache.org>
>>>> wrote:
>>>>>
>>>>>> Bri/yan,
>>>>>>
>>>>>> IMO I think we'd be better off with moving the
>>>>>> nifi-mongodb-client-service-api and nifi-mongodb-services bundles into
>>>>>> the nifi-mongodb-bundle. That's something I missed while reviewing the
>>>>>> PR that put them in standard services. I don't see a direct dependency
>>>>>> on nifi-standard-services, and if the nifi-mongodb-client-service-api
>>>>>> needs the nifi-standard-services-api as a dependency, it can make it a
>>>>>> NAR dependency.
>>>>>>
>>>>>> We might want to visit this as a much broader case, since I believe we
>>>>>> could run into this with other services APIs (Elasticsearch, HBase,
>>>>>> etc.)? Certainly when the Extension Registry becomes a thing, we might
>>>>>> not want "external" service APIs to be part of
>>>>>> nifi-standard-services-api.
>>>>>>
>>>>>> Regards,
>>>>>> Matt
>>>>>>
>>>>>>
>>>>>> On Mon, Mar 26, 2018 at 5:03 PM, Brian Ghigiarelli <
>>> briang...@gmail.com
>>>>>
>>>>>> wrote:
>>>>>>> Thanks, Bryan! Sure enough, we have nifi-standard-services-api-nar
>>> as
>>>> a
>>>>>>> parent to use the standard SSLContextService. Sounds like upgrading
>>>> the
>>>>>>> mongo-java-driver for the base build is the way to go for us.
>>>>>>>
>>>>>>> Thanks again,
>>>>>>> Brian
>>>>>>> On Mon, Mar 26, 2018 at 15:29 Bryan Bende <bbe...@gmail.com> wrote:
>>>>>>>
>>>>>>>> Brian,
>>>>>>>>
>>>>>>>> Is your custom processor using the MongoDBClientService provided by
>>>>>>>> NiFI's standard services API? or does your NAR have a parent of
>>>>>>>> nifi-standard-services-api-nar to use other services?
>>>>>>>>
>>>>>>>> Looking at where the Mongo JARs are from a build of master...
>>>>>>>>
>>>>>>>> find work/nar/ -name *mongo-java*.jar
>>>>>>>>
>>>>>>>>
>>>>>> work/nar//extensions/nifi-standard-services-api-nar-1.6.
>>>> 0-SNAPSHOT.nar-unpacked/META-INF/bundled-dependencies/
>>>> mongo-java-driver-3.2.2.jar
>>>>>>>>
>>>>>>>>
>>>>>> work/nar//extensions/nifi-mongodb-services-nar-1.6.0-
>>>> SNAPSHOT.nar-unpacked/META-INF/bundled-dependencies/
>>>> mongo-java-driver-3.2.2.jar
>>>>>>>>
>>>>>>>>
>>>>>> work/nar//extensions/nifi-mongodb-nar-1.6.0-SNAPSHOT.
>>>> nar-unpacked/META-INF/bundled-dependencies/mongo-java-driver-3.2.2.jar
>>>>>>>>
>>>>>>>> I think the issue is that if your NAR has
>>>>>>>> nifi-standard-services-api-nar as a parent NAR (which it probably
>>>> does
>>>>>>>> to use SSLContext service, or any other standard service) then you
>>>> are
>>>>>>>> getting mongo-java-driver-3.2.2 from MongoDBClientService since we
>>>>>>>> have parent first class loading.
>>>>>>>>
>>>>>>>> Given this situation I think there are only two options... not
>>> having
>>>>>>>> nifi-standard-services-api-nar as a parent (which stops you from
>>>> using
>>>>>>>> all standard services), or upgrading the version of
>>> mongo-java-driver
>>>>>>>> used by MongoDBClientService.
>>>>>>>>
>>>>>>>> -Bryan
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Mar 26, 2018 at 3:16 PM, Brian Ghigiarelli <
>>>> briang...@gmail.com
>>>>>>>
>>>>>>>> wrote:
>>>>>>>>> Hey all,
>>>>>>>>>
>>>>>>>>> Is there a means of troubleshooting a custom NAR that seems to be
>>>>>> having
>>>>>>>>> runtime conflicts by picking up an older JAR that's provided by
>>> the
>>>>>> NiFi
>>>>>>>>> standard bundle?
>>>>>>>>>
>>>>>>>>> In this particular case, I'm using some custom processors that
>>> are
>>>>>> built
>>>>>>>>> against NiFi 1.4.0 and have a dependency on MongoDB 3.6.3. At
>>>> runtime,
>>>>>>>> I'm
>>>>>>>>> seeing the processor use classes from MongoDB 3.2.2 that's
>>>> provided by
>>>>>>>>> NiFi. Both NiFi and the custom NAR compile successfully on their
>>>> own,
>>>>>> but
>>>>>>>>> using the custom NAR in NiFi causes NoSuchMethodError's due to a
>>>> new
>>>>>>>> method
>>>>>>>>> only available in MongoDB 3.4+.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Brian
>>>>>>>>
>>>>>>
>>>>
>>>
>
