Re: Duplicated processors when using nifi processors dependency
Sorry if this duplicates, I sent from the wrong address the first time. I’ve opened a PR on GitHub for this issue: https://github.com/apache/ nifi/pull/2140 Let me know if there’s any other follow up I need to do; I’d love to see this get in for 1.4.0. Christopher On 2017-05-22 06:16, Bryan Bende <b...@gmail.com> wrote: Thanks James.> For the abstract AWS processors, I would go with one of the following...> 1) Another module under nifi-aws-bundle like:> nifi-aws-bunde> nifi-aws-service-api> nifi-aws-service-api-nar> nifi-aws-abstract-processors (or some similar name)> nifi-aws-processors> nifi-aws-nar> 2) A new module under nifi-nar-bundles/nifi-extension-utils like nifi-aws-utils> Either way, the desired result is to have a JAR that someone can> depend on that only brings in the abstract processors.> We have been putting generic abstract processors into> nifi-nar-bundles/nifi-extension-utils /nifi-processor-utils, but since> this case depends on the AWS SDK, then we would be introducing that> dependency to all the NARs that depend on nifi-processor-utils, which> is why I was suggesting a new module.> Let me know if this doesn't make sense.> On Mon, May 22, 2017 at 1:24 AM, James Wing <jv...@gmail.com> wrote:> I created a JIRA for the nifi-aws-bundle structure,> https://issues.apache.org/jira/browse/NIFI-3950.> Is there a best practice for the abstract classes like Robert describes?> Are they a better fit for the service API bundle or the processors bundle?> Or does that become irrelevant once the NAR dependencies are aligned and> they can be references as Java libraries?> Thanks,> James> On Sun, May 21, 2017 at 10:28 AM, Bryan Bende <bb...@gmail.com> wrote:> Currently the AWS NAR contains the processors, controller service API, and> controller service implementation, all in one NAR.> Typically the controller service API should be in its own NAR, so there> should be something like:> nifi-aws-bunde> nifi-aws-service-api> nifi-aws-service-api-nar> nifi-aws-processors> nifi-aws-nar> The nifi-aws-service-api module would have the> AWSCredentialsProviderService interface. The nifi-aws-nar would have a NAR> dependency on nifi-aws-service-api-nar.> Setting it up this way, you nifi-aws-utils project could have a provided> dependency on the nifi-aws-service-api, and then your custom NAR could have> a NAR dependency on nifi-aws-service-api-nar.> There is some more information here about setting up controller services:> https://cwiki.apache.org/confluence/display/NIFI/Maven+> Projects+for+Extensions#MavenProjectsforExtensions-> LinkingProcessorsandControllerServices <https://cwiki.apache.org/> confluence/display/NIFI/Maven+Projects+for+Extensions#> MavenProjectsforExtensions-LinkingProcessorsandControllerServices>> https://github.com/bbende/nifi-dependency-example <> https://github.com/bbende/nifi-dependency-example>> On May 20, 2017, at 10:06 AM, Robert <ro...@gmail.com> wrote:> I tired to do move abstract aws processors from> nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors to newly created> nifi-aws-utils.> Problem that I face now is with AWSCredentialsProviderControllerService> that> imo should stay under nifi-aws-bundle, but it is required by> AbstractAWSCredentialsProviderProcessor that should be moved to> nifi-aws-utils as it is used by abstract aws processors.> What is the preferred way to solve this?> Should I created new 'nifi-aws-service-bundle' under nifi-nar-bundles and> both nifi-aws-processors and nifi-aws-utils can depend on it?> --> View this message in context: http://apache-nifi-developer-> list.39713.n7.nabble.com/Duplicated-processors-when-using-nifi-processors-> dependency-tp15902p15937.html> Sent from the Apache NiFi Developer List mailing list archive at> Nabble.com.>
Re: Duplicated processors when using nifi processors dependency
Thanks James. For the abstract AWS processors, I would go with one of the following... 1) Another module under nifi-aws-bundle like: nifi-aws-bunde nifi-aws-service-api nifi-aws-service-api-nar nifi-aws-abstract-processors (or some similar name) nifi-aws-processors nifi-aws-nar 2) A new module under nifi-nar-bundles/nifi-extension-utils like nifi-aws-utils Either way, the desired result is to have a JAR that someone can depend on that only brings in the abstract processors. We have been putting generic abstract processors into nifi-nar-bundles/nifi-extension-utils /nifi-processor-utils, but since this case depends on the AWS SDK, then we would be introducing that dependency to all the NARs that depend on nifi-processor-utils, which is why I was suggesting a new module. Let me know if this doesn't make sense. On Mon, May 22, 2017 at 1:24 AM, James Wing <jvw...@gmail.com> wrote: > I created a JIRA for the nifi-aws-bundle structure, > https://issues.apache.org/jira/browse/NIFI-3950. > > Is there a best practice for the abstract classes like Robert describes? > Are they a better fit for the service API bundle or the processors bundle? > Or does that become irrelevant once the NAR dependencies are aligned and > they can be references as Java libraries? > > Thanks, > > James > > On Sun, May 21, 2017 at 10:28 AM, Bryan Bende <bbe...@gmail.com> wrote: > >> Currently the AWS NAR contains the processors, controller service API, and >> controller service implementation, all in one NAR. >> >> Typically the controller service API should be in its own NAR, so there >> should be something like: >> >> nifi-aws-bunde >> nifi-aws-service-api >> nifi-aws-service-api-nar >> nifi-aws-processors >> nifi-aws-nar >> >> The nifi-aws-service-api module would have the >> AWSCredentialsProviderService interface. The nifi-aws-nar would have a NAR >> dependency on nifi-aws-service-api-nar. >> >> Setting it up this way, you nifi-aws-utils project could have a provided >> dependency on the nifi-aws-service-api, and then your custom NAR could have >> a NAR dependency on nifi-aws-service-api-nar. >> >> There is some more information here about setting up controller services: >> >> https://cwiki.apache.org/confluence/display/NIFI/Maven+ >> Projects+for+Extensions#MavenProjectsforExtensions- >> LinkingProcessorsandControllerServices <https://cwiki.apache.org/ >> confluence/display/NIFI/Maven+Projects+for+Extensions# >> MavenProjectsforExtensions-LinkingProcessorsandControllerServices> >> >> https://github.com/bbende/nifi-dependency-example < >> https://github.com/bbende/nifi-dependency-example> >> >> >> > On May 20, 2017, at 10:06 AM, Robert <robert.s...@gmail.com> wrote: >> > >> > I tired to do move abstract aws processors from >> > nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors to newly created >> > nifi-aws-utils. >> > >> > Problem that I face now is with AWSCredentialsProviderControllerService >> that >> > imo should stay under nifi-aws-bundle, but it is required by >> > AbstractAWSCredentialsProviderProcessor that should be moved to >> > nifi-aws-utils as it is used by abstract aws processors. >> > >> > What is the preferred way to solve this? >> > Should I created new 'nifi-aws-service-bundle' under nifi-nar-bundles and >> > both nifi-aws-processors and nifi-aws-utils can depend on it? >> > >> > >> > >> > >> > >> > -- >> > View this message in context: http://apache-nifi-developer- >> list.39713.n7.nabble.com/Duplicated-processors-when-using-nifi-processors- >> dependency-tp15902p15937.html >> > Sent from the Apache NiFi Developer List mailing list archive at >> Nabble.com. >> >>
Re: Duplicated processors when using nifi processors dependency
I created a JIRA for the nifi-aws-bundle structure, https://issues.apache.org/jira/browse/NIFI-3950. Is there a best practice for the abstract classes like Robert describes? Are they a better fit for the service API bundle or the processors bundle? Or does that become irrelevant once the NAR dependencies are aligned and they can be references as Java libraries? Thanks, James On Sun, May 21, 2017 at 10:28 AM, Bryan Bende <bbe...@gmail.com> wrote: > Currently the AWS NAR contains the processors, controller service API, and > controller service implementation, all in one NAR. > > Typically the controller service API should be in its own NAR, so there > should be something like: > > nifi-aws-bunde > nifi-aws-service-api > nifi-aws-service-api-nar > nifi-aws-processors > nifi-aws-nar > > The nifi-aws-service-api module would have the > AWSCredentialsProviderService interface. The nifi-aws-nar would have a NAR > dependency on nifi-aws-service-api-nar. > > Setting it up this way, you nifi-aws-utils project could have a provided > dependency on the nifi-aws-service-api, and then your custom NAR could have > a NAR dependency on nifi-aws-service-api-nar. > > There is some more information here about setting up controller services: > > https://cwiki.apache.org/confluence/display/NIFI/Maven+ > Projects+for+Extensions#MavenProjectsforExtensions- > LinkingProcessorsandControllerServices <https://cwiki.apache.org/ > confluence/display/NIFI/Maven+Projects+for+Extensions# > MavenProjectsforExtensions-LinkingProcessorsandControllerServices> > > https://github.com/bbende/nifi-dependency-example < > https://github.com/bbende/nifi-dependency-example> > > > > On May 20, 2017, at 10:06 AM, Robert <robert.s...@gmail.com> wrote: > > > > I tired to do move abstract aws processors from > > nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors to newly created > > nifi-aws-utils. > > > > Problem that I face now is with AWSCredentialsProviderControllerService > that > > imo should stay under nifi-aws-bundle, but it is required by > > AbstractAWSCredentialsProviderProcessor that should be moved to > > nifi-aws-utils as it is used by abstract aws processors. > > > > What is the preferred way to solve this? > > Should I created new 'nifi-aws-service-bundle' under nifi-nar-bundles and > > both nifi-aws-processors and nifi-aws-utils can depend on it? > > > > > > > > > > > > -- > > View this message in context: http://apache-nifi-developer- > list.39713.n7.nabble.com/Duplicated-processors-when-using-nifi-processors- > dependency-tp15902p15937.html > > Sent from the Apache NiFi Developer List mailing list archive at > Nabble.com. > >
Re: Duplicated processors when using nifi processors dependency
Currently the AWS NAR contains the processors, controller service API, and controller service implementation, all in one NAR. Typically the controller service API should be in its own NAR, so there should be something like: nifi-aws-bunde nifi-aws-service-api nifi-aws-service-api-nar nifi-aws-processors nifi-aws-nar The nifi-aws-service-api module would have the AWSCredentialsProviderService interface. The nifi-aws-nar would have a NAR dependency on nifi-aws-service-api-nar. Setting it up this way, you nifi-aws-utils project could have a provided dependency on the nifi-aws-service-api, and then your custom NAR could have a NAR dependency on nifi-aws-service-api-nar. There is some more information here about setting up controller services: https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions#MavenProjectsforExtensions-LinkingProcessorsandControllerServices <https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions#MavenProjectsforExtensions-LinkingProcessorsandControllerServices> https://github.com/bbende/nifi-dependency-example <https://github.com/bbende/nifi-dependency-example> > On May 20, 2017, at 10:06 AM, Robert <robert.s...@gmail.com> wrote: > > I tired to do move abstract aws processors from > nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors to newly created > nifi-aws-utils. > > Problem that I face now is with AWSCredentialsProviderControllerService that > imo should stay under nifi-aws-bundle, but it is required by > AbstractAWSCredentialsProviderProcessor that should be moved to > nifi-aws-utils as it is used by abstract aws processors. > > What is the preferred way to solve this? > Should I created new 'nifi-aws-service-bundle' under nifi-nar-bundles and > both nifi-aws-processors and nifi-aws-utils can depend on it? > > > > > > -- > View this message in context: > http://apache-nifi-developer-list.39713.n7.nabble.com/Duplicated-processors-when-using-nifi-processors-dependency-tp15902p15937.html > Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Re: Duplicated processors when using nifi processors dependency
I tired to do move abstract aws processors from nifi-nar-bundles/nifi-aws-bundle/nifi-aws-processors to newly created nifi-aws-utils. Problem that I face now is with AWSCredentialsProviderControllerService that imo should stay under nifi-aws-bundle, but it is required by AbstractAWSCredentialsProviderProcessor that should be moved to nifi-aws-utils as it is used by abstract aws processors. What is the preferred way to solve this? Should I created new 'nifi-aws-service-bundle' under nifi-nar-bundles and both nifi-aws-processors and nifi-aws-utils can depend on it? -- View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Duplicated-processors-when-using-nifi-processors-dependency-tp15902p15937.html Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Re: Duplicated processors when using nifi processors dependency
Hi Robert, It sounds like your NAR is including nifi-aws-processors, and so is the regular AWS NAR. Previously it wasn't obvious this was happening, and when NiFi started up it was finding those processors in one of the NARs and skipping the other one. With component versioning it is now more specific as to where the processors are coming from. The preferred solution here would be to create a utility project like"nifi-aws-utils" underneath nifi-nar-bundles/nifi-extension-utils: https://github.com/apache/nifi/tree/master/nifi-nar-bundles/nifi-extension-utils This project would then contain the abstract processor, then your custom processors and the regular AWS processors could both depend on and include this JAR. If you wanted to submit a PR/patch that created this utility project, we'd be happy to help get it reviewed/merged, although I realize you would have to be dependent on the latest code in master until a release occurred with the change. Hope that helps. Thanks, Bryan On Thu, May 18, 2017 at 2:58 AM, Robert <robert.s...@gmail.com> wrote: > For my custom processor I am extending AbstractS3Processor. For that I use > nifi-aws-processors as a dependency. > With upgrade to 1.2.0, Nifi now loads also all aws processors that are > packed in nar as a depencency. Is there any way to avoid duplicated > processors that get loaded in that way? > > > > -- > View this message in context: > http://apache-nifi-developer-list.39713.n7.nabble.com/Duplicated-processors-when-using-nifi-processors-dependency-tp15902.html > Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
Duplicated processors when using nifi processors dependency
For my custom processor I am extending AbstractS3Processor. For that I use nifi-aws-processors as a dependency. With upgrade to 1.2.0, Nifi now loads also all aws processors that are packed in nar as a depencency. Is there any way to avoid duplicated processors that get loaded in that way? -- View this message in context: http://apache-nifi-developer-list.39713.n7.nabble.com/Duplicated-processors-when-using-nifi-processors-dependency-tp15902.html Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
