Re: nifi 1.0 extending a standard processor

[Koji Kawamura]

Hello,

Have you been able to find a way to extend the standard processors?

I was wondering what would be the proper way to do that. So I tested few things:

1. Create new custom nar which depends on nifi-standard-processors

First, I tried to add new nar bundle, let's say "nifi-stdpllus-bundle"
following "Maven Projects for Extensions" [1]. Then add a dependency
to nifi-standard-processors so that I can write a processor extends
MergeContent. When I compiled it and tried to start NiFi, I got the
exact same error that you reported [2].
It indicates that the newly added custom nar "nifi-stdplus-bundle"
made NiFi to load standard processors such as DetectDuplicate from
nifi-stdplus-bundle class loader, because the custom nar has
nifi-standard-processors.jar in it.
DetectDuplicate uses
org/apache/nifi/distributed/cache/client/Serializer, which resides in
another jar, nifi-distributed-cache-client-service-api.jar, which is
not contained in the custom jar.

I assume you were in this situation.

2. Add nar reference from custom nar to standard nar

The reason why nifi-standard-processors can use
nifi-distributed-cache-client-service-api.jar is that
nifi-standard-nar has nar reference to nifi-standard-services-api-nar
(which has Serializer) in its pom.

So, I added the same nar dependency in 'nifi-stdplus-bundle/pom.xml'.
Then NiFi started to launch without the previous error.

However, since standard processors are in two nars (standard and
custom), NiFi shows warning messages [3], and also it increases file
size.

3. Specify 'provided' scope

In order to avoid having standard processors in the custom nar file, I
used 'provided' scope for the nifi-standard-processors dependency in
nifi-stdplus-processors. Please check this commit as an example [4].

Hope this helps.

BTW, I'm interested in why you had to extend MergeContent processor in
the first place. Would you elaborate what was the shortcomings? I
wonder if it can be merged into the project. Having required change in
master branch is the best way to extend (enhance) a standard processor
:)


[1] 
https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions
[2]
2016-09-12 11:07:46,861 ERROR [main] org.apache.nifi.NiFi Failure to
launch NiFi due to java.util.ServiceConfigurationError:
org.apache.nifi.processor.Processor: Provider
org.apache.nifi.processors.standard.DetectDuplicate could not be
instantiated
java.util.ServiceConfigurationError:
org.apache.nifi.processor.Processor: Provider
org.apache.nifi.processors.standard.DetectDuplicate could not be
instantiated
[3]
2016-09-12 12:00:43,981 WARN [main]
org.apache.nifi.nar.ExtensionManager Attempt was made to load
org.apache.nifi.processors.standard.AttributesToJSON from
org.apache.nifi.nar.NarClassLoader[./work/nar/extensions/nifi-standard-nar-0.7.0.nar-unpacked]
but that class name is already loaded/registered from
org.apache.nifi.nar.NarClassLoader[./work/nar/extensions/nifi-stdplus-nar-0.7.0.nar-unpacked].
This may cause unpredictable behavior.  Order of NARs is not
guaranteed.
[4] 
https://github.com/ijokarumawak/nifi/commit/134ff5dfc144abe463630ec9d4ab332bd16f22ac

Thanks,
Koji

On Fri, Sep 9, 2016 at 11:25 PM, bsabin  wrote:
> Hi,
>
> I have a custom processor that extends the MergeContent proc and when NiFi
> starts I have this error in logs:
>
> 2016-09-09 18:17:00,607 ERROR [main] org.apache.nifi.NiFi Failure to launch
> NiFi due to java.util.ServiceConfigurationError:
> org.apache.nifi.processor.Processor: Provider
> org.apache.nifi.processors.standard.DetectDuplicate could not be
> instantiated
> java.util.ServiceConfigurationError: org.apache.nifi.processor.Processor:
> Provider org.apache.nifi.processors.standard.DetectDuplicate could not be
> instantiated
> at java.util.ServiceLoader.fail(ServiceLoader.java:232)
> ~[na:1.8.0_101]
> at java.util.ServiceLoader.access$100(ServiceLoader.java:185)
> ~[na:1.8.0_101]
> at
> java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384)
> ~[na:1.8.0_101]
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
> ~[na:1.8.0_101]
> at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
> ~[na:1.8.0_101]
> at
> org.apache.nifi.nar.ExtensionManager.loadExtensions(ExtensionManager.java:107)
> ~[nifi-nar-utils-1.0.0.jar:1.0.0]
> at
> org.apache.nifi.nar.ExtensionManager.discoverExtensions(ExtensionManager.java:88)
> ~[nifi-nar-utils-1.0.0.jar:1.0.0]
> at org.apache.nifi.NiFi.(NiFi.java:135)
> ~[nifi-runtime-1.0.0.jar:1.0.0]
> at org.apache.nifi.NiFi.main(NiFi.java:243)
> ~[nifi-runtime-1.0.0.jar:1.0.0]
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/nifi/distributed/cache/client/Serializer
> at java.lang.Class.getDeclaredConstructors0(Native Method)
> ~[na:1.8.0_101]
> at java.lang.Class.privateGetDeclaredConstructors(Class.java:2671)
> ~[na:1.8.0_101]
> at 

Re: Custom expression language functions

[Andrew Grande]

I think there is a decent compromise available. Or maybe it's there
already? Invoking a public static function via reflection.  There are
deployment and packaging limitations, sure, but at least it makes it easier
for users to invoke utility functions (or custom ones).

Andrew

On Sun, Sep 11, 2016, 5:24 PM Joe Witt  wrote:

> Gunjan
>
> Can you show some specific el statements youd like to be able to do?
>
> Thanks
> Joe
>
> On Sep 11, 2016 4:24 PM, "Gunjan Dave"  wrote:
>
> > Hi Joe,
> > Having the ability to extend or customize EL functions is important.
> Infact
> > few important functions like ability to fetch from distributed cache or
> > readily using Certain java claases like java.lang.*, Java.io.*,
> > java.match.* etc are missing.
> >
> > adding processor is certainly feasible but its an extra hop. With the
> > ability to write global functions it can be achived in single processor
> > instead of two.
> >
> > this would prove most useful in advanced rules tab of updateattribute
> > processor.
> >
> >
> >
> > On Sun, Sep 11, 2016, 12:07 PM Gunjan Dave 
> > wrote:
> >
> > > Hi Team, is there a way I can create custom functions which can then be
> > > referred in expression language and also be reusable.
> > >
> > > Thanks
> > > GUNJAN
> > >
> >
>

Re: Anyone having issues running nifi-kafka-0-9-processors junits?

[Oleg Zhurakousky]

Andre

Couple of questions
1. Since KafkaPublisher class exists in all 3 Kafka processors (0.9 and 0.10 
Kafka clients) under the same package name, which one does it happen in? I am 
assuming its intermittent.
2. How often does it happen?

My suspicion is that this test is using embedded Kafka together with NiFi test 
mocks and there may be slight possibility of a timing issue under the load of 
the build which means the test may need to be tuned. But if this test fails on 
0.10 only it could also mean that we may need to look a little deeper (since I 
can’t remember seeng this failure yet) which could relate to some of the 
changes in Kafka 10 clients

Cheers
Oleg

> On Sep 11, 2016, at 8:02 AM, Andre F de Miranda  wrote:
> 
> Hi there,
> 
> is anyone else having frequent errors on some of the Kafka processors?
> 
> Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 22.248 sec
> <<< FAILURE! - in org.apache.nifi.processors.kafka.pubsub.KafkaPublisherTest
> validateSuccessfulSendAsDelimited(org.apache.nifi.processors.kafka.pubsub.KafkaPublisherTest)
> Time elapsed: 1.413 sec  <<< ERROR!
> kafka.consumer.ConsumerTimeoutException: null
> at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:69)
> at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:33)
> at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:66)
> at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:58)
> at kafka.utils.IteratorTemplate.next(IteratorTemplate.scala:38)
> at kafka.consumer.ConsumerIterator.next(ConsumerIterator.scala:46)
> at
> org.apache.nifi.processors.kafka.pubsub.KafkaPublisherTest.validateSuccessfulSendAsDelimited(KafkaPublisherTest.java:115)
> 
> It seems to happen only (or at least more frequently) when build is
> performed with -T 2.0C
> 
> 
> Cheers

Re: Custom expression language functions

[Joe Witt]

Gunjan

Can you show some specific el statements youd like to be able to do?

Thanks
Joe

On Sep 11, 2016 4:24 PM, "Gunjan Dave"  wrote:

> Hi Joe,
> Having the ability to extend or customize EL functions is important. Infact
> few important functions like ability to fetch from distributed cache or
> readily using Certain java claases like java.lang.*, Java.io.*,
> java.match.* etc are missing.
>
> adding processor is certainly feasible but its an extra hop. With the
> ability to write global functions it can be achived in single processor
> instead of two.
>
> this would prove most useful in advanced rules tab of updateattribute
> processor.
>
>
>
> On Sun, Sep 11, 2016, 12:07 PM Gunjan Dave 
> wrote:
>
> > Hi Team, is there a way I can create custom functions which can then be
> > referred in expression language and also be reusable.
> >
> > Thanks
> > GUNJAN
> >
>

Re: Custom expression language functions

[Gunjan Dave]

Hi Joe,
Having the ability to extend or customize EL functions is important. Infact
few important functions like ability to fetch from distributed cache or
readily using Certain java claases like java.lang.*, Java.io.*,
java.match.* etc are missing.

adding processor is certainly feasible but its an extra hop. With the
ability to write global functions it can be achived in single processor
instead of two.

this would prove most useful in advanced rules tab of updateattribute
processor.



On Sun, Sep 11, 2016, 12:07 PM Gunjan Dave 
wrote:

> Hi Team, is there a way I can create custom functions which can then be
> referred in expression language and also be reusable.
>
> Thanks
> GUNJAN
>

Anyone having issues running nifi-kafka-0-9-processors junits?

[Andre F de Miranda]

Hi there,

is anyone else having frequent errors on some of the Kafka processors?

Tests run: 6, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 22.248 sec
<<< FAILURE! - in org.apache.nifi.processors.kafka.pubsub.KafkaPublisherTest
validateSuccessfulSendAsDelimited(org.apache.nifi.processors.kafka.pubsub.KafkaPublisherTest)
 Time elapsed: 1.413 sec  <<< ERROR!
kafka.consumer.ConsumerTimeoutException: null
at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:69)
at kafka.consumer.ConsumerIterator.makeNext(ConsumerIterator.scala:33)
at kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:66)
at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:58)
at kafka.utils.IteratorTemplate.next(IteratorTemplate.scala:38)
at kafka.consumer.ConsumerIterator.next(ConsumerIterator.scala:46)
at
org.apache.nifi.processors.kafka.pubsub.KafkaPublisherTest.validateSuccessfulSendAsDelimited(KafkaPublisherTest.java:115)

It seems to happen only (or at least more frequently) when build is
performed with -T 2.0C


Cheers

Re: Custom expression language functions

[Joe Witt]

Hello

At this time expression language functions are not a designed extension
point.  You can of course alter non public parts of the codebase but that
may prove a tricky path for upgrades.  I do recommend simply writing a
processor to do the sort of manipulation you're looking for.  If you find
that proves too unwieldy let's discuss more.

Thanks
Joe

On Sep 11, 2016 7:17 AM, "Gunjan Dave"  wrote:

> Hi Team, is there a way I can create custom functions which can then be
> referred in expression language and also be reusable.
>
> Thanks
> GUNJAN
>

Custom expression language functions

[Gunjan Dave]

Hi Team, is there a way I can create custom functions which can then be
referred in expression language and also be reusable.

Thanks
GUNJAN