Re: Request for nifi custom processor development guidance in python

[Pierre Villard]

Hi,

If you're using NiFi 2.0, then you can use Python. I recommend the below
resources:
https://www.youtube.com/watch?v=9Oi_6nFmbPg
https://nifi.apache.org/documentation/nifi-2.0.0-M2/html/python-developer-guide.html

HTH,
Pierre

Le mer. 17 avr. 2024 à 15:18, Usha Kiran Mahato 
a écrit :

> Hi Team,
>
> I am Usha, a python developer. Recently I got a project to work on nifi. To
> implement Client specific requirement, I have to build a custom processor
> in nifi. Currently I use java to build that.
>
> As you know python is reach in library, I know jython is there but it
> supports till python 2.7. So there are some limitations.
>
> Is there any way, so that I can build my custom processor in python instead
> of java with the LTS python version.
>
> It will be very helpful to me.
>
> Thanking you
>
> Regards
> Usha Kiran Mahato
> email-ushakiranmaha...@gmail.com
> Ph No.- +91 858 285 2477/
> +91 890 637 6213
>

Request for nifi custom processor development guidance in python

[Usha Kiran Mahato]

Hi Team,

I am Usha, a python developer. Recently I got a project to work on nifi. To
implement Client specific requirement, I have to build a custom processor
in nifi. Currently I use java to build that.

As you know python is reach in library, I know jython is there but it
supports till python 2.7. So there are some limitations.

Is there any way, so that I can build my custom processor in python instead
of java with the LTS python version.

It will be very helpful to me.

Thanking you

Regards
Usha Kiran Mahato
email-ushakiranmaha...@gmail.com
Ph No.- +91 858 285 2477/
+91 890 637 6213

Re: Custom-processor configuration suggestions

[Michael Moser]

es like IP geo enrichment where the dataset on which
> you'd make the decision is larger and more importantly subject to change
> over time. In such cases the ideal state is:
> (A) The reference dataset(s) is hosted at a RESTful endpoint and can be
> periodically pulled and stored some place local/easily accessible.
> (B) The processor knows where to look for this reference dataset download
> and is able to hot reload it on the fly to include understanding that the
> needed datasets might not yet be made available and it should yield until
> it sees them and loads them.
>
> Thanks
>
> On Wed, Sep 27, 2023 at 11:51 AM Russell Bateman 
> wrote:
>
> > I'm posting this plea for suggestions as I'm short on imagination here.
> >
> > We have some custom processors that need extraordinary amounts of
> > configuration of the sort a flow writer would have to copy and paste
> > in--huge amounts of Yaml, regular expressions, etc. This is what our
> > flow writers are already doing. It would be easier to insert a filename
> > or -path, but...
> >
> > ...asking a custom processor to perform filesystem I/O is icky because
> > of unpredictable filesystem access post installation. Thinking about how
> > installation is beyond my control, I don't want to make installation
> > messy, etc. Containers, Kubernetes deployment, etc. complicate this.
> >
> > I thought of wiring /GetFile/ to a subdirectory (problematic, but less
> > so?) and accepting files as input to pass on to needy processors who
> > would recognize, adopt and incorporate configuration based on
> > higher-level and simpler cues posted by flow writers as property values.
> >
> > Assuming you both grok and are interested in what I'm asking, do you
> > have thoughts, cautionary statements or even cat-calls to offer? Maybe
> > there are obvious answers I'm just not thinking of.
> >
> > Profuse thanks,
> >
> > Russ
>

Re: Custom-processor configuration suggestions

[Otto Fowler]

```

   ┌──┐
   │  │
   │  processor instance  │┐
   │  ││
   └──┘│
  .─.
   │
   ,─'   '─.
   │
 ,' `.
   │
╱ ╲
   │
   ╱   ╲
   │
  ; :
   │
┌─┐ │  configuration
│
   │ │
│  ┌─▶│authority│
   │ │
Configruation service │  │  : ;
   ├┬───▶│
│──┘   ╲   ╱
   ┌──┐│││
│   ╲ ╱
   │  │││
└─┘╲   ╱
   │  processor instance  │┘│
  `.   ,'
   │  │ │
'─. ,─'
   └──┘ │
   `───'
│
│
│
│
│
│
│
┌──┐│
│  ││
│  processor instance  │┤
│  ││
└──┘│
│
│
│
│
┌──┐│
│  ││
│  processor instance  │┘
│  │
└──┘
```

You can also make a shared service component that loads the configurations
by some means and serves them to the processors.
The service can get the configurations however makes sense for you ( from
REST API  like Joe is saying, to reading from disk or something ).


```



On September 27, 2023 at 4:04:56 PM, Joe Witt (joe.w...@gmail.com) wrote:

Russ

It sounds like what you have is a case of significant reference data you
need made available to various instances of this processor that knows how
to use that reference state to do its function.

This is similar to cases like IP geo enrichment where the dataset on which
you'd make the decision is larger and more importantly subject to change
over time. In such cases the ideal state is:
(A) The reference dataset(s) is hosted at a RESTful endpoint and can be
periodically pulled and stored some place local/easily accessible.
(B) The processor knows where to look for this reference dataset download
and is able to hot reload it on the fly to include understanding that the
needed datasets might not yet be made available and it should yield until
it sees them and loads them.

Thanks

On Wed, Sep 27, 2023 at 11:51 AM Russell Bateman 
wrote:

> I'm posting this plea for suggestions as I'm short on imagination here.
>
> We have some custom processors that need extraordinary amounts of
> configuration of the sort a flow writer would have to copy and paste
> in--huge amounts of Yaml, regular expressions, etc. This is what our
> flow writers are already doing. It would be easier to insert a filename
> or -path, but...
>
> ...asking a custom processor to perform filesystem I/O is icky because
> of unpredictable filesystem access post installation. Thinking about how
> installation is beyond my control, I don't want to make installation
> messy, etc. Containers, Kubernetes deployment, etc. complicate this.
>
> I thought of wiring /GetFile/ to a subdirectory (problematic, but less
> so?) and accepting files as input to pass on to needy processors who
> would recognize, adopt and incorporate configuration based on
> higher-level and simpler cues posted by flow writers as property values.
>
> Assuming you both grok and are interested in what I'm asking, do you

Re: Custom-processor configuration suggestions

[Joe Witt]

Russ

It sounds like what you have is a case of significant reference data you
need made available to various instances of this processor that knows how
to use that reference state to do its function.

This is similar to cases like IP geo enrichment where the dataset on which
you'd make the decision is larger and more importantly subject to change
over time.  In such cases the ideal state is:
(A) The reference dataset(s) is hosted at a RESTful endpoint and can be
periodically pulled and stored some place local/easily accessible.
(B) The processor knows where to look for this reference dataset download
and is able to hot reload it on the fly to include understanding that the
needed datasets might not yet be made available and it should yield until
it sees them and loads them.

Thanks

On Wed, Sep 27, 2023 at 11:51 AM Russell Bateman 
wrote:

> I'm posting this plea for suggestions as I'm short on imagination here.
>
> We have some custom processors that need extraordinary amounts of
> configuration of the sort a flow writer would have to copy and paste
> in--huge amounts of Yaml, regular expressions, etc. This is what our
> flow writers are already doing. It would be easier to insert a filename
> or -path, but...
>
> ...asking a custom processor to perform filesystem I/O is icky because
> of unpredictable filesystem access post installation. Thinking about how
> installation is beyond my control, I don't want to make installation
> messy, etc. Containers, Kubernetes deployment, etc. complicate this.
>
> I thought of wiring /GetFile/ to a subdirectory (problematic, but less
> so?) and accepting files as input to pass on to needy processors who
> would recognize, adopt and incorporate configuration based on
> higher-level and simpler cues posted by flow writers as property values.
>
> Assuming you both grok and are interested in what I'm asking, do you
> have thoughts, cautionary statements or even cat-calls to offer? Maybe
> there are obvious answers I'm just not thinking of.
>
> Profuse thanks,
>
> Russ

Custom-processor configuration suggestions

[Russell Bateman]

I'm posting this plea for suggestions as I'm short on imagination here.

We have some custom processors that need extraordinary amounts of 
configuration of the sort a flow writer would have to copy and paste 
in--huge amounts of Yaml, regular expressions, etc. This is what our 
flow writers are already doing. It would be easier to insert a filename 
or -path, but...


...asking a custom processor to perform filesystem I/O is icky because 
of unpredictable filesystem access post installation. Thinking about how 
installation is beyond my control, I don't want to make installation 
messy, etc. Containers, Kubernetes deployment, etc. complicate this.


I thought of wiring /GetFile/ to a subdirectory (problematic, but less 
so?) and accepting files as input to pass on to needy processors who 
would recognize, adopt and incorporate configuration based on 
higher-level and simpler cues posted by flow writers as property values.


Assuming you both grok and are interested in what I'm asking, do you 
have thoughts, cautionary statements or even cat-calls to offer? Maybe 
there are obvious answers I'm just not thinking of.


Profuse thanks,

Russ

Re: Custom processor once had properties, has no more now, but they still show up

[Mark Bean]

The flow.xml.gz/flow.json.gz has the (now removed) properties as part of
the definition. Therefore, they still appear on the updated processor, but
now as dynamic properties. The only recourse is to remove the properties
from existing processor instances, or to delete the processor and
instantiate a new one.

One way we deal with this situation is to deprecate the property, but keep
it as part of the processor definition. It has no functional value, and the
documentation indicates that it is deprecated. This gives users a period of
time to adjust their flows. Eventually, when a new major release version of
the custom NAR is released, the deprecated properties can be removed
completely.

-Mark


On Thu, Jul 20, 2023 at 7:43 PM Russell Bateman 
wrote:

> I have a custom processor I modified losing a few properties because a
> specification change made them useless. I removed them. The processor
> works, but in configuration for this processor (in the NiFi UI), the
> processor appears to have kept them, i.e.: they're not disappearing. I
> would have expected trash cans on each, but they're just still there.
>
> Yes, the processor bears the latest NAR version number and there is no
> "duplicate" or older NAR in the /extensions/ subdirectory.
>
> Is my best recourse to remove the processor from the flow and re-add it?
> This is not a problem, but I'll have down-streamers with their own flows
> I'm not in contact with who will find this behavior unsettling. I could
> issue documentation, but I try to keep extra work to a minimum for them.
>
> Thanks for any thoughts.
>
> Russ

Custom processor once had properties, has no more now, but they still show up

[Russell Bateman]

I have a custom processor I modified losing a few properties because a 
specification change made them useless. I removed them. The processor 
works, but in configuration for this processor (in the NiFi UI), the 
processor appears to have kept them, i.e.: they're not disappearing. I 
would have expected trash cans on each, but they're just still there.


Yes, the processor bears the latest NAR version number and there is no 
"duplicate" or older NAR in the /extensions/ subdirectory.


Is my best recourse to remove the processor from the flow and re-add it? 
This is not a problem, but I'll have down-streamers with their own flows 
I'm not in contact with who will find this behavior unsettling. I could 
issue documentation, but I try to keep extra work to a minimum for them.


Thanks for any thoughts.

Russ

Re: Queries regarding Nifi custom processor for Azure Data Explorer

[Joey Frazee]

Hi Tanmaya,

There’s been some community interest in ADX processors in the past. I’m 
curious, is this something your team is interested in contributing back to 
Apache NiFi, or do you plan to release it as a Microsoft open source project?

Either is fine of course.

One thing to consider is that conceptually data sources in Spark are usually a 
little different and hide a lot of details. NiFi processors tend to do fewer 
things and get composed into a flow. You could stuff most of the things you’re 
doing onto a single processor but if you have a good simple processor for 
writing to ADX some of the other stuff you mention might just be logic in a 
flow.

-joey

> On Oct 3, 2022, at 5:56 AM, Mike Thomsen  wrote:
> 
> I think that is at least partly doable within NiFi (ex yes, you can
> restrict processors to the primary node in a cluster), but I would
> recommend you consider a different approach for NiFi. Unlike Spark,
> NiFi is perfectly content to stream in huge amounts of data
> continuously without any temporary storage besides its repositories
> (flowfile, content, etc). Therefore, I think a potentially easier
> solution would be for your team to explore creating a connector
> between NiFi and Azure Data Explorer that allows the latter to
> firehose the former with data as it comes in and let the chips fall
> where they may.
> 
> You might find some useful concepts in the Kafka processors for things
> like managing a continuous flow of record data from a stream and
> converting it to blocks of NiFi record data (see the Record API in our
> documentation for details).
> 
> FWIW, I manage a data set in AWS w/ NiFi that is over 50TB compressed,
> and a fairly generic 5 node NiFi cluster crushes that data on cheap
> EC2 instances without issue. So handling TBs of data is something
> fairly out of the box for NiFi if you're worried about that.
> 
>> On Sat, Oct 1, 2022 at 12:05 AM Tanmaya Panda
>>  wrote:
>> 
>> Hi Team,
>> 
>> We at Microsoft opensource, are developing a custom for Azure Data Explorer 
>> sink connector for Apache Nifi. What we want to achieve is transactionality 
>> with data ingestion. The source of the processor can be TBs of telemetry 
>> data as well as CDC logs. What that means is in case of any failure while 
>> ingesting data of particular partition to ADX, will delete/cleanup any other 
>> ingested data of other partitions. Since Azure Data Explorer is an append 
>> only database, unfortunately we cant perform delete on the ingested data of 
>> same partition or other partition. So to achieve this kind of 
>> transactionality of large ingested data, we are thinking to implement 
>> something similar we have done for Apache Spark ADX connector. We will be 
>> creating temporary tables inside Azure Data Explorer before ingesting to the 
>> actual tables. The worker nodes in apache spark will create these temporary 
>> tables and report the ingestion status to the driver node. The driver node 
>> on receiving the success status of all the worker nodes performs ingestion 
>> on the actual table or else the ingestion is aborted along with cleanup of 
>> temporary table. So basically we aggregate the worker node task status in 
>> the driver node in spark to take further decision on whether to ingest data 
>> into ADX table or not.
>> Question 1: Is this possible in Apache Nifi follows a zero master cluster 
>> strategy as opposed to master-slave architecture of apache spark?
>> Question 2: In our custom processor in Nifi, is it possible to run custom 
>> code of a particular partition say the cluster coordinator node. Also is it 
>> possible to get the details of the partition inside the processor?
>> Question 3: Is is to possible to get the details of tasks executed at the 
>> various partitions and take decisions based on the task status. Can all 
>> these be done inside the same processor.
>> 
>> Thanks,
>> Tanmaya

Re: Queries regarding Nifi custom processor for Azure Data Explorer

[Mike Thomsen]

I think that is at least partly doable within NiFi (ex yes, you can
restrict processors to the primary node in a cluster), but I would
recommend you consider a different approach for NiFi. Unlike Spark,
NiFi is perfectly content to stream in huge amounts of data
continuously without any temporary storage besides its repositories
(flowfile, content, etc). Therefore, I think a potentially easier
solution would be for your team to explore creating a connector
between NiFi and Azure Data Explorer that allows the latter to
firehose the former with data as it comes in and let the chips fall
where they may.

You might find some useful concepts in the Kafka processors for things
like managing a continuous flow of record data from a stream and
converting it to blocks of NiFi record data (see the Record API in our
documentation for details).

FWIW, I manage a data set in AWS w/ NiFi that is over 50TB compressed,
and a fairly generic 5 node NiFi cluster crushes that data on cheap
EC2 instances without issue. So handling TBs of data is something
fairly out of the box for NiFi if you're worried about that.

On Sat, Oct 1, 2022 at 12:05 AM Tanmaya Panda
 wrote:
>
> Hi Team,
>
> We at Microsoft opensource, are developing a custom for Azure Data Explorer 
> sink connector for Apache Nifi. What we want to achieve is transactionality 
> with data ingestion. The source of the processor can be TBs of telemetry data 
> as well as CDC logs. What that means is in case of any failure while 
> ingesting data of particular partition to ADX, will delete/cleanup any other 
> ingested data of other partitions. Since Azure Data Explorer is an append 
> only database, unfortunately we cant perform delete on the ingested data of 
> same partition or other partition. So to achieve this kind of 
> transactionality of large ingested data, we are thinking to implement 
> something similar we have done for Apache Spark ADX connector. We will be 
> creating temporary tables inside Azure Data Explorer before ingesting to the 
> actual tables. The worker nodes in apache spark will create these temporary 
> tables and report the ingestion status to the driver node. The driver node on 
> receiving the success status of all the worker nodes performs ingestion on 
> the actual table or else the ingestion is aborted along with cleanup of 
> temporary table. So basically we aggregate the worker node task status in the 
> driver node in spark to take further decision on whether to ingest data into 
> ADX table or not.
> Question 1: Is this possible in Apache Nifi follows a zero master cluster 
> strategy as opposed to master-slave architecture of apache spark?
> Question 2: In our custom processor in Nifi, is it possible to run custom 
> code of a particular partition say the cluster coordinator node. Also is it 
> possible to get the details of the partition inside the processor?
> Question 3: Is is to possible to get the details of tasks executed at the 
> various partitions and take decisions based on the task status. Can all these 
> be done inside the same processor.
>
> Thanks,
> Tanmaya

Queries regarding Nifi custom processor for Azure Data Explorer

[Tanmaya Panda]

Hi Team,

We at Microsoft opensource, are developing a custom for Azure Data Explorer 
sink connector for Apache Nifi. What we want to achieve is transactionality 
with data ingestion. The source of the processor can be TBs of telemetry data 
as well as CDC logs. What that means is in case of any failure while ingesting 
data of particular partition to ADX, will delete/cleanup any other ingested 
data of other partitions. Since Azure Data Explorer is an append only database, 
unfortunately we cant perform delete on the ingested data of same partition or 
other partition. So to achieve this kind of transactionality of large ingested 
data, we are thinking to implement something similar we have done for Apache 
Spark ADX connector. We will be creating temporary tables inside Azure Data 
Explorer before ingesting to the actual tables. The worker nodes in apache 
spark will create these temporary tables and report the ingestion status to the 
driver node. The driver node on receiving the success status of all the worker 
nodes performs ingestion on the actual table or else the ingestion is aborted 
along with cleanup of temporary table. So basically we aggregate the worker 
node task status in the driver node in spark to take further decision on 
whether to ingest data into ADX table or not.
Question 1: Is this possible in Apache Nifi follows a zero master cluster 
strategy as opposed to master-slave architecture of apache spark?
Question 2: In our custom processor in Nifi, is it possible to run custom code 
of a particular partition say the cluster coordinator node. Also is it possible 
to get the details of the partition inside the processor?
Question 3: Is is to possible to get the details of tasks executed at the 
various partitions and take decisions based on the task status. Can all these 
be done inside the same processor.

Thanks,
Tanmaya

Re: How to manage security artifacts from a custom processor

[Mike Thomsen]

> However, where/how do I install the certificates I'll be given for use?

The JDK keytool can be used to create the JKS artifacts you need.

On Tue, Jul 5, 2022 at 11:33 PM Russell Bateman  wrote:
>
> I appreciate the responses. I will try out the canonical
> /StandardSSLContextService/ first (since that's what I am using with
> Kafka), then imitate the other sample depending.
>
> However, where/how do I install the certificates I'll be given for use?
> I would expect something for certain representing the third-party
> service in a truststore and maybe another (a private key) in a keystore.
>
>
> On 7/5/22 16:30, Russell Bateman wrote:
> > From a custom processor, I intend to interface with a third-party
> > service (via simple HTTP client), however, I would need as I
> > understand it to
> >
> > a) maintain a private key by which I can identify myself to that
> > third-party service and
> > b) maintain a trusted-store certificate by which I can guarantee
> > the identity of the service.
> >
> > This is pretty far outside my own experience. I have been reading on
> > how this is achieved in Java, but in my mind a complication arises
> > from the fact that a custom NiFi processor lives within NiFi's JVM. My
> > question is therefore, how can I control the certificates and
> > authorities for my use in or associated with NiFi's JVM. Clearly, I
> > don't grok this well enough even to ask the question; I'm hoping
> > someone can see through what I'm asking and point me in a good
> > direction to study.
> >
> > I've written a pile of successful and useful custom NiFi processors to
> > cover proprietary needs, so custom-processor writing isn't a mystery.
> > Certificates, keys, trusts and security in general still is.
> >
> > Profuse thanks,
> >
> > Russ

Re: How to manage security artifacts from a custom processor

[Russell Bateman]

I appreciate the responses. I will try out the canonical 
/StandardSSLContextService/ first (since that's what I am using with 
Kafka), then imitate the other sample depending.


However, where/how do I install the certificates I'll be given for use? 
I would expect something for certain representing the third-party 
service in a truststore and maybe another (a private key) in a keystore.



On 7/5/22 16:30, Russell Bateman wrote:
From a custom processor, I intend to interface with a third-party 
service (via simple HTTP client), however, I would need as I 
understand it to


a) maintain a private key by which I can identify myself to that
third-party service and
b) maintain a trusted-store certificate by which I can guarantee
the identity of the service.

This is pretty far outside my own experience. I have been reading on 
how this is achieved in Java, but in my mind a complication arises 
from the fact that a custom NiFi processor lives within NiFi's JVM. My 
question is therefore, how can I control the certificates and 
authorities for my use in or associated with NiFi's JVM. Clearly, I 
don't grok this well enough even to ask the question; I'm hoping 
someone can see through what I'm asking and point me in a good 
direction to study.


I've written a pile of successful and useful custom NiFi processors to 
cover proprietary needs, so custom-processor writing isn't a mystery. 
Certificates, keys, trusts and security in general still is.


Profuse thanks,

Russ

Re: How to manage security artifacts from a custom processor

[Mike Thomsen]

Example of how to do this:

https://github.com/apache/nifi/blob/aa61494fc3a68b4806784f67ad837ee821d26da4/nifi-nar-bundles/nifi-standard-services/nifi-oauth2-provider-bundle/nifi-oauth2-provider-service/src/main/java/org/apache/nifi/oauth2/OAuth2TokenProviderImpl.java

On Tue, Jul 5, 2022 at 8:31 PM Otto Fowler  wrote:
>
>  Usually, you would write you custom processor to support the
> StandardSSLSocketService:
>
> https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-ssl-context-service-nar/1.16.3/org.apache.nifi.ssl.StandardSSLContextService/index.html
>
>
>
>
> From: Russell Bateman  
> Reply: dev@nifi.apache.org  
> Date: July 5, 2022 at 18:30:46
> To: NiFi Developers List  
> Subject:  How to manage security artifacts from a custom processor
>
> From a custom processor, I intend to interface with a third-party
> br/>servicee (via simple HTTP client), however, I would need as I
> understand br//>it to
>
> a) maintain a private key by which I can identify myself to that
> third-party service and
> b) maintain a trusted-store certificate by which I can guarantee the
> identity of the service.
>
> This is pretty far outside my own experience. I have been reading on how
> br/>this is achieved in Java, but in my mind a complication aarises from
> the br/>fact that a custom NiFFi processor lives within NiFi's JVM. My
> question br/>is therefore, how can I control the ceertificates and
> authorities for my br/>use in or associated with NiFFi's JVM. Clearly, I
> don't grok this well br/>enough even to ask the qquestion; I'm hoping
> someone can see through what br/>I'm asking and point me in a good
> direction to study.
>
> I've written a pile of successful and useful custom NiFi processors to
> br/>cover proprietary needs, so custom-processor writing isn''t a mystery.
> br/>Certificates, keys, trusts and security in general still is.
>
> Profuse thanks,
>
> Russ

Re: How to manage security artifacts from a custom processor

[Otto Fowler]

 Usually, you would write you custom processor to support the
StandardSSLSocketService:

https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-ssl-context-service-nar/1.16.3/org.apache.nifi.ssl.StandardSSLContextService/index.html




From: Russell Bateman  
Reply: dev@nifi.apache.org  
Date: July 5, 2022 at 18:30:46
To: NiFi Developers List  
Subject:  How to manage security artifacts from a custom processor

>From a custom processor, I intend to interface with a third-party
br/>servicee (via simple HTTP client), however, I would need as I
understand br//>it to

a) maintain a private key by which I can identify myself to that
third-party service and
b) maintain a trusted-store certificate by which I can guarantee the
identity of the service.

This is pretty far outside my own experience. I have been reading on how
br/>this is achieved in Java, but in my mind a complication aarises from
the br/>fact that a custom NiFFi processor lives within NiFi's JVM. My
question br/>is therefore, how can I control the ceertificates and
authorities for my br/>use in or associated with NiFFi's JVM. Clearly, I
don't grok this well br/>enough even to ask the qquestion; I'm hoping
someone can see through what br/>I'm asking and point me in a good
direction to study.

I've written a pile of successful and useful custom NiFi processors to
br/>cover proprietary needs, so custom-processor writing isn''t a mystery.
br/>Certificates, keys, trusts and security in general still is.

Profuse thanks,

Russ

How to manage security artifacts from a custom processor

[Russell Bateman]

From a custom processor, I intend to interface with a third-party 
service (via simple HTTP client), however, I would need as I understand 
it to


   a) maintain a private key by which I can identify myself to that
   third-party service and
   b) maintain a trusted-store certificate by which I can guarantee the
   identity of the service.

This is pretty far outside my own experience. I have been reading on how 
this is achieved in Java, but in my mind a complication arises from the 
fact that a custom NiFi processor lives within NiFi's JVM. My question 
is therefore, how can I control the certificates and authorities for my 
use in or associated with NiFi's JVM. Clearly, I don't grok this well 
enough even to ask the question; I'm hoping someone can see through what 
I'm asking and point me in a good direction to study.


I've written a pile of successful and useful custom NiFi processors to 
cover proprietary needs, so custom-processor writing isn't a mystery. 
Certificates, keys, trusts and security in general still is.


Profuse thanks,

Russ

Nifi custom processor to consume two flow files at once.

[Vibhath Ileperuma]

Hi All,

We are planning to develop a custom Nifi processor which consumes two files
at once.
We have a set of files which contains two types of data; say 'A' type data
and 'B' type data. This processor receives these two type files from two
upstream connections.
Processor needs to get one file from both the connections at once and do
the processing.
As I understand, even though we connect multiple upstream connections as
inputs to a processor, in the code, it treats all the data coming from a
single upstream queue.
Is there a way to specify the number of input connections to the processor
and take one file from each processor? If not, what is the flow file
reading order if multiple input connections are available for a processor.?

Thank You.

Best regards,

Vibhath.

Re: Custom Processor Query

[Simon Bence]

Hi Muhammad,

I am not sure what scenario you intend to cover, but in many cases, the 
“scripted” processor family is a satisfying solution to work with Python.

I suggest you to take a look of the InvokeScriptedProcessor and the 
ScriptedTransformRecord. Depending on your needs, they might help you to 
execute Python based logic.

Kind regards,
Bence

> On 2021. Sep 21., at 16:21, M. Adeel Ijaz  wrote:
> 
> To whom it may concern
> 
> Hi. I hope you are doing well. I want to define my custom processor in Nifi
> with Python. I couldn't find any source. Please let me know if you have any
> demo or guide regarding this. Looking forward to hearing from you soon.
> 
> Kind Regards
> Muhammad Adeel Ijaz

Custom Processor Query

[M. Adeel Ijaz]

To whom it may concern

Hi. I hope you are doing well. I want to define my custom processor in Nifi
with Python. I couldn't find any source. Please let me know if you have any
demo or guide regarding this. Looking forward to hearing from you soon.

Kind Regards
Muhammad Adeel Ijaz

Using JmsConnectionFactoryProvider controller service in custom processor

[Ashutosh Vaish]

=
Hi Team. 

I'm working on a JMS based custom processor which interacts with IBM MQ. I need 
to do a request reply over JMS IBM MQ queues. For this I wanted to leverage 
JndiJmsConnectionFactoryProvider of NiFi. I included the following dependencies 
into my processor nar, nifi-jms-cf-service-nar 
<https://mvnrepository.com/artifact/org.apache.nifi/nifi-jms-cf-service-nar> 
nifi-jms-processors-nar 
<https://mvnrepository.com/artifact/org.apache.nifi/nifi-jms-processors-nar>. 
Once I run the processor I get the below warning on the nifi-app.log WARN 
[main] o.a.n.n.StandardExtensionDiscoveringManager. Controller Service 
org.apache.nifi.jms.cf.JMSConnectionFactoryProvider is bundled with its 
supporting API's org.apache.nifi.jms.cf 
<http://org.apache.nifi.jms.cf/>.JMSConnectionFactoryProviderDefinition. The 
service API's shouldn't be bundled with the implementation. If I don't add the 
above mentioned nar I'm getting class not found errors for the referenced 
controller service JMSConnectionFactoryProviderDefinition. Please advice. 
(edited) 

Thanks
Ashutosh

NiFi Custom Processor Add in Our Major Flow

[Aswin K]

Hi Team,

I need an help as we have a requirement to add one additional processor in our 
major data flow. But I can create a custom processor and add it in the major 
flow.
As i am bit confused wheather in case of any issues occured in the JVM side 
where i can put some heavy lift in the major flow.

Could anybody please recommend wheather i can integrated the custom processor 
in the major flow or any other monitoring capabilites i can do before taking 
any decisions. Please help me out on this.

Re: How to restrict custom processor execution time

[sanjeet rath]

Thanks Chris & Joe for the wonderfull ideas.
Now i am clear how to acheive this functionality.

Regards,
Sanjeet

On Thu, 12 Aug 2021, 11:06 pm Chris Sampson,
 wrote:

> Consider using a static class member variable maybe (i.e. use a java-based
> solution instead of trying to find a nifi specific one). Bear in mind that
> "last execution" would then be reset if the node on which the processor is
> scheduled gets restarted. If you're in a clustered environment then the
> processor scheduling would move to another node automatically, but that may
> trigger another execution within the same hour.
>
> You could start using State in the processor (local or cluster) if you
> wanted a nifi-based solution, although you'll have to consider what happens
> when the state is reset, e.g. by a user in the UI or if the node on which
> the processor is scheduled changes within the cluster (and you're using
> local state) - would this be OK or cause a problem, etc.? Using
> remote/cluster state would allow the same "last execution" to be stored for
> all nodes across the cluster, so even if the processor scheduling is moved
> between nodes, it should be able to determine whether to run.
>
> There are @TriggerSerially and @PrimaryNodeOnly annotations for processors
> to ensure they can only be scheduled on a single node (current "primary")
> within a cluster and to only run one instance at any one time.
>
> @DefaultSchedule annotation can be used to configure the default processor
> schedule (e.g. every "1 hour"), but a user could still change that in the
> UI.
>
> Last bit of advice from me would be to call "context.yield" in "onTrigger"
> if the processor detects it should skip the current execution because it's
> within the same hour as the "last execution" - this allows nifi to schedule
> other processors to run instead of immediately retrying this processor and
> taking up one of the available threads in the pool (if someone accidentally
> configures it to run continually).
>
>
> Cheers,
>
> Chris Sampson
>
> On Thu, 12 Aug 2021, 18:05 sanjeet rath,  wrote:
>
> > Hi Joe,
> >
> > Keeping track of last execution & comparing with current time looks good
> > solution for my use case.There is no additional complication also.
> >
> > I am thinking of storing the "last execution time" in a processor
> > parameter.after every succesfull execution i will populate the value .so
> > Unless "current time" of execution > last execution + 60 min . The
> > processor won't do it function.
> >
> > Is there any way can i hide this parameter in UI.
> >
> > Regards,
> > Sanjeet
> >
> > On Thu, 12 Aug 2021, 10:18 pm Joe Witt,  wrote:
> >
> > > Sanjeet
> > >
> > > What about keeping track of 'last execution' and 'current time' to
> > > give you enough to decide if it has been at least 60 mins? Are there
> > > additional complications to consider?
> > >
> > > Thanks
> > >
> > > On Thu, Aug 12, 2021 at 9:43 AM sanjeet rath 
> > > wrote:
> > > >
> > > > Thanks , Joe for the quick reply.
> > > > I was wondering how can i figure out inside the processor's Ontrigger
> > > > method  when it was run previously.
> > > > Basicaly want to implement ur suggestion of restricting the execution
> > > > inside the processor's code .
> > > >
> > > > Regards,
> > > > Sanjeet
> > > >
> > > > On Thu, 12 Aug 2021, 9:43 pm Joe Witt,  wrote:
> > > >
> > > > > Sanjeet
> > > > >
> > > > > We dont presently allow the processor developer to put such a
> > > > > restriction into the code.  Your best bet for now would be to
> > document
> > > > > that the processor even if scheduled more than once in a one hour
> > > > > period will not execute its function more than once.  Then in your
> > > > > code you can protect/ensure it only truly does the execution once
> and
> > > > > hour.
> > > > >
> > > > > Thanks
> > > > > Joe
> > > > >
> > > > > On Thu, Aug 12, 2021 at 9:10 AM sanjeet rath <
> rath.sanj...@gmail.com
> > >
> > > > > wrote:
> > > > > >
> > > > > > Hi ,
> > > > > >
> > > > > > I am building a custom processor and there is restriction i want
> to
> > > put
> > > > > for
> > > > > > the  processor that it should not schedule to run 2 times in 1
> hour
> > > time
> > > > > > period.
> > > > > >
> > > > > > I can acheive this by passing "run schedule" 60 mins.
> > > > > >
> > > > > > Is there any other way i can do in my custom processor code, So
> > that
> > > it
> > > > > > won't allow the user to select "run schedule " time less than 60
> > > > > > mins.basically similar to we can restrict the procesor to execute
> > on
> > > > > > prinary node.
> > > > > >
> > > > > > Any other thought is really helpfull.
> > > > > >
> > > > > > Thanks,
> > > > > > Sanjeet
> > > > >
> > >
> >
>

Re: How to restrict custom processor execution time

[Chris Sampson]

Consider using a static class member variable maybe (i.e. use a java-based
solution instead of trying to find a nifi specific one). Bear in mind that
"last execution" would then be reset if the node on which the processor is
scheduled gets restarted. If you're in a clustered environment then the
processor scheduling would move to another node automatically, but that may
trigger another execution within the same hour.

You could start using State in the processor (local or cluster) if you
wanted a nifi-based solution, although you'll have to consider what happens
when the state is reset, e.g. by a user in the UI or if the node on which
the processor is scheduled changes within the cluster (and you're using
local state) - would this be OK or cause a problem, etc.? Using
remote/cluster state would allow the same "last execution" to be stored for
all nodes across the cluster, so even if the processor scheduling is moved
between nodes, it should be able to determine whether to run.

There are @TriggerSerially and @PrimaryNodeOnly annotations for processors
to ensure they can only be scheduled on a single node (current "primary")
within a cluster and to only run one instance at any one time.

@DefaultSchedule annotation can be used to configure the default processor
schedule (e.g. every "1 hour"), but a user could still change that in the
UI.

Last bit of advice from me would be to call "context.yield" in "onTrigger"
if the processor detects it should skip the current execution because it's
within the same hour as the "last execution" - this allows nifi to schedule
other processors to run instead of immediately retrying this processor and
taking up one of the available threads in the pool (if someone accidentally
configures it to run continually).


Cheers,

Chris Sampson

On Thu, 12 Aug 2021, 18:05 sanjeet rath,  wrote:

> Hi Joe,
>
> Keeping track of last execution & comparing with current time looks good
> solution for my use case.There is no additional complication also.
>
> I am thinking of storing the "last execution time" in a processor
> parameter.after every succesfull execution i will populate the value .so
> Unless "current time" of execution > last execution + 60 min . The
> processor won't do it function.
>
> Is there any way can i hide this parameter in UI.
>
> Regards,
> Sanjeet
>
> On Thu, 12 Aug 2021, 10:18 pm Joe Witt,  wrote:
>
> > Sanjeet
> >
> > What about keeping track of 'last execution' and 'current time' to
> > give you enough to decide if it has been at least 60 mins? Are there
> > additional complications to consider?
> >
> > Thanks
> >
> > On Thu, Aug 12, 2021 at 9:43 AM sanjeet rath 
> > wrote:
> > >
> > > Thanks , Joe for the quick reply.
> > > I was wondering how can i figure out inside the processor's Ontrigger
> > > method  when it was run previously.
> > > Basicaly want to implement ur suggestion of restricting the execution
> > > inside the processor's code .
> > >
> > > Regards,
> > > Sanjeet
> > >
> > > On Thu, 12 Aug 2021, 9:43 pm Joe Witt,  wrote:
> > >
> > > > Sanjeet
> > > >
> > > > We dont presently allow the processor developer to put such a
> > > > restriction into the code.  Your best bet for now would be to
> document
> > > > that the processor even if scheduled more than once in a one hour
> > > > period will not execute its function more than once.  Then in your
> > > > code you can protect/ensure it only truly does the execution once and
> > > > hour.
> > > >
> > > > Thanks
> > > > Joe
> > > >
> > > > On Thu, Aug 12, 2021 at 9:10 AM sanjeet rath  >
> > > > wrote:
> > > > >
> > > > > Hi ,
> > > > >
> > > > > I am building a custom processor and there is restriction i want to
> > put
> > > > for
> > > > > the  processor that it should not schedule to run 2 times in 1 hour
> > time
> > > > > period.
> > > > >
> > > > > I can acheive this by passing "run schedule" 60 mins.
> > > > >
> > > > > Is there any other way i can do in my custom processor code, So
> that
> > it
> > > > > won't allow the user to select "run schedule " time less than 60
> > > > > mins.basically similar to we can restrict the procesor to execute
> on
> > > > > prinary node.
> > > > >
> > > > > Any other thought is really helpfull.
> > > > >
> > > > > Thanks,
> > > > > Sanjeet
> > > >
> >
>

Re: How to restrict custom processor execution time

[sanjeet rath]

Hi Joe,

Keeping track of last execution & comparing with current time looks good
solution for my use case.There is no additional complication also.

I am thinking of storing the "last execution time" in a processor
parameter.after every succesfull execution i will populate the value .so
Unless "current time" of execution > last execution + 60 min . The
processor won't do it function.

Is there any way can i hide this parameter in UI.

Regards,
Sanjeet

On Thu, 12 Aug 2021, 10:18 pm Joe Witt,  wrote:

> Sanjeet
>
> What about keeping track of 'last execution' and 'current time' to
> give you enough to decide if it has been at least 60 mins? Are there
> additional complications to consider?
>
> Thanks
>
> On Thu, Aug 12, 2021 at 9:43 AM sanjeet rath 
> wrote:
> >
> > Thanks , Joe for the quick reply.
> > I was wondering how can i figure out inside the processor's Ontrigger
> > method  when it was run previously.
> > Basicaly want to implement ur suggestion of restricting the execution
> > inside the processor's code .
> >
> > Regards,
> > Sanjeet
> >
> > On Thu, 12 Aug 2021, 9:43 pm Joe Witt,  wrote:
> >
> > > Sanjeet
> > >
> > > We dont presently allow the processor developer to put such a
> > > restriction into the code.  Your best bet for now would be to document
> > > that the processor even if scheduled more than once in a one hour
> > > period will not execute its function more than once.  Then in your
> > > code you can protect/ensure it only truly does the execution once and
> > > hour.
> > >
> > > Thanks
> > > Joe
> > >
> > > On Thu, Aug 12, 2021 at 9:10 AM sanjeet rath 
> > > wrote:
> > > >
> > > > Hi ,
> > > >
> > > > I am building a custom processor and there is restriction i want to
> put
> > > for
> > > > the  processor that it should not schedule to run 2 times in 1 hour
> time
> > > > period.
> > > >
> > > > I can acheive this by passing "run schedule" 60 mins.
> > > >
> > > > Is there any other way i can do in my custom processor code, So that
> it
> > > > won't allow the user to select "run schedule " time less than 60
> > > > mins.basically similar to we can restrict the procesor to execute on
> > > > prinary node.
> > > >
> > > > Any other thought is really helpfull.
> > > >
> > > > Thanks,
> > > > Sanjeet
> > >
>

Re: How to restrict custom processor execution time

[Joe Witt]

Sanjeet

What about keeping track of 'last execution' and 'current time' to
give you enough to decide if it has been at least 60 mins? Are there
additional complications to consider?

Thanks

On Thu, Aug 12, 2021 at 9:43 AM sanjeet rath  wrote:
>
> Thanks , Joe for the quick reply.
> I was wondering how can i figure out inside the processor's Ontrigger
> method  when it was run previously.
> Basicaly want to implement ur suggestion of restricting the execution
> inside the processor's code .
>
> Regards,
> Sanjeet
>
> On Thu, 12 Aug 2021, 9:43 pm Joe Witt,  wrote:
>
> > Sanjeet
> >
> > We dont presently allow the processor developer to put such a
> > restriction into the code.  Your best bet for now would be to document
> > that the processor even if scheduled more than once in a one hour
> > period will not execute its function more than once.  Then in your
> > code you can protect/ensure it only truly does the execution once and
> > hour.
> >
> > Thanks
> > Joe
> >
> > On Thu, Aug 12, 2021 at 9:10 AM sanjeet rath 
> > wrote:
> > >
> > > Hi ,
> > >
> > > I am building a custom processor and there is restriction i want to put
> > for
> > > the  processor that it should not schedule to run 2 times in 1 hour time
> > > period.
> > >
> > > I can acheive this by passing "run schedule" 60 mins.
> > >
> > > Is there any other way i can do in my custom processor code, So that it
> > > won't allow the user to select "run schedule " time less than 60
> > > mins.basically similar to we can restrict the procesor to execute on
> > > prinary node.
> > >
> > > Any other thought is really helpfull.
> > >
> > > Thanks,
> > > Sanjeet
> >

Re: How to restrict custom processor execution time

[sanjeet rath]

Thanks , Joe for the quick reply.
I was wondering how can i figure out inside the processor's Ontrigger
method  when it was run previously.
Basicaly want to implement ur suggestion of restricting the execution
inside the processor's code .

Regards,
Sanjeet

On Thu, 12 Aug 2021, 9:43 pm Joe Witt,  wrote:

> Sanjeet
>
> We dont presently allow the processor developer to put such a
> restriction into the code.  Your best bet for now would be to document
> that the processor even if scheduled more than once in a one hour
> period will not execute its function more than once.  Then in your
> code you can protect/ensure it only truly does the execution once and
> hour.
>
> Thanks
> Joe
>
> On Thu, Aug 12, 2021 at 9:10 AM sanjeet rath 
> wrote:
> >
> > Hi ,
> >
> > I am building a custom processor and there is restriction i want to put
> for
> > the  processor that it should not schedule to run 2 times in 1 hour time
> > period.
> >
> > I can acheive this by passing "run schedule" 60 mins.
> >
> > Is there any other way i can do in my custom processor code, So that it
> > won't allow the user to select "run schedule " time less than 60
> > mins.basically similar to we can restrict the procesor to execute on
> > prinary node.
> >
> > Any other thought is really helpfull.
> >
> > Thanks,
> > Sanjeet
>

Re: How to restrict custom processor execution time

[Joe Witt]

Sanjeet

We dont presently allow the processor developer to put such a
restriction into the code.  Your best bet for now would be to document
that the processor even if scheduled more than once in a one hour
period will not execute its function more than once.  Then in your
code you can protect/ensure it only truly does the execution once and
hour.

Thanks
Joe

On Thu, Aug 12, 2021 at 9:10 AM sanjeet rath  wrote:
>
> Hi ,
>
> I am building a custom processor and there is restriction i want to put for
> the  processor that it should not schedule to run 2 times in 1 hour time
> period.
>
> I can acheive this by passing "run schedule" 60 mins.
>
> Is there any other way i can do in my custom processor code, So that it
> won't allow the user to select "run schedule " time less than 60
> mins.basically similar to we can restrict the procesor to execute on
> prinary node.
>
> Any other thought is really helpfull.
>
> Thanks,
> Sanjeet

How to restrict custom processor execution time

[sanjeet rath]

Hi ,

I am building a custom processor and there is restriction i want to put for
the  processor that it should not schedule to run 2 times in 1 hour time
period.

I can acheive this by passing "run schedule" 60 mins.

Is there any other way i can do in my custom processor code, So that it
won't allow the user to select "run schedule " time less than 60
mins.basically similar to we can restrict the procesor to execute on
prinary node.

Any other thought is really helpfull.

Thanks,
Sanjeet

Developing a custom processor which manages it state

[Vibhath Ileperuma]

Hi all,

I'm developing a custom processor which requires state management.
According to the NIFI developer guide, the entire state map should be less
than 1MB if we are doing the state management in cluster scope. However,
since the state map can be grown more than 1MB in my case, I'm planning to
use an external db which should be connected via a JDBC connection.
Here, can we configure NIFI to use an external db for state management
instead of default zookeeper mechanism? Or do I need to implement the whole
state management by myself?
If it is required to implement the state management by myself, I'm
wondering if you can point out the things which I should be careful of.

Thank You.
Best Regards,
Vibhath.

Re: Failing to update custom processor properties names, displayNames, etc.

[Russell Bateman]

Andy,

You're right. I was a caching issue with Chrome. I don't know why I 
didn't think to try that--forest for the trees, I guess. Thank you.


Russ

On 7/17/20 5:58 PM, Andy LoPresto wrote:

Russell,

Have you verified this is not a browser caching issue? They are pernicious and 
it sounds like this could be an example. If you’re sure it’s not, verify the 
API calls using your browser’s Developer Tools to see what properties are 
actually being returned by the server when inspecting a component to see if the 
correct values are present. If so, it’s likely a UI bug, and if not, the new 
code is not being properly loaded/used by NiFi’s server application.


Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69


On Jul 17, 2020, at 3:48 PM, Russell Bateman  wrote:

If I have changed a custom processor's PropertyDescriptor.nameand/or 
.displayName,including changes I have made to my /additionalDetails.html, /and 
I have:

- removed that processor from my test flow or removed /flow.xml.gz/ altogether
- removed my NAR from /${NIFI_ROOT}///custom-lib/ and bounced NiFi with the new 
custom NAR copied

yet nothing changes (in usage or in additional details), what am I overlooking?

Let's set aside the vaguely confusing semantic distinction between nameand 
displayName, I just want NiFi to forget my new custom processor completely and 
then accept my new version as if brand new including all the changes I have 
made.

Thanks for any suggestions.

Re: Failing to update custom processor properties names, displayNames, etc.

[Andy LoPresto]

Russell, 

Have you verified this is not a browser caching issue? They are pernicious and 
it sounds like this could be an example. If you’re sure it’s not, verify the 
API calls using your browser’s Developer Tools to see what properties are 
actually being returned by the server when inspecting a component to see if the 
correct values are present. If so, it’s likely a UI bug, and if not, the new 
code is not being properly loaded/used by NiFi’s server application. 


Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
He/Him
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Jul 17, 2020, at 3:48 PM, Russell Bateman  wrote:
> 
> If I have changed a custom processor's PropertyDescriptor.nameand/or 
> .displayName,including changes I have made to my /additionalDetails.html, 
> /and I have:
> 
> - removed that processor from my test flow or removed flow.xml.gz altogether
> - removed my NAR from /${NIFI_ROOT}///custom-lib/ and bounced NiFi with the 
> new custom NAR copied
> 
> yet nothing changes (in usage or in additional details), what am I 
> overlooking?
> 
> Let's set aside the vaguely confusing semantic distinction between nameand 
> displayName, I just want NiFi to forget my new custom processor completely 
> and then accept my new version as if brand new including all the changes I 
> have made.
> 
> Thanks for any suggestions.

Failing to update custom processor properties names, displayNames, etc.

[Russell Bateman]

If I have changed a custom processor's PropertyDescriptor.nameand/or 
.displayName,including changes I have made to my 
/additionalDetails.html, /and I have:


- removed that processor from my test flow or removed flow.xml.gz altogether
- removed my NAR from /${NIFI_ROOT}///custom-lib/ and bounced NiFi with 
the new custom NAR copied


yet nothing changes (in usage or in additional details), what am I 
overlooking?


Let's set aside the vaguely confusing semantic distinction between 
nameand displayName, I just want NiFi to forget my new custom processor 
completely and then accept my new version as if brand new including all 
the changes I have made.


Thanks for any suggestions.

Re: Duplicate Artifacts from Custom Processor

[Bryan Bende]

Hello,

This likely means you have a dependency somewhere on the standard
processors, which you shouldn’t have.

Thanks,

Bryan

On Fri, Jul 10, 2020 at 7:23 AM Riley, Andrew
 wrote:

> I’ve created a Custom Processor that include access to the
> SSLContextService Controller Service.
>
>
>
> Including the standard service api nar as a dependency in the nar pom as
> described;
> https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions#MavenProjectsforExtensions-LinkingProcessorsandControllerServices
>
> Allows the nar to build and work. However my Nifi instance now has
> duplicates of many common processors with my Custom processor version
> number, as shown below. How can this be avoided?
>
>
>
>
>
>
>
> I’ve also added the question to StackOverflow, but there doesn’t seem to
> be much NiFi traffic there.
>
>
> https://stackoverflow.com/questions/62831723/nifi-custom-processor-introduces-duplicates-of-standard-artifacts
>
>
>
> Kind Regards,
>
> Andrew
>
>
>
>
>
> --
>
> Capgemini is a trading name used by the Capgemini Group of companies which
> includes Capgemini UK plc, a company registered in England and Wales
> (number 943935) whose registered office is at No. 1, Forge End, Woking,
> Surrey, GU21 6DB
> <https://www.google.com/maps/search/No.+1,+Forge+End,+Woking,+Surrey,+GU21+6DB?entry=gmail=g>
> .
> This message contains information that may be privileged or confidential
> and is the property of the Capgemini Group. It is intended only for the
> person to whom it is addressed. If you are not the intended recipient, you
> are not authorized to read, print, retain, copy, disseminate, distribute,
> or use this message or any part thereof. If you receive this message in
> error, please notify the sender immediately and delete all copies of this
> message.
>
-- 
Sent from Gmail Mobile

Duplicate Artifacts from Custom Processor

[Riley, Andrew]

I've created a Custom Processor that include access to the SSLContextService 
Controller Service.

Including the standard service api nar as a dependency in the nar pom as 
described; 
https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions#MavenProjectsforExtensions-LinkingProcessorsandControllerServices
Allows the nar to build and work. However my Nifi instance now has duplicates 
of many common processors with my Custom processor version number, as shown 
below. How can this be avoided?

[cid:image001.jpg@01D656A9.986117D0]


I've also added the question to StackOverflow, but there doesn't seem to be 
much NiFi traffic there.
https://stackoverflow.com/questions/62831723/nifi-custom-processor-introduces-duplicates-of-standard-artifacts

Kind Regards,
Andrew





Capgemini is a trading name used by the Capgemini Group of companies which 
includes Capgemini UK plc, a company registered in England and Wales (number 
943935) whose registered office is at No. 1, Forge End, Woking, Surrey, GU21 
6DB.
This message contains information that may be privileged or confidential and is 
the property of the Capgemini Group. It is intended only for the person to whom 
it is addressed. If you are not the intended recipient, you are not authorized 
to read, print, retain, copy, disseminate, distribute, or use this message or 
any part thereof. If you receive this message in error, please notify the 
sender immediately and delete all copies of this message.

Re: Managing custom processor logging statements during JUnit testing

[Russell Bateman]

Profuse thanks, Andy. I will get back to confirm this. Something else 
came up in the meantime.


On 3/9/20 1:35 PM, Andy LoPresto wrote:

Russell,

You can put a file called logback-test.xml in the src/test/resources/ directory 
of your custom processor module. This file is the same format as logback.xml 
but will supersede it for unit test execution. You can configure whatever level 
of logging severity and specificity in the same way you would in the production 
file.

Here is an example in nifi-security-utils [1].

[1] 
https://github.com/apache/nifi/blob/master/nifi-commons/nifi-security-utils/src/test/resources/logback-test.xml
 
<https://github.com/apache/nifi/blob/master/nifi-commons/nifi-security-utils/src/test/resources/logback-test.xml>

Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69


On Mar 6, 2020, at 12:13 PM, Russell Bateman  wrote:

I'm interested in getting log statements that may occur from my *custom 
processor* code at the INFO, DEBUGand TRACElevels when running unit tests for 
it. This means that I would like to set (programmatically at runtime or by 
configuration) what I am used to setting in /${NIFI_ROOT}///conf/logback.xml/, 
but effectively in/during my unit tests (using NiFi's TestRunner). And, 
obviously, I'd like to see those log statements come out to the console.

Thanks for any advice.

Re: Managing custom processor logging statements during JUnit testing

[Andy LoPresto]

Russell, 

You can put a file called logback-test.xml in the src/test/resources/ directory 
of your custom processor module. This file is the same format as logback.xml 
but will supersede it for unit test execution. You can configure whatever level 
of logging severity and specificity in the same way you would in the production 
file. 

Here is an example in nifi-security-utils [1]. 

[1] 
https://github.com/apache/nifi/blob/master/nifi-commons/nifi-security-utils/src/test/resources/logback-test.xml
 
<https://github.com/apache/nifi/blob/master/nifi-commons/nifi-security-utils/src/test/resources/logback-test.xml>

Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Mar 6, 2020, at 12:13 PM, Russell Bateman  wrote:
> 
> I'm interested in getting log statements that may occur from my *custom 
> processor* code at the INFO, DEBUGand TRACElevels when running unit tests for 
> it. This means that I would like to set (programmatically at runtime or by 
> configuration) what I am used to setting in 
> /${NIFI_ROOT}///conf/logback.xml/, but effectively in/during my unit tests 
> (using NiFi's TestRunner). And, obviously, I'd like to see those log 
> statements come out to the console.
> 
> Thanks for any advice.
> 

Managing custom processor logging statements during JUnit testing

[Russell Bateman]

I'm interested in getting log statements that may occur from my *custom 
processor* code at the INFO, DEBUGand TRACElevels when running unit 
tests for it. This means that I would like to set (programmatically at 
runtime or by configuration) what I am used to setting in 
/${NIFI_ROOT}///conf/logback.xml/, but effectively in/during my unit 
tests (using NiFi's TestRunner). And, obviously, I'd like to see those 
log statements come out to the console.


Thanks for any advice.

Custom Processor Upgrade

[Bimal Mehta]

Hi,

If we have a custom processor that was created with NiFi 1.6, what are the
steps we need to follow to make it work in 1.9?
Is there some sort of steps that explains the jar and pom updates we need
to do for making it work in 1.9?

Re: question about Nifi Custom processor

[Mike Thomsen]

   session.transfer(output, REL_ORIGINAL);

should be

   session.transfer(inputFlowFile, REL_ORIGINAL);

On Wed, Jul 24, 2019 at 3:35 PM Mike Thomsen  wrote:

> Also, here is a pattern we tend to use for reading using try-with
>
>
> FlowFile output = session.create(inputFlowFile); //create linkage in
> provenance
> try (InputStream is = session.read(inputFlowFile); OutputStream os =
> session.write(output);) {
> //Do stuff with streams.
>
> is.close();
>os.close(); //These need to be closed before transfer
>session.transfer(output, REL_SUCCESS);
>session.transfer(output, REL_ORIGINAL);
> } catch (Exception ex) {
> getLogger().error("", ex);
> session.remove(output);
> session.transfer(inputFlowFile, REL_ERROR);
> }
>
> On Wed, Jul 24, 2019 at 2:28 PM Mike Thomsen 
> wrote:
>
>> Two problems I see right here are this:
>>
>> flowfile = session.create();
>> flowfile = session.get();
>>
>>
>> The second call will replace the newly created flowfile that comes from
>> session.create() with either null or the next element in the queue. That
>> will result in you having a flowfile created in the session that you cannot
>> either transfer to a downstream relationship with session.transfer() or
>> remove with session.remove(). End result there will be the session will
>> fail to commit because it doesn't know what it should do with that empty
>> flowfile.
>>
>> Second:
>>
>> new FileReader("JSON_PATH")
>>
>>
>> That will result in java.io.FileReader looking for a file named
>> "JSON_PATH" in the current working folder. Unless that is your intent, you
>> need to remove the quotes and make sure it's pointing to a real path on the
>> file system.
>>
>> On Tue, Jul 23, 2019 at 10:13 PM Farooq Mustafa 
>> wrote:
>>
>>> Hello,
>>>
>>>
>>> I am writing my first custom processor. I followed the example here
>>> https://www.nifi.rocks/developing-a-custom-apache-nifi-processor-json/
>>>
>>>
>>> In the settings, I have /tmp/jsoniput/sample1.json file
>>>
>>>
>>> The source code;
>>>
>>> public void onTrigger(final ProcessContext context, final ProcessSession 
>>> session) throws ProcessException {
>>>
>>>   final AtomicReference value = new AtomicReference<>();
>>>   getLogger().warn("sample - Farooq - 1 , value = <" + value.toString() + 
>>> ">");
>>>
>>>   FlowFile flowfile = session.get();
>>>   getLogger().warn("sample - Farooq - 2");
>>>
>>>   if ( null == flowfile) {
>>> getLogger().warn("sample - Farooq - 2.1 - flowfile is NULL *** ");
>>> flowfile = session.create();
>>> flowfile = session.get();
>>> getLogger().warn("sample - Farooq - 2.2 - session.create() / 
>>> session.get() --- NOT SURE ");
>>>   }
>>>   getLogger().warn("sample - Farooq - 2.3 JSON_PATH :" + 
>>> context.getProperty(JSON_PATH).getValue());
>>>
>>>   session.read(flowfile, new InputStreamCallback() {
>>> @Override
>>> public void process(InputStream in) throws IOException {
>>>
>>>   try {
>>> getLogger().warn("sample - Farooq - 3 JSON_PATH :" + 
>>> context.getProperty(JSON_PATH).getValue());
>>> Object obj = new JSONParser().parse(new FileReader("JSON_PATH"));
>>> getLogger().warn("sample - Farooq - 3.1 ");
>>>
>>>
>>>
>>> The log output:
>>>
>>> 2019-07-20 13:58:42,625 WARN [Timer-Driven Process Thread-4]
>>> o.a.nifi.processors.sitc.JsonProcessor
>>> JsonProcessor[id=fd8fe4ca-016b-1000-a6ed-c0e224bc7267] sample - Farooq - 1
>>> , value = 
>>>
>>> 2019-07-20 13:58:42,625 WARN [Timer-Driven Process Thread-4]
>>> o.a.nifi.processors.sitc.JsonProcessor
>>> JsonProcessor[id=fd8fe4ca-016b-1000-a6ed-c0e224bc7267] sample - Farooq - 2
>>>
>>> 2019-07-20 13:58:42,625 WARN [Timer-Driven Process Thread-4]
>>> o.a.nifi.processors.sitc.JsonProcessor
>>> JsonProcessor[id=fd8fe4ca-016b-1000-a6ed-c0e224bc7267] sample - Farooq -
>>> 2.1 - flowfile is NULL ***
>>>
>>> 2019-07-20 13:58:42,625 WARN [Timer-Driven Process Thread-4]
>>> o.a.nifi.processors.sitc.JsonProcessor
>>> JsonProcessor[id=fd8fe4ca-016b-1000-a6ed-c0e224bc7267] sample - Farooq -
>>> 2.2 - s

Re: question about Nifi Custom processor

[Mike Thomsen]

Also, here is a pattern we tend to use for reading using try-with


FlowFile output = session.create(inputFlowFile); //create linkage in
provenance
try (InputStream is = session.read(inputFlowFile); OutputStream os =
session.write(output);) {
//Do stuff with streams.

is.close();
   os.close(); //These need to be closed before transfer
   session.transfer(output, REL_SUCCESS);
   session.transfer(output, REL_ORIGINAL);
} catch (Exception ex) {
getLogger().error("", ex);
session.remove(output);
session.transfer(inputFlowFile, REL_ERROR);
}

On Wed, Jul 24, 2019 at 2:28 PM Mike Thomsen  wrote:

> Two problems I see right here are this:
>
> flowfile = session.create();
> flowfile = session.get();
>
>
> The second call will replace the newly created flowfile that comes from
> session.create() with either null or the next element in the queue. That
> will result in you having a flowfile created in the session that you cannot
> either transfer to a downstream relationship with session.transfer() or
> remove with session.remove(). End result there will be the session will
> fail to commit because it doesn't know what it should do with that empty
> flowfile.
>
> Second:
>
> new FileReader("JSON_PATH")
>
>
> That will result in java.io.FileReader looking for a file named
> "JSON_PATH" in the current working folder. Unless that is your intent, you
> need to remove the quotes and make sure it's pointing to a real path on the
> file system.
>
> On Tue, Jul 23, 2019 at 10:13 PM Farooq Mustafa 
> wrote:
>
>> Hello,
>>
>>
>> I am writing my first custom processor. I followed the example here
>> https://www.nifi.rocks/developing-a-custom-apache-nifi-processor-json/
>>
>>
>> In the settings, I have /tmp/jsoniput/sample1.json file
>>
>>
>> The source code;
>>
>> public void onTrigger(final ProcessContext context, final ProcessSession 
>> session) throws ProcessException {
>>
>>   final AtomicReference value = new AtomicReference<>();
>>   getLogger().warn("sample - Farooq - 1 , value = <" + value.toString() + 
>> ">");
>>
>>   FlowFile flowfile = session.get();
>>   getLogger().warn("sample - Farooq - 2");
>>
>>   if ( null == flowfile) {
>> getLogger().warn("sample - Farooq - 2.1 - flowfile is NULL *** ");
>> flowfile = session.create();
>> flowfile = session.get();
>> getLogger().warn("sample - Farooq - 2.2 - session.create() / 
>> session.get() --- NOT SURE ");
>>   }
>>   getLogger().warn("sample - Farooq - 2.3 JSON_PATH :" + 
>> context.getProperty(JSON_PATH).getValue());
>>
>>   session.read(flowfile, new InputStreamCallback() {
>> @Override
>> public void process(InputStream in) throws IOException {
>>
>>   try {
>> getLogger().warn("sample - Farooq - 3 JSON_PATH :" + 
>> context.getProperty(JSON_PATH).getValue());
>> Object obj = new JSONParser().parse(new FileReader("JSON_PATH"));
>> getLogger().warn("sample - Farooq - 3.1 ");
>>
>>
>>
>> The log output:
>>
>> 2019-07-20 13:58:42,625 WARN [Timer-Driven Process Thread-4]
>> o.a.nifi.processors.sitc.JsonProcessor
>> JsonProcessor[id=fd8fe4ca-016b-1000-a6ed-c0e224bc7267] sample - Farooq - 1
>> , value = 
>>
>> 2019-07-20 13:58:42,625 WARN [Timer-Driven Process Thread-4]
>> o.a.nifi.processors.sitc.JsonProcessor
>> JsonProcessor[id=fd8fe4ca-016b-1000-a6ed-c0e224bc7267] sample - Farooq - 2
>>
>> 2019-07-20 13:58:42,625 WARN [Timer-Driven Process Thread-4]
>> o.a.nifi.processors.sitc.JsonProcessor
>> JsonProcessor[id=fd8fe4ca-016b-1000-a6ed-c0e224bc7267] sample - Farooq -
>> 2.1 - flowfile is NULL ***
>>
>> 2019-07-20 13:58:42,625 WARN [Timer-Driven Process Thread-4]
>> o.a.nifi.processors.sitc.JsonProcessor
>> JsonProcessor[id=fd8fe4ca-016b-1000-a6ed-c0e224bc7267] sample - Farooq -
>> 2.2 - session.create() / session.get() --- NOT SURE
>>
>> 2019-07-20 13:58:42,625 WARN [Timer-Driven Process Thread-4]
>> o.a.nifi.processors.sitc.JsonProcessor
>> JsonProcessor[id=fd8fe4ca-016b-1000-a6ed-c0e224bc7267] sample - Farooq -
>> 2.3 JSON_PATH :/tmp/jsoninput
>>
>> 2019-07-20 13:58:42,626 ERROR [Timer-Driven Process Thread-4]
>> o.a.nifi.processors.sitc.JsonProcessor
>> JsonProcessor[id=fd8fe4ca-016b-1000-a6ed-c0e224bc7267]
>> JsonProcessor[id=fd8fe4ca-016b-1000-a6ed-c0e224bc7267] failed to process
>> session due to java.lang.NullPointerException; Processor Administratively
>> Yielded for 1 sec: java.lang.NullPointerException
>>
>> java.lang.NullPointerException: null
>>
>> at
>> org.apache.nifi.controller.repository.StandardProcessSession.getRecord(StandardProcessSession.java:574)
>>
>> at
>> org.apache.nifi.controller.repository.StandardProcessSession.validateRecordState(StandardProcessSession.java:3132)
>>
>> at
>> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2187)
>>
>>
>> And the basic flow:
>> [image: image.png]
>>
>>
>>
>> Any ideas or guidance will be appreciated.
>>
>>
>> Thanks
>>
>> Farooq
>>
>

Re: question about Nifi Custom processor

[Mike Thomsen]

Two problems I see right here are this:

flowfile = session.create();
flowfile = session.get();


The second call will replace the newly created flowfile that comes from
session.create() with either null or the next element in the queue. That
will result in you having a flowfile created in the session that you cannot
either transfer to a downstream relationship with session.transfer() or
remove with session.remove(). End result there will be the session will
fail to commit because it doesn't know what it should do with that empty
flowfile.

Second:

new FileReader("JSON_PATH")


That will result in java.io.FileReader looking for a file named "JSON_PATH"
in the current working folder. Unless that is your intent, you need to
remove the quotes and make sure it's pointing to a real path on the file
system.

On Tue, Jul 23, 2019 at 10:13 PM Farooq Mustafa 
wrote:

> Hello,
>
>
> I am writing my first custom processor. I followed the example here
> https://www.nifi.rocks/developing-a-custom-apache-nifi-processor-json/
>
>
> In the settings, I have /tmp/jsoniput/sample1.json file
>
>
> The source code;
>
> public void onTrigger(final ProcessContext context, final ProcessSession 
> session) throws ProcessException {
>
>   final AtomicReference value = new AtomicReference<>();
>   getLogger().warn("sample - Farooq - 1 , value = <" + value.toString() + 
> ">");
>
>   FlowFile flowfile = session.get();
>   getLogger().warn("sample - Farooq - 2");
>
>   if ( null == flowfile) {
> getLogger().warn("sample - Farooq - 2.1 - flowfile is NULL *** ");
> flowfile = session.create();
> flowfile = session.get();
> getLogger().warn("sample - Farooq - 2.2 - session.create() / 
> session.get() --- NOT SURE ");
>   }
>   getLogger().warn("sample - Farooq - 2.3 JSON_PATH :" + 
> context.getProperty(JSON_PATH).getValue());
>
>   session.read(flowfile, new InputStreamCallback() {
> @Override
> public void process(InputStream in) throws IOException {
>
>   try {
> getLogger().warn("sample - Farooq - 3 JSON_PATH :" + 
> context.getProperty(JSON_PATH).getValue());
> Object obj = new JSONParser().parse(new FileReader("JSON_PATH"));
> getLogger().warn("sample - Farooq - 3.1 ");
>
>
>
> The log output:
>
> 2019-07-20 13:58:42,625 WARN [Timer-Driven Process Thread-4]
> o.a.nifi.processors.sitc.JsonProcessor
> JsonProcessor[id=fd8fe4ca-016b-1000-a6ed-c0e224bc7267] sample - Farooq - 1
> , value = 
>
> 2019-07-20 13:58:42,625 WARN [Timer-Driven Process Thread-4]
> o.a.nifi.processors.sitc.JsonProcessor
> JsonProcessor[id=fd8fe4ca-016b-1000-a6ed-c0e224bc7267] sample - Farooq - 2
>
> 2019-07-20 13:58:42,625 WARN [Timer-Driven Process Thread-4]
> o.a.nifi.processors.sitc.JsonProcessor
> JsonProcessor[id=fd8fe4ca-016b-1000-a6ed-c0e224bc7267] sample - Farooq -
> 2.1 - flowfile is NULL ***
>
> 2019-07-20 13:58:42,625 WARN [Timer-Driven Process Thread-4]
> o.a.nifi.processors.sitc.JsonProcessor
> JsonProcessor[id=fd8fe4ca-016b-1000-a6ed-c0e224bc7267] sample - Farooq -
> 2.2 - session.create() / session.get() --- NOT SURE
>
> 2019-07-20 13:58:42,625 WARN [Timer-Driven Process Thread-4]
> o.a.nifi.processors.sitc.JsonProcessor
> JsonProcessor[id=fd8fe4ca-016b-1000-a6ed-c0e224bc7267] sample - Farooq -
> 2.3 JSON_PATH :/tmp/jsoninput
>
> 2019-07-20 13:58:42,626 ERROR [Timer-Driven Process Thread-4]
> o.a.nifi.processors.sitc.JsonProcessor
> JsonProcessor[id=fd8fe4ca-016b-1000-a6ed-c0e224bc7267]
> JsonProcessor[id=fd8fe4ca-016b-1000-a6ed-c0e224bc7267] failed to process
> session due to java.lang.NullPointerException; Processor Administratively
> Yielded for 1 sec: java.lang.NullPointerException
>
> java.lang.NullPointerException: null
>
> at
> org.apache.nifi.controller.repository.StandardProcessSession.getRecord(StandardProcessSession.java:574)
>
> at
> org.apache.nifi.controller.repository.StandardProcessSession.validateRecordState(StandardProcessSession.java:3132)
>
> at
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2187)
>
>
> And the basic flow:
> [image: image.png]
>
>
>
> Any ideas or guidance will be appreciated.
>
>
> Thanks
>
> Farooq
>

Re: question about Nifi Custom processor

[Phillip Grenier]

So a couple notes based on your debug statements.

"2.1 -> 2.2" If you do not plan to provide an input to this processor and
run it on cron or timer, you can just do the session.create at the top.
This will create a new flowfile. If you do expect an incoming flow file
just return in the null check and the next trigger will try the get() again
until it receives a flowfile.

"3 -> 3.1"  You are not providing the actual string, you are passing a
static value of "JSON_PATH". I would do a "final String jsonPath =
context.getProperty(JSON_PATH);" around comment 2.3 and then use jsonPath
in your FileReader.

Object obj = new JSONParser().parse(new FileReader("JSON_PATH"));


Hope this helps,

Phillip

On Tue, Jul 23, 2019 at 10:13 PM Farooq Mustafa 
wrote:

> Hello,
>
>
> I am writing my first custom processor. I followed the example here
> https://www.nifi.rocks/developing-a-custom-apache-nifi-processor-json/
>
>
> In the settings, I have /tmp/jsoniput/sample1.json file
>
>
> The source code;
>
> public void onTrigger(final ProcessContext context, final ProcessSession 
> session) throws ProcessException {
>
>   final AtomicReference value = new AtomicReference<>();
>   getLogger().warn("sample - Farooq - 1 , value = <" + value.toString() + 
> ">");
>
>   FlowFile flowfile = session.get();
>   getLogger().warn("sample - Farooq - 2");
>
>   if ( null == flowfile) {
> getLogger().warn("sample - Farooq - 2.1 - flowfile is NULL *** ");
> flowfile = session.create();
> flowfile = session.get();
> getLogger().warn("sample - Farooq - 2.2 - session.create() / 
> session.get() --- NOT SURE ");
>   }
>   getLogger().warn("sample - Farooq - 2.3 JSON_PATH :" + 
> context.getProperty(JSON_PATH).getValue());
>
>   session.read(flowfile, new InputStreamCallback() {
> @Override
> public void process(InputStream in) throws IOException {
>
>   try {
> getLogger().warn("sample - Farooq - 3 JSON_PATH :" + 
> context.getProperty(JSON_PATH).getValue());
> Object obj = new JSONParser().parse(new FileReader("JSON_PATH"));
> getLogger().warn("sample - Farooq - 3.1 ");
>
>
>
> The log output:
>
> 2019-07-20 13:58:42,625 WARN [Timer-Driven Process Thread-4]
> o.a.nifi.processors.sitc.JsonProcessor
> JsonProcessor[id=fd8fe4ca-016b-1000-a6ed-c0e224bc7267] sample - Farooq - 1
> , value = 
>
> 2019-07-20 13:58:42,625 WARN [Timer-Driven Process Thread-4]
> o.a.nifi.processors.sitc.JsonProcessor
> JsonProcessor[id=fd8fe4ca-016b-1000-a6ed-c0e224bc7267] sample - Farooq - 2
>
> 2019-07-20 13:58:42,625 WARN [Timer-Driven Process Thread-4]
> o.a.nifi.processors.sitc.JsonProcessor
> JsonProcessor[id=fd8fe4ca-016b-1000-a6ed-c0e224bc7267] sample - Farooq -
> 2.1 - flowfile is NULL ***
>
> 2019-07-20 13:58:42,625 WARN [Timer-Driven Process Thread-4]
> o.a.nifi.processors.sitc.JsonProcessor
> JsonProcessor[id=fd8fe4ca-016b-1000-a6ed-c0e224bc7267] sample - Farooq -
> 2.2 - session.create() / session.get() --- NOT SURE
>
> 2019-07-20 13:58:42,625 WARN [Timer-Driven Process Thread-4]
> o.a.nifi.processors.sitc.JsonProcessor
> JsonProcessor[id=fd8fe4ca-016b-1000-a6ed-c0e224bc7267] sample - Farooq -
> 2.3 JSON_PATH :/tmp/jsoninput
>
> 2019-07-20 13:58:42,626 ERROR [Timer-Driven Process Thread-4]
> o.a.nifi.processors.sitc.JsonProcessor
> JsonProcessor[id=fd8fe4ca-016b-1000-a6ed-c0e224bc7267]
> JsonProcessor[id=fd8fe4ca-016b-1000-a6ed-c0e224bc7267] failed to process
> session due to java.lang.NullPointerException; Processor Administratively
> Yielded for 1 sec: java.lang.NullPointerException
>
> java.lang.NullPointerException: null
>
> at
> org.apache.nifi.controller.repository.StandardProcessSession.getRecord(StandardProcessSession.java:574)
>
> at
> org.apache.nifi.controller.repository.StandardProcessSession.validateRecordState(StandardProcessSession.java:3132)
>
> at
> org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2187)
>
>
> And the basic flow:
> [image: image.png]
>
>
>
> Any ideas or guidance will be appreciated.
>
>
> Thanks
>
> Farooq
>

question about Nifi Custom processor

[Farooq Mustafa]

Hello,


I am writing my first custom processor. I followed the example here
https://www.nifi.rocks/developing-a-custom-apache-nifi-processor-json/


In the settings, I have /tmp/jsoniput/sample1.json file


The source code;

public void onTrigger(final ProcessContext context, final
ProcessSession session) throws ProcessException {

  final AtomicReference value = new AtomicReference<>();
  getLogger().warn("sample - Farooq - 1 , value = <" + value.toString() + ">");

  FlowFile flowfile = session.get();
  getLogger().warn("sample - Farooq - 2");

  if ( null == flowfile) {
getLogger().warn("sample - Farooq - 2.1 - flowfile is NULL *** ");
flowfile = session.create();
flowfile = session.get();
getLogger().warn("sample - Farooq - 2.2 - session.create() /
session.get() --- NOT SURE ");
  }
  getLogger().warn("sample - Farooq - 2.3 JSON_PATH :" +
context.getProperty(JSON_PATH).getValue());

  session.read(flowfile, new InputStreamCallback() {
@Override
public void process(InputStream in) throws IOException {

  try {
getLogger().warn("sample - Farooq - 3 JSON_PATH :" +
context.getProperty(JSON_PATH).getValue());
Object obj = new JSONParser().parse(new FileReader("JSON_PATH"));
getLogger().warn("sample - Farooq - 3.1 ");



The log output:

2019-07-20 13:58:42,625 WARN [Timer-Driven Process Thread-4]
o.a.nifi.processors.sitc.JsonProcessor
JsonProcessor[id=fd8fe4ca-016b-1000-a6ed-c0e224bc7267] sample - Farooq - 1
, value = 

2019-07-20 13:58:42,625 WARN [Timer-Driven Process Thread-4]
o.a.nifi.processors.sitc.JsonProcessor
JsonProcessor[id=fd8fe4ca-016b-1000-a6ed-c0e224bc7267] sample - Farooq - 2

2019-07-20 13:58:42,625 WARN [Timer-Driven Process Thread-4]
o.a.nifi.processors.sitc.JsonProcessor
JsonProcessor[id=fd8fe4ca-016b-1000-a6ed-c0e224bc7267] sample - Farooq -
2.1 - flowfile is NULL ***

2019-07-20 13:58:42,625 WARN [Timer-Driven Process Thread-4]
o.a.nifi.processors.sitc.JsonProcessor
JsonProcessor[id=fd8fe4ca-016b-1000-a6ed-c0e224bc7267] sample - Farooq -
2.2 - session.create() / session.get() --- NOT SURE

2019-07-20 13:58:42,625 WARN [Timer-Driven Process Thread-4]
o.a.nifi.processors.sitc.JsonProcessor
JsonProcessor[id=fd8fe4ca-016b-1000-a6ed-c0e224bc7267] sample - Farooq -
2.3 JSON_PATH :/tmp/jsoninput

2019-07-20 13:58:42,626 ERROR [Timer-Driven Process Thread-4]
o.a.nifi.processors.sitc.JsonProcessor
JsonProcessor[id=fd8fe4ca-016b-1000-a6ed-c0e224bc7267]
JsonProcessor[id=fd8fe4ca-016b-1000-a6ed-c0e224bc7267] failed to process
session due to java.lang.NullPointerException; Processor Administratively
Yielded for 1 sec: java.lang.NullPointerException

java.lang.NullPointerException: null

at
org.apache.nifi.controller.repository.StandardProcessSession.getRecord(StandardProcessSession.java:574)

at
org.apache.nifi.controller.repository.StandardProcessSession.validateRecordState(StandardProcessSession.java:3132)

at
org.apache.nifi.controller.repository.StandardProcessSession.read(StandardProcessSession.java:2187)


And the basic flow:
[image: image.png]



Any ideas or guidance will be appreciated.


Thanks

Farooq

Re: Contributing to NIFI (Hazelcast Custom processor)

[Milan Das]

Thanks Mike.
I have created https://issues.apache.org/jira/browse/NIFI-5786
I will create a PR with NIFI-5786 and request for review.

Thanks,
Milan Das

On 11/4/18, 8:00 PM, "Mike Thomsen"  wrote:

Milan,

Yeah, add at least a Jira ticket that describes what you did. Doesn't have
to be multiple, unless you just want to break it down into smaller and
faster reviews. (ex. if the PR is 10k LOC and 4 processors, might be faster
to make it 4 tickets)

Thanks,

Mike

On Sun, Nov 4, 2018 at 7:46 PM Milan Das  wrote:

> Hello PMP,
>
> I am writing a custom processor to use Hazelcast through NIFI.  I went
>  through the complete documentation at
> 
https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide#ContributorGuide-HowtocontributetoApacheNiFi
> .
>
> Before I create a PR do I need to open any JIRA tickets for added features
> ?
>
>
>
> Thanks,
>
> Milan Das
>
>
>
>

Re: Contributing to NIFI (Hazelcast Custom processor)

[Mike Thomsen]

Milan,

Yeah, add at least a Jira ticket that describes what you did. Doesn't have
to be multiple, unless you just want to break it down into smaller and
faster reviews. (ex. if the PR is 10k LOC and 4 processors, might be faster
to make it 4 tickets)

Thanks,

Mike

On Sun, Nov 4, 2018 at 7:46 PM Milan Das  wrote:

> Hello PMP,
>
> I am writing a custom processor to use Hazelcast through NIFI.  I went
>  through the complete documentation at
> https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide#ContributorGuide-HowtocontributetoApacheNiFi
> .
>
> Before I create a PR do I need to open any JIRA tickets for added features
> ?
>
>
>
> Thanks,
>
> Milan Das
>
>
>
>

Contributing to NIFI (Hazelcast Custom processor)

[Milan Das]

Hello PMP,

I am writing a custom processor to use Hazelcast through NIFI.  I went  through 
the complete documentation at 
https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide#ContributorGuide-HowtocontributetoApacheNiFi
 .

Before I create a PR do I need to open any JIRA tickets for added features ?

 

Thanks,

Milan Das

 

Re: Compiling custom processor

[Bryan Bende]

That is accurate. Generally the intention isn't really to extend
processors across NARs. It is better to create a shared module with a
base class that both processors can extend from, or utility code they
can both call, but in the case that shared module doesn't exist then
the main option is to copy over the code you need.

On Wed, Oct 17, 2018 at 2:52 PM John McGinn  wrote:
>
> Thanks Bryan.
>
> That was a mistake on my part. I thought that I could use part of the NiFi 
> standard processors in my custom processor. I suppose this also means it's 
> not possible to extend a standard processor class, say Wait, and override a 
> method, for instance, onTrigger(), to do different work. Instead, as you 
> state, I should copy the portions I would need to my custom processor. Is 
> that accurate?
>
> In the end, as I was looking through how to modify Wait, I think it is bigger 
> than the need that I have, so I restarted with a new generated processor 
> project, and just added the cache service, and appropriate relationships, and 
> simple logic to control flow through my processor. I'll work on getting a 
> github account and uploading my simple processor.
>
> Thanks for the assistance
> John McGinn
>
> 
> On Tue, 10/16/18, Bryan Bende  wrote:
>
>  Subject: Re: Compiling custom processor
>  To: figgie...@sbcglobal.net
>  Date: Tuesday, October 16, 2018, 4:45 PM
>
>  You shouldn't really try to
>  depend on code inside other NARs like
>  standard processors, you'll want to copy
>  WaitNotifyProtocol into your
>  own NAR.
>  On
>  Tue, Oct 16, 2018 at 4:40 PM John McGinn 
>  wrote:
>  >
>  > The
>  commons-lang3-3.7.jar file is located in the lib/bootstrap
>  directory, therefore, I figured it could be considered
>  provided. I've since change it to normal scope, and it
>  is included in my NAR and that error goes away. So, this
>  > then leans on the next error I get, which
>  is no class found for the WaitNotifyProtocol in
>  nifi.processors.standard. Is there an easy way to figure out
>  what NiFi class/object I'm using, and what NAR should be
>  listed as provided? I tried to add
>  nifi-standard-processors-nar, but that doesn't exist for
>  maven, nor does it exist in the lib directory. I looked
>  inside the nifi-standard-nar NAR file, and the
>  nifi-standard-processors is there, and inside that is
>  WaitNotifyProtocol, but yet, my maven won't build if I
>  have nifi-standard-nar listed, either with type of NAR, or
>  scope of provided.
>  >
>  
>  > On Tue, 10/16/18, Bryan Bende 
>  wrote:
>  >
>  >  Subject:
>  Re: Compiling custom processor
>  >  To: dev@nifi.apache.org,
>  figgie...@sbcglobal.net
>  >  Date: Tuesday, October 16, 2018, 4:01
>  PM
>  >
>  >  Any reason
>  why the commons-lang3
>  >  dependency is
>  provided and not normal
>  >  compile
>  scope?
>  >
>  >  Generally
>  the only cases where a dependency
>  >
>  should be provided are
>  >  when a parent
>  NAR
>  >  provides it (the API scenario
>  earlier), or when it
>  >  is a jar
>  directly in lib which means it is on
>  >
>  the classpath of
>  >  everything (like
>  nifi-api
>  >  or slf4j).
>  >  On Tue, Oct 16, 2018 at 3:47 PM John
>  McGinn
>  >  
>  >  wrote:
>  >  >
>  >  > I have a
>  >
>  directory /home/user/nifi-nars, which is listed in the
>  >  nifi.properties file located under the
>  nifi-assembly
>  >  directory from the
>  nifi-master code. That's where I
>  >
>  redropped my NAR file at.
>  >  >
>  >  > The error was "due to
>  uncaught
>  >  Exception:
>  java.lang.NoClassDefFoundError:
>  >
>  org/apache/commons/lang3/StringUtils". I added a
>  >  provided dependency of version 3.4 for
>  commons-lang3 in my
>  >  processors
>  pom.xml. That worked initially, but with the
>  >  redropping of the NAR file, the error
>  occurs. The stacktrace
>  >  points to the
>  session.get() lambda function trying to use
>  >  StringUtils.isBlank(). (Again, this is a
>  copy and renamed
>  >  class of the Wait
>  Processor. Trying to turn it into a Yield,
>  >  as it were, hold unless it's free to
>  move forward.)
>  >  >
>  >
>  
>  >  > On Tue, 10/16/18, Bryan Bende
>  
>  >  wrote:
>  >  >
>  >  >  Subject:
>  >
>  Re: Compiling custom processor
>  >  >
>  To: dev@nifi.apache.org
>  > 

Re: Compiling custom processor

[John McGinn]

Thanks Bryan.

That was a mistake on my part. I thought that I could use part of the NiFi 
standard processors in my custom processor. I suppose this also means it's not 
possible to extend a standard processor class, say Wait, and override a method, 
for instance, onTrigger(), to do different work. Instead, as you state, I 
should copy the portions I would need to my custom processor. Is that accurate?

In the end, as I was looking through how to modify Wait, I think it is bigger 
than the need that I have, so I restarted with a new generated processor 
project, and just added the cache service, and appropriate relationships, and 
simple logic to control flow through my processor. I'll work on getting a 
github account and uploading my simple processor.

Thanks for the assistance
John McGinn


On Tue, 10/16/18, Bryan Bende  wrote:

 Subject: Re: Compiling custom processor
 To: figgie...@sbcglobal.net
 Date: Tuesday, October 16, 2018, 4:45 PM
 
 You shouldn't really try to
 depend on code inside other NARs like
 standard processors, you'll want to copy
 WaitNotifyProtocol into your
 own NAR.
 On
 Tue, Oct 16, 2018 at 4:40 PM John McGinn 
 wrote:
 >
 > The
 commons-lang3-3.7.jar file is located in the lib/bootstrap
 directory, therefore, I figured it could be considered
 provided. I've since change it to normal scope, and it
 is included in my NAR and that error goes away. So, this
 > then leans on the next error I get, which
 is no class found for the WaitNotifyProtocol in
 nifi.processors.standard. Is there an easy way to figure out
 what NiFi class/object I'm using, and what NAR should be
 listed as provided? I tried to add
 nifi-standard-processors-nar, but that doesn't exist for
 maven, nor does it exist in the lib directory. I looked
 inside the nifi-standard-nar NAR file, and the
 nifi-standard-processors is there, and inside that is
 WaitNotifyProtocol, but yet, my maven won't build if I
 have nifi-standard-nar listed, either with type of NAR, or
 scope of provided.
 >
 
 > On Tue, 10/16/18, Bryan Bende 
 wrote:
 >
 >  Subject:
 Re: Compiling custom processor
 >  To: dev@nifi.apache.org,
 figgie...@sbcglobal.net
 >  Date: Tuesday, October 16, 2018, 4:01
 PM
 >
 >  Any reason
 why the commons-lang3
 >  dependency is
 provided and not normal
 >  compile
 scope?
 >
 >  Generally
 the only cases where a dependency
 > 
 should be provided are
 >  when a parent
 NAR
 >  provides it (the API scenario
 earlier), or when it
 >  is a jar
 directly in lib which means it is on
 > 
 the classpath of
 >  everything (like
 nifi-api
 >  or slf4j).
 >  On Tue, Oct 16, 2018 at 3:47 PM John
 McGinn
 >  
 >  wrote:
 >  >
 >  > I have a
 > 
 directory /home/user/nifi-nars, which is listed in the
 >  nifi.properties file located under the
 nifi-assembly
 >  directory from the
 nifi-master code. That's where I
 > 
 redropped my NAR file at.
 >  >
 >  > The error was "due to
 uncaught
 >  Exception:
 java.lang.NoClassDefFoundError:
 > 
 org/apache/commons/lang3/StringUtils". I added a
 >  provided dependency of version 3.4 for
 commons-lang3 in my
 >  processors
 pom.xml. That worked initially, but with the
 >  redropping of the NAR file, the error
 occurs. The stacktrace
 >  points to the
 session.get() lambda function trying to use
 >  StringUtils.isBlank(). (Again, this is a
 copy and renamed
 >  class of the Wait
 Processor. Trying to turn it into a Yield,
 >  as it were, hold unless it's free to
 move forward.)
 >  >
 > 
 
 >  > On Tue, 10/16/18, Bryan Bende
 
 >  wrote:
 >  >
 >  >  Subject:
 > 
 Re: Compiling custom processor
 >  > 
 To: dev@nifi.apache.org
 >  >  Date: Tuesday, October 16, 2018,
 3:34
 >  PM
 > 
 >
 >  >  There
 >  shouldn't be any issue
 >  >  doing
 > 
 that, generally you just copy the
 > 
 >
 >  new
 > 
 >  NAR into the lib directory and
 > 
 restart.
 >  >
 > 
 >  What
 >  were the specifics of the
 class not found
 >  >  error?
 >  >
 >  >  On
 Tue, Oct 16, 2018 at
 >  >  3:24 PM
 John McGinn 
 >  >  wrote:
 > 
 >  >
 >  >  > And
 I'm
 >  >
 > 
 back.
 >  >  >
 >  >
 >  > So, I
 had my
 >  >  processor running,
 >  no errors, so things were good.
 >  >
 >  >
 >  >  > I then add 2 if
 >  >  statements with logger messages,
 run the
 >  same mvn command,
 >  >  copied the new NAR
 >  on top of the old one, same versionId
 and
 >  >  all, and now I get a class
 not found
 >  error, that didn't
 >  >  occur before I
 >  stopp

Re: Compiling custom processor

[Bryan Bende]

Any reason why the commons-lang3 dependency is provided and not normal
compile scope?

Generally the only cases where a dependency should be provided are
when a parent NAR provides it (the API scenario earlier), or when it
is a jar directly in lib which means it is on the classpath of
everything (like nifi-api or slf4j).
On Tue, Oct 16, 2018 at 3:47 PM John McGinn  wrote:
>
> I have a directory /home/user/nifi-nars, which is listed in the 
> nifi.properties file located under the nifi-assembly directory from the 
> nifi-master code. That's where I redropped my NAR file at.
>
> The error was "due to uncaught Exception: java.lang.NoClassDefFoundError: 
> org/apache/commons/lang3/StringUtils". I added a provided dependency of 
> version 3.4 for commons-lang3 in my processors pom.xml. That worked 
> initially, but with the redropping of the NAR file, the error occurs. The 
> stacktrace points to the session.get() lambda function trying to use 
> StringUtils.isBlank(). (Again, this is a copy and renamed class of the Wait 
> Processor. Trying to turn it into a Yield, as it were, hold unless it's free 
> to move forward.)
> 
> On Tue, 10/16/18, Bryan Bende  wrote:
>
>  Subject: Re: Compiling custom processor
>  To: dev@nifi.apache.org
>  Date: Tuesday, October 16, 2018, 3:34 PM
>
>  There shouldn't be any issue
>  doing that, generally you just copy the
>  new
>  NAR into the lib directory and restart.
>
>  What were the specifics of the class not found
>  error?
>
>  On Tue, Oct 16, 2018 at
>  3:24 PM John McGinn 
>  wrote:
>  >
>  > And I'm
>  back.
>  >
>  > So, I had my
>  processor running, no errors, so things were good.
>  >
>  > I then add 2 if
>  statements with logger messages, run the same mvn command,
>  copied the new NAR on top of the old one, same versionId and
>  all, and now I get a class not found error, that didn't
>  occur before I stopped, redeployed and started. And I
>  can't delete the event in the queue even though source
>  and destination are stopped.
>  >
>  > Is there problems when redeploying a
>  custom processor NAR on top of itself? Even if the NiFi
>  instance is stopped? Serialization issues?
>  >
>  > Frustratedly
>  yours,
>  > John McGinn
>  >
>  >
>  
>  > On Tue, 10/16/18, John McGinn 
>  wrote:
>  >
>  >  Subject:
>  Re: Compiling custom processor
>  >  To: dev@nifi.apache.org
>  >  Date: Tuesday, October 16, 2018, 1:21
>  PM
>  >
>  >  Thanks
>  Bryan,
>  >
>  >  I
>  restarted with a fresh maven
>  >
>  generate, and only added the API to the processor pom,
>  and
>  >  then the API-NAR to the NAR pom.
>  I then figured out a
>  >  subsequent
>  dependency (lang3), and got that in, and now my
>  >  processor shows up in local NiFi
>  instance.
>  >
>  >
>  Regarding the copying of the standard
>  >
>  processor, when I copied Wait over, I renamed it from
>  Wait
>  >  to MyProcessorWait to be unique,
>  and not copied. I'm just
>  >
>  attempting changes to Wait without modifying the actual
>  one.
>  >  Also, thanks on the
>  META-INF/services file information.
>  >
>  >  Now I can debug and test.
>  >
>  >  Sincerely,
>  >  John McGinn
>  >
>  >
>  
>  >  On Tue, 10/16/18, Bryan Bende 
>  >  wrote:
>  >
>  >   Subject: Re: Compiling custom
>  >  processor
>  >   To: dev@nifi.apache.org
>  >   Date: Tuesday, October 16, 2018,
>  9:52
>  >  AM
>  >
>  >   In general, if your processor
>  >   uses a controller service then the
>  >   processors pom file needs a provided
>  >  dependency
>  >   on
>  the API of the CS,
>  >   and your NAR pom
>  needs
>  >   a NAR dependency on the NAR
>  where the
>  >  CS API
>  >   is.
>  >
>  >   Example is
>  >
>  shown here in the section linking
>  >
>  processors and
>  >   controller
>  >   services - 
> https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions
>  >
>  >   Also, you do not
>  want to
>  >   include NiFI's standard
>  processors in
>  >  your
>  >   own NAR, this will result in two
>  >  copies of
>  >   every
>  standard processor.
>  >   You will want
>  to
>  >   copy whatever code you need into
>  your
>  >  own NAR.
>  >
>  >   In you last statement, if you
>  >   made a new p

Re: Compiling custom processor

[John McGinn]

I have a directory /home/user/nifi-nars, which is listed in the nifi.properties 
file located under the nifi-assembly directory from the nifi-master code. 
That's where I redropped my NAR file at.

The error was "due to uncaught Exception: java.lang.NoClassDefFoundError: 
org/apache/commons/lang3/StringUtils". I added a provided dependency of version 
3.4 for commons-lang3 in my processors pom.xml. That worked initially, but with 
the redropping of the NAR file, the error occurs. The stacktrace points to the 
session.get() lambda function trying to use StringUtils.isBlank(). (Again, this 
is a copy and renamed class of the Wait Processor. Trying to turn it into a 
Yield, as it were, hold unless it's free to move forward.)

On Tue, 10/16/18, Bryan Bende  wrote:

 Subject: Re: Compiling custom processor
 To: dev@nifi.apache.org
 Date: Tuesday, October 16, 2018, 3:34 PM
 
 There shouldn't be any issue
 doing that, generally you just copy the
 new
 NAR into the lib directory and restart.
 
 What were the specifics of the class not found
 error?
 
 On Tue, Oct 16, 2018 at
 3:24 PM John McGinn 
 wrote:
 >
 > And I'm
 back.
 >
 > So, I had my
 processor running, no errors, so things were good.
 >
 > I then add 2 if
 statements with logger messages, run the same mvn command,
 copied the new NAR on top of the old one, same versionId and
 all, and now I get a class not found error, that didn't
 occur before I stopped, redeployed and started. And I
 can't delete the event in the queue even though source
 and destination are stopped.
 >
 > Is there problems when redeploying a
 custom processor NAR on top of itself? Even if the NiFi
 instance is stopped? Serialization issues?
 >
 > Frustratedly
 yours,
 > John McGinn
 >
 >
 
 > On Tue, 10/16/18, John McGinn 
 wrote:
 >
 >  Subject:
 Re: Compiling custom processor
 >  To: dev@nifi.apache.org
 >  Date: Tuesday, October 16, 2018, 1:21
 PM
 >
 >  Thanks
 Bryan,
 >
 >  I
 restarted with a fresh maven
 > 
 generate, and only added the API to the processor pom,
 and
 >  then the API-NAR to the NAR pom.
 I then figured out a
 >  subsequent
 dependency (lang3), and got that in, and now my
 >  processor shows up in local NiFi
 instance.
 >
 > 
 Regarding the copying of the standard
 > 
 processor, when I copied Wait over, I renamed it from
 Wait
 >  to MyProcessorWait to be unique,
 and not copied. I'm just
 > 
 attempting changes to Wait without modifying the actual
 one.
 >  Also, thanks on the
 META-INF/services file information.
 >
 >  Now I can debug and test.
 >
 >  Sincerely,
 >  John McGinn
 >
 > 
 ----
 >  On Tue, 10/16/18, Bryan Bende 
 >  wrote:
 >
 >   Subject: Re: Compiling custom
 >  processor
 >   To: dev@nifi.apache.org
 >   Date: Tuesday, October 16, 2018,
 9:52
 >  AM
 >
 >   In general, if your processor
 >   uses a controller service then the
 >   processors pom file needs a provided
 >  dependency
 >   on
 the API of the CS,
 >   and your NAR pom
 needs
 >   a NAR dependency on the NAR
 where the
 >  CS API
 >   is.
 >
 >   Example is
 >  
 shown here in the section linking
 > 
 processors and
 >   controller
 >   services - 
 >https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions
 >
 >   Also, you do not
 want to
 >   include NiFI's standard
 processors in
 >  your
 >   own NAR, this will result in two
 >  copies of
 >   every
 standard processor.
 >   You will want
 to
 >   copy whatever code you need into
 your
 >  own NAR.
 >
 >   In you last statement, if you
 >   made a new processor in
 >   nifi-standard-processors and it
 didn't
 >  show
 >   up, it is most likely
 >   because you didn't
 >   update META-INF/services file to
 >  include the new
 >  
 processor.
 >   On Tue, Oct 16, 2018
 at
 >   9:36 AM John McGinn
 >   
 >   wrote:
 >   >
 >   > Ok, I am
 >  
 far from a maven expert, and am
 > 
 struggling on this
 >   problem.
 >   >
 >   > I
 created
 >   a new project using the
 maven generate
 >  process, and
 compiled
 >   that sample processor and
 everything
 >  was fine, and I could
 >   see it in my local NiFi instance. I
 >  then copied over the
 >   Wait processor on top of the
 >  MyProcessor.java class, changed
 >   the package name, and the class
 name,
 >  and attempted to
 >   compile. Got errors due to
 dependency
 >  issues. This is where
 >   I get confused.
 >  
 >
 >   > In
 >  
 the processor directory, I modify the
 > 
 pom.xml with a
 >   dependency line for
 nifi-api,
 >  nifi-utils,
 >
 > 
 nifi-distri

Re: Compiling custom processor

[Bryan Bende]

There shouldn't be any issue doing that, generally you just copy the
new NAR into the lib directory and restart.

What were the specifics of the class not found error?

On Tue, Oct 16, 2018 at 3:24 PM John McGinn  wrote:
>
> And I'm back.
>
> So, I had my processor running, no errors, so things were good.
>
> I then add 2 if statements with logger messages, run the same mvn command, 
> copied the new NAR on top of the old one, same versionId and all, and now I 
> get a class not found error, that didn't occur before I stopped, redeployed 
> and started. And I can't delete the event in the queue even though source and 
> destination are stopped.
>
> Is there problems when redeploying a custom processor NAR on top of itself? 
> Even if the NiFi instance is stopped? Serialization issues?
>
> Frustratedly yours,
> John McGinn
>
> 
> On Tue, 10/16/18, John McGinn  wrote:
>
>  Subject: Re: Compiling custom processor
>  To: dev@nifi.apache.org
>  Date: Tuesday, October 16, 2018, 1:21 PM
>
>  Thanks Bryan,
>
>  I restarted with a fresh maven
>  generate, and only added the API to the processor pom, and
>  then the API-NAR to the NAR pom. I then figured out a
>  subsequent dependency (lang3), and got that in, and now my
>  processor shows up in local NiFi instance.
>
>  Regarding the copying of the standard
>  processor, when I copied Wait over, I renamed it from Wait
>  to MyProcessorWait to be unique, and not copied. I'm just
>  attempting changes to Wait without modifying the actual one.
>  Also, thanks on the META-INF/services file information.
>
>  Now I can debug and test.
>
>  Sincerely,
>  John McGinn
>
>  ----
>  On Tue, 10/16/18, Bryan Bende 
>  wrote:
>
>   Subject: Re: Compiling custom
>  processor
>   To: dev@nifi.apache.org
>   Date: Tuesday, October 16, 2018, 9:52
>  AM
>
>   In general, if your processor
>   uses a controller service then the
>   processors pom file needs a provided
>  dependency
>   on the API of the CS,
>   and your NAR pom needs
>   a NAR dependency on the NAR where the
>  CS API
>   is.
>
>   Example is
>   shown here in the section linking
>  processors and
>   controller
>   services - 
> https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions
>
>   Also, you do not want to
>   include NiFI's standard processors in
>  your
>   own NAR, this will result in two
>  copies of
>   every standard processor.
>   You will want to
>   copy whatever code you need into your
>  own NAR.
>
>   In you last statement, if you
>   made a new processor in
>   nifi-standard-processors and it didn't
>  show
>   up, it is most likely
>   because you didn't
>   update META-INF/services file to
>  include the new
>   processor.
>   On Tue, Oct 16, 2018 at
>   9:36 AM John McGinn
>   
>   wrote:
>   >
>   > Ok, I am
>   far from a maven expert, and am
>  struggling on this
>   problem.
>   >
>   > I created
>   a new project using the maven generate
>  process, and compiled
>   that sample processor and everything
>  was fine, and I could
>   see it in my local NiFi instance. I
>  then copied over the
>   Wait processor on top of the
>  MyProcessor.java class, changed
>   the package name, and the class name,
>  and attempted to
>   compile. Got errors due to dependency
>  issues. This is where
>   I get confused.
>   >
>   > In
>   the processor directory, I modify the
>  pom.xml with a
>   dependency line for nifi-api,
>  nifi-utils,
>
>  nifi-distribute-cache-client-service-api and
>   nifi-standard-processors. This seemed
>  to make things work,
>   and I get a 33 meg NAR file, which
>  contains lots of JARs. I
>   load that up, and I get a problem with
>  CalculateStats, or
>   similar, because of
>  RecordReaderFactory. I go back in and
>   include
>  nifi-record-serialization-services, and it compiles,
>   and the JAR is included in the NAR
>  file now, and I still get
>   class not found.
>   >
>   >
>   I've also tried to change the
>  dependencies to a status
>   of provided, and therefore get a
>  minimal 25k or so NAR file
>   with no JARs included. I get the same
>  issue with the class
>   not found.
>   >
>   > I had
>   also attempted to copy Wait.java to
>  AnotherWait.java within
>   the actually NiFi source code, and it
>  compiles, and shows up
>   in the JARs and NARs as I'd expect,
>  but I cannot get to
>   it within my NiFi instance.
>   >
>   > Can someone let me know what I'm
>  doing
>   wrong.
>   >
>   > Thanks,
>   > John McGinn
>

Re: Compiling custom processor

[John McGinn]

And I'm back.

So, I had my processor running, no errors, so things were good.

I then add 2 if statements with logger messages, run the same mvn command, 
copied the new NAR on top of the old one, same versionId and all, and now I get 
a class not found error, that didn't occur before I stopped, redeployed and 
started. And I can't delete the event in the queue even though source and 
destination are stopped.

Is there problems when redeploying a custom processor NAR on top of itself? 
Even if the NiFi instance is stopped? Serialization issues?

Frustratedly yours,
John McGinn


On Tue, 10/16/18, John McGinn  wrote:

 Subject: Re: Compiling custom processor
 To: dev@nifi.apache.org
 Date: Tuesday, October 16, 2018, 1:21 PM
 
 Thanks Bryan,
 
 I restarted with a fresh maven
 generate, and only added the API to the processor pom, and
 then the API-NAR to the NAR pom. I then figured out a
 subsequent dependency (lang3), and got that in, and now my
 processor shows up in local NiFi instance.
 
 Regarding the copying of the standard
 processor, when I copied Wait over, I renamed it from Wait
 to MyProcessorWait to be unique, and not copied. I'm just
 attempting changes to Wait without modifying the actual one.
 Also, thanks on the META-INF/services file information.
 
 Now I can debug and test.
 
 Sincerely,
 John McGinn
 
 
 On Tue, 10/16/18, Bryan Bende 
 wrote:
 
  Subject: Re: Compiling custom
 processor
  To: dev@nifi.apache.org
  Date: Tuesday, October 16, 2018, 9:52
 AM
  
  In general, if your processor
  uses a controller service then the
  processors pom file needs a provided
 dependency
  on the API of the CS,
  and your NAR pom needs
  a NAR dependency on the NAR where the
 CS API
  is.
  
  Example is
  shown here in the section linking
 processors and
  controller
  services - 
https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions
  
  Also, you do not want to
  include NiFI's standard processors in
 your
  own NAR, this will result in two
 copies of
  every standard processor.
  You will want to
  copy whatever code you need into your
 own NAR.
  
  In you last statement, if you
  made a new processor in
  nifi-standard-processors and it didn't
 show
  up, it is most likely
  because you didn't
  update META-INF/services file to
 include the new
  processor.
  On Tue, Oct 16, 2018 at
  9:36 AM John McGinn
  
  wrote:
  >
  > Ok, I am
  far from a maven expert, and am
 struggling on this
  problem.
  >
  > I created
  a new project using the maven generate
 process, and compiled
  that sample processor and everything
 was fine, and I could
  see it in my local NiFi instance. I
 then copied over the
  Wait processor on top of the
 MyProcessor.java class, changed
  the package name, and the class name,
 and attempted to
  compile. Got errors due to dependency
 issues. This is where
  I get confused.
  >
  > In
  the processor directory, I modify the
 pom.xml with a
  dependency line for nifi-api,
 nifi-utils,
 
 nifi-distribute-cache-client-service-api and
  nifi-standard-processors. This seemed
 to make things work,
  and I get a 33 meg NAR file, which
 contains lots of JARs. I
  load that up, and I get a problem with
 CalculateStats, or
  similar, because of
 RecordReaderFactory. I go back in and
  include
 nifi-record-serialization-services, and it compiles,
  and the JAR is included in the NAR
 file now, and I still get
  class not found.
  >
  >
  I've also tried to change the
 dependencies to a status
  of provided, and therefore get a
 minimal 25k or so NAR file
  with no JARs included. I get the same
 issue with the class
  not found.
  >
  > I had
  also attempted to copy Wait.java to
 AnotherWait.java within
  the actually NiFi source code, and it
 compiles, and shows up
  in the JARs and NARs as I'd expect,
 but I cannot get to
  it within my NiFi instance.
  >
  > Can someone let me know what I'm
 doing
  wrong.
  >
  > Thanks,
  > John McGinn
 

Re: Compiling custom processor

[John McGinn]

Thanks Bryan,

I restarted with a fresh maven generate, and only added the API to the 
processor pom, and then the API-NAR to the NAR pom. I then figured out a 
subsequent dependency (lang3), and got that in, and now my processor shows up 
in local NiFi instance.

Regarding the copying of the standard processor, when I copied Wait over, I 
renamed it from Wait to MyProcessorWait to be unique, and not copied. I'm just 
attempting changes to Wait without modifying the actual one. Also, thanks on 
the META-INF/services file information.

Now I can debug and test.

Sincerely,
John McGinn


On Tue, 10/16/18, Bryan Bende  wrote:

 Subject: Re: Compiling custom processor
 To: dev@nifi.apache.org
 Date: Tuesday, October 16, 2018, 9:52 AM
 
 In general, if your processor
 uses a controller service then the
 processors pom file needs a provided dependency
 on the API of the CS,
 and your NAR pom needs
 a NAR dependency on the NAR where the CS API
 is.
 
 Example is
 shown here in the section linking processors and
 controller
 services - 
https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions
 
 Also, you do not want to
 include NiFI's standard processors in your
 own NAR, this will result in two copies of
 every standard processor.
 You will want to
 copy whatever code you need into your own NAR.
 
 In you last statement, if you
 made a new processor in
 nifi-standard-processors and it didn't show
 up, it is most likely
 because you didn't
 update META-INF/services file to include the new
 processor.
 On Tue, Oct 16, 2018 at
 9:36 AM John McGinn
 
 wrote:
 >
 > Ok, I am
 far from a maven expert, and am struggling on this
 problem.
 >
 > I created
 a new project using the maven generate process, and compiled
 that sample processor and everything was fine, and I could
 see it in my local NiFi instance. I then copied over the
 Wait processor on top of the MyProcessor.java class, changed
 the package name, and the class name, and attempted to
 compile. Got errors due to dependency issues. This is where
 I get confused.
 >
 > In
 the processor directory, I modify the pom.xml with a
 dependency line for nifi-api, nifi-utils,
 nifi-distribute-cache-client-service-api and
 nifi-standard-processors. This seemed to make things work,
 and I get a 33 meg NAR file, which contains lots of JARs. I
 load that up, and I get a problem with CalculateStats, or
 similar, because of RecordReaderFactory. I go back in and
 include nifi-record-serialization-services, and it compiles,
 and the JAR is included in the NAR file now, and I still get
 class not found.
 >
 >
 I've also tried to change the dependencies to a status
 of provided, and therefore get a minimal 25k or so NAR file
 with no JARs included. I get the same issue with the class
 not found.
 >
 > I had
 also attempted to copy Wait.java to AnotherWait.java within
 the actually NiFi source code, and it compiles, and shows up
 in the JARs and NARs as I'd expect, but I cannot get to
 it within my NiFi instance.
 >
 > Can someone let me know what I'm doing
 wrong.
 >
 > Thanks,
 > John McGinn

Re: Compiling custom processor

[Mike Thomsen]

If you want to look at a large Maven project that builds several custom
NARs from one unified build system, take a look at my data generation
bundle here:

https://github.com/MikeThomsen/nifi-datageneration-bundle

On Tue, Oct 16, 2018 at 9:53 AM Bryan Bende  wrote:

> In general, if your processor uses a controller service then the
> processors pom file needs a provided dependency on the API of the CS,
> and your NAR pom needs a NAR dependency on the NAR where the CS API
> is.
>
> Example is shown here in the section linking processors and controller
> services -
> https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions
>
> Also, you do not want to include NiFI's standard processors in your
> own NAR, this will result in two copies of every standard processor.
> You will want to copy whatever code you need into your own NAR.
>
> In you last statement, if you made a new processor in
> nifi-standard-processors and it didn't show up, it is most likely
> because you didn't update META-INF/services file to include the new
> processor.
> On Tue, Oct 16, 2018 at 9:36 AM John McGinn
>  wrote:
> >
> > Ok, I am far from a maven expert, and am struggling on this problem.
> >
> > I created a new project using the maven generate process, and compiled
> that sample processor and everything was fine, and I could see it in my
> local NiFi instance. I then copied over the Wait processor on top of the
> MyProcessor.java class, changed the package name, and the class name, and
> attempted to compile. Got errors due to dependency issues. This is where I
> get confused.
> >
> > In the processor directory, I modify the pom.xml with a dependency line
> for nifi-api, nifi-utils, nifi-distribute-cache-client-service-api and
> nifi-standard-processors. This seemed to make things work, and I get a 33
> meg NAR file, which contains lots of JARs. I load that up, and I get a
> problem with CalculateStats, or similar, because of RecordReaderFactory. I
> go back in and include nifi-record-serialization-services, and it compiles,
> and the JAR is included in the NAR file now, and I still get class not
> found.
> >
> > I've also tried to change the dependencies to a status of provided, and
> therefore get a minimal 25k or so NAR file with no JARs included. I get the
> same issue with the class not found.
> >
> > I had also attempted to copy Wait.java to AnotherWait.java within the
> actually NiFi source code, and it compiles, and shows up in the JARs and
> NARs as I'd expect, but I cannot get to it within my NiFi instance.
> >
> > Can someone let me know what I'm doing wrong.
> >
> > Thanks,
> > John McGinn
>

Re: Compiling custom processor

[Bryan Bende]

In general, if your processor uses a controller service then the
processors pom file needs a provided dependency on the API of the CS,
and your NAR pom needs a NAR dependency on the NAR where the CS API
is.

Example is shown here in the section linking processors and controller
services - 
https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions

Also, you do not want to include NiFI's standard processors in your
own NAR, this will result in two copies of every standard processor.
You will want to copy whatever code you need into your own NAR.

In you last statement, if you made a new processor in
nifi-standard-processors and it didn't show up, it is most likely
because you didn't update META-INF/services file to include the new
processor.
On Tue, Oct 16, 2018 at 9:36 AM John McGinn
 wrote:
>
> Ok, I am far from a maven expert, and am struggling on this problem.
>
> I created a new project using the maven generate process, and compiled that 
> sample processor and everything was fine, and I could see it in my local NiFi 
> instance. I then copied over the Wait processor on top of the 
> MyProcessor.java class, changed the package name, and the class name, and 
> attempted to compile. Got errors due to dependency issues. This is where I 
> get confused.
>
> In the processor directory, I modify the pom.xml with a dependency line for 
> nifi-api, nifi-utils, nifi-distribute-cache-client-service-api and 
> nifi-standard-processors. This seemed to make things work, and I get a 33 meg 
> NAR file, which contains lots of JARs. I load that up, and I get a problem 
> with CalculateStats, or similar, because of RecordReaderFactory. I go back in 
> and include nifi-record-serialization-services, and it compiles, and the JAR 
> is included in the NAR file now, and I still get class not found.
>
> I've also tried to change the dependencies to a status of provided, and 
> therefore get a minimal 25k or so NAR file with no JARs included. I get the 
> same issue with the class not found.
>
> I had also attempted to copy Wait.java to AnotherWait.java within the 
> actually NiFi source code, and it compiles, and shows up in the JARs and NARs 
> as I'd expect, but I cannot get to it within my NiFi instance.
>
> Can someone let me know what I'm doing wrong.
>
> Thanks,
> John McGinn

Compiling custom processor

[John McGinn]

Ok, I am far from a maven expert, and am struggling on this problem.

I created a new project using the maven generate process, and compiled that 
sample processor and everything was fine, and I could see it in my local NiFi 
instance. I then copied over the Wait processor on top of the MyProcessor.java 
class, changed the package name, and the class name, and attempted to compile. 
Got errors due to dependency issues. This is where I get confused.

In the processor directory, I modify the pom.xml with a dependency line for 
nifi-api, nifi-utils, nifi-distribute-cache-client-service-api and 
nifi-standard-processors. This seemed to make things work, and I get a 33 meg 
NAR file, which contains lots of JARs. I load that up, and I get a problem with 
CalculateStats, or similar, because of RecordReaderFactory. I go back in and 
include nifi-record-serialization-services, and it compiles, and the JAR is 
included in the NAR file now, and I still get class not found.

I've also tried to change the dependencies to a status of provided, and 
therefore get a minimal 25k or so NAR file with no JARs included. I get the 
same issue with the class not found.

I had also attempted to copy Wait.java to AnotherWait.java within the actually 
NiFi source code, and it compiles, and shows up in the JARs and NARs as I'd 
expect, but I cannot get to it within my NiFi instance.

Can someone let me know what I'm doing wrong.

Thanks,
John McGinn

Re: get access token inside custom processor

[Andy LoPresto]

Suraj,

There is no concept of a “logged in user” in NiFi. Many (or none) users may be 
accessing the NiFi UI at the same time, but this is completely independent of 
the flow operation. The component state is running/stopped/etc. regardless of 
the user viewing the UI.


Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Aug 29, 2018, at 10:26 AM, Chamoli, Suraj Ramesh 
>  wrote:
> 
> Hi,
> Is there any way where we can get the USERID  of same as the $hostname{()} 
> and ${ip()} of a nifi server.
> 
> On 2018/02/27 14:31:26, Bryan Bende http://gmail.com/>> 
> wrote:
>> Hello,>
>> 
>> Your custom processor would be the same as if you were writing an>
>> external client program.>
>> 
>> You would need to provide the processor with a username and password>
>> in the processor properties, and then it would need to make a call to>
>> the token REST end-point.>
>> 
>> Processors don't run as the user from the web UI, they run on behalf>
>> of the NiFi framework and have no idea which user started/stopped>
>> them.>
>> 
>> Thanks,>
>> 
>> Bryan>
>> 
>> On Tue, Feb 27, 2018 at 1:27 AM, 尹文才 http://gmail.com/>> 
>> wrote:>
>>> Hi guys, I'm trying to invoke some nifi rest apis inside my custom>
>>> processor, the nifi I'm using is nifi 1.4.0 and it's a 3 node secured>
>>> cluster., the username and password are kept inside a ldap server.>
>>> I know that in a secured nifi cluster, in order to make any request I need>
>>> the access token, my question is how could I get the access token in my>
>>> custom processor? Thanks. (I think the token should be>
>>> available somewhere after successful login right?)>
>>>> 
>>> regards,>
>>> ben>



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: get access token inside custom processor

[Chamoli, Suraj Ramesh]

Hi,
Is there any way where we can get the USERID  of same as the $hostname{()} and 
${ip()} of a nifi server.

On 2018/02/27 14:31:26, Bryan Bende  wrote:
> Hello,>
>
> Your custom processor would be the same as if you were writing an>
> external client program.>
>
> You would need to provide the processor with a username and password>
> in the processor properties, and then it would need to make a call to>
> the token REST end-point.>
>
> Processors don't run as the user from the web UI, they run on behalf>
> of the NiFi framework and have no idea which user started/stopped>
> them.>
>
> Thanks,>
>
> Bryan>
>
> On Tue, Feb 27, 2018 at 1:27 AM, 尹文才  wrote:>
> > Hi guys, I'm trying to invoke some nifi rest apis inside my custom>
> > processor, the nifi I'm using is nifi 1.4.0 and it's a 3 node secured>
> > cluster., the username and password are kept inside a ldap server.>
> > I know that in a secured nifi cluster, in order to make any request I need>
> > the access token, my question is how could I get the access token in my>
> > custom processor? Thanks. (I think the token should be>
> > available somewhere after successful login right?)>
> >>
> > regards,>
> > ben>
>

Re: Custom Processor Bundle Contribution

[Prashanth Venkatesan]

Thanks Pierre for the reply. I will check and follow the procedure.

Regards,
Prashanth V

On Tue, Jun 12, 2018 at 1:01 PM, Pierre Villard  wrote:

> Hi Prashanth,
>
> First step is to create a JIRA describing the new feature you want to
> contribute [1].
> Then you can have a look at the contributor guide [2] to help you submit a
> pull request on Github [2].
>
> Let us know if something is not clear.
>
> [1] https://issues.apache.org/jira/projects/NIFI
> [2] https://cwiki.apache.org/confluence/display/NIFI/Contributor+Guide
>
> Pierre
>
> 2018-06-12 9:23 GMT+02:00 Prashanth Venkatesan  >:
>
> > Hi Team,
> >
> > I created network related custom processor and would like to contribute
> it
> > to NiFi source. I want to know the procedure to contribute to NiFi.
> Please
> > provide me the detailed guidelines..
> >
> > --
> > *With regards,*
> > Prashanth V
> >
>



-- 
*With regards,*
Prashanth V

Fwd: Custom Processor Bundle Contribution

[Prashanth Venkatesan]

Hi Team,

I created network related custom processor and would like to contribute it
to NiFi source. I want to know the procedure to contribute to NiFi. Please
provide me the detailed guidelines..

-- 
*With regards,*
Prashanth V

Re: How to capture failure bulletins in custom processor

[Pierre Villard]

Hi,

Here is an article I wrote few months ago if it can help you getting
started:
https://pierrevillard.com/2017/05/13/monitoring-nifi-site2site-reporting-tasks/

Pierre

2018-05-25 14:52 GMT+02:00 Matt Burgess <mattyb...@apache.org>:

> You can use a SiteToSiteBulletinReportingTask [1], it will send flow
> files containing bulletins to an Input Port on the root process group
> (aka top-level canvas), then you can send to whichever processors you
> like for further operations.
>
> Regards,
> Matt
>
> [1] https://nifi.apache.org/docs/nifi-docs/components/org.
> apache.nifi/nifi-site-to-site-reporting-nar/1.6.0/org.
> apache.nifi.reporting.SiteToSiteBulletinReportingTask/index.html
>
>
> On Fri, May 25, 2018 at 8:46 AM, Brajendra Mishra
> <brajendra_mis...@persistent.com> wrote:
> > Hi Team,
> >
> > Could you please let me know how can I get bulletins when failure has
> happened in executing flow file?
> > I need to capture these bulletins in  custom processor (java code) and
> want to send to next processor where I will create new error message in
> desired format.
> >
> > Brajendra Mishra
> >
> > DISCLAIMER
> > ==
> > This e-mail may contain privileged and confidential information which is
> the property of Persistent Systems Ltd. It is intended only for the use of
> the individual or entity to which it is addressed. If you are not the
> intended recipient, you are not authorized to read, retain, copy, print,
> distribute or use this message. If you have received this communication in
> error, please notify the sender and delete all copies of this message.
> Persistent Systems Ltd. does not accept any liability for virus infected
> mails.
>

Re: How to capture failure bulletins in custom processor

[Matt Burgess]

You can use a SiteToSiteBulletinReportingTask [1], it will send flow
files containing bulletins to an Input Port on the root process group
(aka top-level canvas), then you can send to whichever processors you
like for further operations.

Regards,
Matt

[1] 
https://nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/nifi-site-to-site-reporting-nar/1.6.0/org.apache.nifi.reporting.SiteToSiteBulletinReportingTask/index.html


On Fri, May 25, 2018 at 8:46 AM, Brajendra Mishra
<brajendra_mis...@persistent.com> wrote:
> Hi Team,
>
> Could you please let me know how can I get bulletins when failure has 
> happened in executing flow file?
> I need to capture these bulletins in  custom processor (java code) and want 
> to send to next processor where I will create new error message in desired 
> format.
>
> Brajendra Mishra
>
> DISCLAIMER
> ==
> This e-mail may contain privileged and confidential information which is the 
> property of Persistent Systems Ltd. It is intended only for the use of the 
> individual or entity to which it is addressed. If you are not the intended 
> recipient, you are not authorized to read, retain, copy, print, distribute or 
> use this message. If you have received this communication in error, please 
> notify the sender and delete all copies of this message. Persistent Systems 
> Ltd. does not accept any liability for virus infected mails.

How to capture failure bulletins in custom processor

[Brajendra Mishra]

Hi Team,

Could you please let me know how can I get bulletins when failure has happened 
in executing flow file?
I need to capture these bulletins in  custom processor (java code) and want to 
send to next processor where I will create new error message in desired format.

Brajendra Mishra

DISCLAIMER
==
This e-mail may contain privileged and confidential information which is the 
property of Persistent Systems Ltd. It is intended only for the use of the 
individual or entity to which it is addressed. If you are not the intended 
recipient, you are not authorized to read, retain, copy, print, distribute or 
use this message. If you have received this communication in error, please 
notify the sender and delete all copies of this message. Persistent Systems 
Ltd. does not accept any liability for virus infected mails.

Re: Nifi Custom Processor Import Error

[Sivaprasanna]

You have to build it locally after downloading or cloning.

-
Sivaprasanna

On Fri, 20 Apr 2018 at 4:16 PM, rishabprasad005 
wrote:

> How can we download a maven dependency for nifi having version
> "1.7.0-SNAPSHOT". I tried doing this but gets an error since the latest
> version is 1.6.0.
>
>
>
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
>

Re: Nifi Custom Processor Import Error

[rishabprasad005]

How can we download a maven dependency for nifi having version
"1.7.0-SNAPSHOT". I tried doing this but gets an error since the latest
version is 1.6.0.



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Re: Nifi Custom Processor Import Error

[Pierre Villard]

Hi,

It's not in 1.5.0/1.6.0. It's only in master code at the moment.

Pierre

2018-04-17 23:17 GMT+02:00 Mike Thomsen <mikerthom...@gmail.com>:

> It was added after 1.5. Might be on 1.6. Definitely in 1.7.0-Snapshot.
> On Tue, Apr 17, 2018 at 4:56 PM Brendan Rebuck <brendan.reb...@gmail.com>
> wrote:
>
> > I am currently working on creating a custom processor. I have library
> > downloaded and referenced for nifi-api-1.5.0.jar.
> >
> > When I use the "import org.apache.nifi.expression.
> ExpressionLanguageScope"
> >
> > It gives me "error: cannot find symbol
> > [ERROR]   symbol:   class ExpressionLanguageScope
> > [ERROR]   location: package org.apache.nifi.expression"
> >
> > I have looked into the .jar file I am using and it doesn't have the
> > ExpressionLanguageScope.class. I see that the file should be in there at
> "
> >
> > https://github.com/apache/nifi/tree/master/nifi-api/src/
> main/java/org/apache/nifi/expression
> > ".
> >
> > Any help on resolving this issue would be great. Thank you.
> >
>

Re: Nifi Custom Processor Import Error

[Mike Thomsen]

It was added after 1.5. Might be on 1.6. Definitely in 1.7.0-Snapshot.
On Tue, Apr 17, 2018 at 4:56 PM Brendan Rebuck <brendan.reb...@gmail.com>
wrote:

> I am currently working on creating a custom processor. I have library
> downloaded and referenced for nifi-api-1.5.0.jar.
>
> When I use the "import org.apache.nifi.expression.ExpressionLanguageScope"
>
> It gives me "error: cannot find symbol
> [ERROR]   symbol:   class ExpressionLanguageScope
> [ERROR]   location: package org.apache.nifi.expression"
>
> I have looked into the .jar file I am using and it doesn't have the
> ExpressionLanguageScope.class. I see that the file should be in there at "
>
> https://github.com/apache/nifi/tree/master/nifi-api/src/main/java/org/apache/nifi/expression
> ".
>
> Any help on resolving this issue would be great. Thank you.
>

Nifi Custom Processor Import Error

[Brendan Rebuck]

I am currently working on creating a custom processor. I have library
downloaded and referenced for nifi-api-1.5.0.jar.

When I use the "import org.apache.nifi.expression.ExpressionLanguageScope"

It gives me "error: cannot find symbol
[ERROR]   symbol:   class ExpressionLanguageScope
[ERROR]   location: package org.apache.nifi.expression"

I have looked into the .jar file I am using and it doesn't have the
ExpressionLanguageScope.class. I see that the file should be in there at "
https://github.com/apache/nifi/tree/master/nifi-api/src/main/java/org/apache/nifi/expression
".

Any help on resolving this issue would be great. Thank you.

Re: get access token inside custom processor

[尹文才]

Thanks Bryan I got it, I thought the processor would be running as the user
that had logged into the NIFI web UI.

Regards,
Ben

2018-02-27 22:31 GMT+08:00 Bryan Bende <bbe...@gmail.com>:

> Hello,
>
> Your custom processor would be the same as if you were writing an
> external client program.
>
> You would need to provide the processor with a username and password
> in the processor properties, and then it would need to make a call to
> the token REST end-point.
>
> Processors don't run as the user from the web UI, they run on behalf
> of the NiFi framework and have no idea which user started/stopped
> them.
>
> Thanks,
>
> Bryan
>
> On Tue, Feb 27, 2018 at 1:27 AM, 尹文才 <batman...@gmail.com> wrote:
> > Hi guys, I'm trying to invoke some nifi rest apis inside my custom
> > processor, the nifi I'm using is nifi 1.4.0 and it's a 3 node secured
> > cluster., the username and password are kept inside a ldap server.
> > I know that in a secured nifi cluster, in order to make any request I
> need
> > the access token, my question is how could I get the access token in my
> > custom processor? Thanks. (I think the token should be
> > available somewhere after successful login right?)
> >
> > regards,
> > ben
>

Re: get access token inside custom processor

[Bryan Bende]

Hello,

Your custom processor would be the same as if you were writing an
external client program.

You would need to provide the processor with a username and password
in the processor properties, and then it would need to make a call to
the token REST end-point.

Processors don't run as the user from the web UI, they run on behalf
of the NiFi framework and have no idea which user started/stopped
them.

Thanks,

Bryan

On Tue, Feb 27, 2018 at 1:27 AM, 尹文才 <batman...@gmail.com> wrote:
> Hi guys, I'm trying to invoke some nifi rest apis inside my custom
> processor, the nifi I'm using is nifi 1.4.0 and it's a 3 node secured
> cluster., the username and password are kept inside a ldap server.
> I know that in a secured nifi cluster, in order to make any request I need
> the access token, my question is how could I get the access token in my
> custom processor? Thanks. (I think the token should be
> available somewhere after successful login right?)
>
> regards,
> ben

get access token inside custom processor

[尹文才]

Hi guys, I'm trying to invoke some nifi rest apis inside my custom
processor, the nifi I'm using is nifi 1.4.0 and it's a 3 node secured
cluster., the username and password are kept inside a ldap server.
I know that in a secured nifi cluster, in order to make any request I need
the access token, my question is how could I get the access token in my
custom processor? Thanks. (I think the token should be
available somewhere after successful login right?)

regards,
ben

Re: Nifi "event driven" scheduling strategy Custom Processor

[Michael Moser]

Hello!

An "event" in the context of the NiFi @EventDriven annotation is the
arrival of a flow file in the queue that is an input to a processor.  The
only purpose of @EventDriven is to save system resources when compared to
the default Timer Driven strategy, in which a processor polls its input
queues for flow files to work on.

If this matches your expectation for the Event Driven scheduling strategy,
then you must generate an "event" with another processor on the NiFi
graph.  This could be as simple as a GenerateFlowFile processor that runs
on a predictable Timer Driven period or Cron Driven schedule.  You could
also use another ingress [1], or data ingestion processor [2], to create a
new flow file containing an event.

[1] -
http://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#ingress
[2] -
http://nifi.apache.org/docs/nifi-docs/html/getting-started.html#data-ingestion

I hope this helps,
-- Mike



On Wed, Dec 27, 2017 at 10:34 AM, Milan Das <m...@interset.com> wrote:

> I am trying to build a custom processor with “Event Driven”. I added
>  “@EventDriven” annotation. It is showing EventDriven in the scheduling
> strategy.
>
> Question: how to send an event to this processor to intiate the flow? What
> are my options?
>
>
>
> Regards,
>
>
>
> [image: graph]
>
> *Milan Das*
> Sr. System Architect
>
> email: m...@interset.com
>
> [image: dIn icon] <https://www.linkedin.com/in/milandas/>
>
> www.interset.com
>
>
>
>
>
>
>

Nifi "event driven" scheduling strategy Custom Processor

[Milan Das]

I am trying to build a custom processor with “Event Driven”. I added  
“@EventDriven” annotation. It is showing EventDriven in the scheduling strategy.

Question: how to send an event to this processor to intiate the flow? What are 
my options?

 

Regards,

 

Milan Das
Sr. System Architect
email: m...@interset.com

www.interset.com

 

 

 

Re: Nifi:How can i remove “.” from “.filename” in nifi custom processor

[sally]

actually  it never  fiishes  writing process  i mean  exceptions  like this 
can't rename  file ".conf.xml"



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Can't use renameTo in nifi custom processor code

[sally]

One  of the  most  important  part  of my code is  that  it  should put  file
in folder.   i use  this  code  for  renaming ".conf.xml"   to  "conf.xml" 
but it  doesn't  work  properly:
 final Path rootDirPath =
get("C://Users//user//Desktop//try2//nifi-1.3.0//1");
final Path tempCopyFile =
rootDirPath.resolve("."+flowFile.getAttribute(CoreAttributes.FILENAME.key()));
final Path dotCopyFile =tempCopyFile;
Path tempDotCopyFile = null;
tempDotCopyFile = dotCopyFile;

final Path
copyFile=rootDirPath.resolve(flowFile.getAttribute(CoreAttributes.FILENAME.key()));
Path finalCopyFile = copyFile;
if (!Files.exists(rootDirPath)) {
if (true) {
Files.createDirectories(rootDirPath);
}
}


final Path finalCopyFileDir = finalCopyFile.getParent();
if (Files.exists(finalCopyFileDir)) { // check if too many
files already
final int numFiles =
finalCopyFileDir.toFile().list().length;

if (numFiles >= 10) {
flowFile = session.penalize(flowFile);
logger.warn("Penalizing {} and routing to 'failure'
because the output directory {} has {} files, which exceeds the "
+ "configured maximum number of files", new
Object[]{flowFile, finalCopyFileDir, numFiles});
session.transfer(flowFile, REL_FAILURE);
return;
}
}
session.exportTo(flowFile,dotCopyFile,false);

//dotCopyFile.toFile().renameTo(finalCopyFile.toFile());

logger.info("added {} to flow", new
Object[]{flowFile,flowFile1});
boolean renamed = false;
for (int i = 0; i < 10; i++) { // try rename up to 10 times.
if
(dotCopyFile.toFile().renameTo(finalCopyFile.toFile())) {
renamed = true;
break;// rename was successful
}
Thread.sleep(100L);// try waiting a few ms to let
whatever might cause rename failure to resolve
}

if (!renamed) {
if (Files.exists(dotCopyFile) &&
dotCopyFile.toFile().delete()) {
logger.debug("Deleted dot copy file {}", new
Object[]{dotCopyFile});
}
throw new ProcessException("Could not rename: " +
dotCopyFile);
} else {
logger.info("Produced copy of {} at location {}", new
Object[]{flowFile, finalCopyFile});
}



--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/

Re: Nifi:How can i remove “.” from “.filename” in nifi custom processor

[Jeff]

Sally,

The . in front of the filename is used while the contents of the file are
being written, so that the file will not be picked up by processors like
ListFile (if configured to watch the same directory to which files are
being written) until the writing is complete.  Can you confirm that the
complete file contents have been written, and the .filename is still
present in the directory?

On Thu, Oct 5, 2017 at 7:47 AM sally  wrote:

>
> I have send updated file to directory used a logic of putFile but it save
> file as i ".filenam.xml" and can't get it by getFile Processor how can
> remove "." from this file, i have read that putFile makes it automatically
> i
> mean adding dot in front of flowfile name, but how can i remove this dot?
> final Path tempCopyFile =
>
> rootDirPath.resolve("."+flowFile.getAttribute(CoreAttributes.FILENAME.key()));
> this is how it works in putFile how can i make same in my proessor without
> adding "." to filename?
>
>
>
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
>

Re: Custom processor UI exception

[Koji Kawamura]

Glad to hear you already figured it out. Thanks for your updates!

Koji

On Mon, May 22, 2017 at 10:41 AM, Yuri Krysko <yuri.kry...@mcdean.com> wrote:
> Hi Koji,
>
> Thank you for your response. Indeed, you are right. I have sorted it out
> already :) I had a custom Abstract processor class which I was extending,
> and misused init method.
>
> Thanks again!
>
> On 5/21/17, 9:38 PM, "Koji Kawamura" <ijokaruma...@gmail.com> wrote:
>
>>Hello Yuri,
>>
> >From the stack trace (line numbers) I assume you are using NiFi 1.2.0.
>>If so, I think NullPointer can be thrown if your custom processor
>>returns null from its getRelationships() method. A list of possible
>>relationships from a processor should be returned, if the processor
>>doesn't have outgoing relationship, then return
>>Collections.emptySet().
>>
>>Otherwise, please let us know which version you are using.
>>
>>Thanks,
>>Koji
>>
>>On Sat, May 20, 2017 at 12:41 AM, Yuri Krysko <yuri.kry...@mcdean.com>
>>wrote:
>>> Hello Devs,
>>>
>>> I am working on a custom processor for NiFi and I am getting the below
>>>exception in the nifi-user.log when I try to add the processor to the
>>>flow canvas. Could you please advise what could I be doing wrong. No
>>>errors in the nifi-app.log though. Thanks!
>>>
>>>
>>> 2017-05-19 11:31:07,983 ERROR [NiFi Web Server-22]
>>>o.a.nifi.web.api.config.ThrowableMapper An unexpected error has
>>>occurred: java.lang.NullPointerException. Returning Internal Server
>>>Error response.
>>>
>>> java.lang.NullPointerException: null
>>>
>>> at
>>>org.apache.nifi.web.api.dto.DtoFactory.createProcessorDto(DtoFactory.java
>>>:2226)
>>>
>>> at
>>>org.apache.nifi.web.api.dto.DtoFactory.createFlowDto(DtoFactory.java:1884
>>>)
>>>
>>> at
>>>org.apache.nifi.web.api.dto.DtoFactory.createProcessGroupFlowDto(DtoFacto
>>>ry.java:1726)
>>>
>>> at
>>>org.apache.nifi.web.StandardNiFiServiceFacade.getProcessGroupFlow(Standar
>>>dNiFiServiceFacade.java:3055)
>>>
>>> at
>>>org.apache.nifi.web.StandardNiFiServiceFacade$$FastClassBySpringCGLIB$$35
>>>8780e0.invoke()
>>>
>>> at
>>>org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
>>>
>>> at
>>>org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.inv
>>>okeJoinpoint(CglibAopProxy.java:720)
>>>
>>> at
>>>org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Refl
>>>ectiveMethodInvocation.java:157)
>>>
>>> at
>>>org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proce
>>>ed(MethodInvocationProceedingJoinPoint.java:85)
>>>
>>> at
>>>org.apache.nifi.web.NiFiServiceFacadeLock.proceedWithReadLock(NiFiService
>>>FacadeLock.java:137)
>>>
>>> at
>>>org.apache.nifi.web.NiFiServiceFacadeLock.getLock(NiFiServiceFacadeLock.j
>>>ava:108)
>>>
>>> at sun.reflect.GeneratedMethodAccessor79.invoke(Unknown Source)
>>>
>>> at
>>>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI
>>>mpl.java:43)
>>>
>>> at java.lang.reflect.Method.invoke(Method.java:498)
>>>
>>> at
>>>org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodW
>>>ithGivenArgs(AbstractAspectJAdvice.java:621)
>>>
>>> at
>>>org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(
>>>AbstractAspectJAdvice.java:610)
>>>
>>> at
>>>org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundA
>>>dvice.java:68)
>>>
>>> at
>>>org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Refl
>>>ectiveMethodInvocation.java:179)
>>>
>>> at
>>>org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(Ex
>>>poseInvocationInterceptor.java:92)
>>>
>>> at
>>>org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Refl
>>>ectiveMethodInvocation.java:179)
>>>
>>> at
>>>org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor
>>>.intercept(CglibAopProxy.java:655)
>>>
>>> at
>>>org.apache.nifi.web.StandardNiFiServiceFacade$$EnhancerBySpringCGLIB$$c6b
>>>bb5bc.getProcessGroupFlow()
>&g

Re: Custom processor UI exception

[Yuri Krysko]

Hi Koji,

Thank you for your response. Indeed, you are right. I have sorted it out
already :) I had a custom Abstract processor class which I was extending,
and misused init method.

Thanks again!

On 5/21/17, 9:38 PM, "Koji Kawamura" <ijokaruma...@gmail.com> wrote:

>Hello Yuri,
>
>From the stack trace (line numbers) I assume you are using NiFi 1.2.0.
>If so, I think NullPointer can be thrown if your custom processor
>returns null from its getRelationships() method. A list of possible
>relationships from a processor should be returned, if the processor
>doesn't have outgoing relationship, then return
>Collections.emptySet().
>
>Otherwise, please let us know which version you are using.
>
>Thanks,
>Koji
>
>On Sat, May 20, 2017 at 12:41 AM, Yuri Krysko <yuri.kry...@mcdean.com>
>wrote:
>> Hello Devs,
>>
>> I am working on a custom processor for NiFi and I am getting the below
>>exception in the nifi-user.log when I try to add the processor to the
>>flow canvas. Could you please advise what could I be doing wrong. No
>>errors in the nifi-app.log though. Thanks!
>>
>>
>> 2017-05-19 11:31:07,983 ERROR [NiFi Web Server-22]
>>o.a.nifi.web.api.config.ThrowableMapper An unexpected error has
>>occurred: java.lang.NullPointerException. Returning Internal Server
>>Error response.
>>
>> java.lang.NullPointerException: null
>>
>> at
>>org.apache.nifi.web.api.dto.DtoFactory.createProcessorDto(DtoFactory.java
>>:2226)
>>
>> at
>>org.apache.nifi.web.api.dto.DtoFactory.createFlowDto(DtoFactory.java:1884
>>)
>>
>> at
>>org.apache.nifi.web.api.dto.DtoFactory.createProcessGroupFlowDto(DtoFacto
>>ry.java:1726)
>>
>> at
>>org.apache.nifi.web.StandardNiFiServiceFacade.getProcessGroupFlow(Standar
>>dNiFiServiceFacade.java:3055)
>>
>> at
>>org.apache.nifi.web.StandardNiFiServiceFacade$$FastClassBySpringCGLIB$$35
>>8780e0.invoke()
>>
>> at
>>org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
>>
>> at
>>org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.inv
>>okeJoinpoint(CglibAopProxy.java:720)
>>
>> at
>>org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Refl
>>ectiveMethodInvocation.java:157)
>>
>> at
>>org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proce
>>ed(MethodInvocationProceedingJoinPoint.java:85)
>>
>> at
>>org.apache.nifi.web.NiFiServiceFacadeLock.proceedWithReadLock(NiFiService
>>FacadeLock.java:137)
>>
>> at
>>org.apache.nifi.web.NiFiServiceFacadeLock.getLock(NiFiServiceFacadeLock.j
>>ava:108)
>>
>> at sun.reflect.GeneratedMethodAccessor79.invoke(Unknown Source)
>>
>> at
>>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI
>>mpl.java:43)
>>
>> at java.lang.reflect.Method.invoke(Method.java:498)
>>
>> at
>>org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodW
>>ithGivenArgs(AbstractAspectJAdvice.java:621)
>>
>> at
>>org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(
>>AbstractAspectJAdvice.java:610)
>>
>> at
>>org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundA
>>dvice.java:68)
>>
>> at
>>org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Refl
>>ectiveMethodInvocation.java:179)
>>
>> at
>>org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(Ex
>>poseInvocationInterceptor.java:92)
>>
>> at
>>org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Refl
>>ectiveMethodInvocation.java:179)
>>
>> at
>>org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor
>>.intercept(CglibAopProxy.java:655)
>>
>> at
>>org.apache.nifi.web.StandardNiFiServiceFacade$$EnhancerBySpringCGLIB$$c6b
>>bb5bc.getProcessGroupFlow()
>>
>> at org.apache.nifi.web.api.FlowResource.getFlow(FlowResource.java:375)
>>
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>
>> at
>>sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java
>>:62)
>>
>> at
>>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI
>>mpl.java:43)
>>
>> at java.lang.reflect.Method.invoke(Method.java:498)
>>
>> at
>>com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethod
&g

Re: Custom processor UI exception

[Koji Kawamura]

Hello Yuri,

>From the stack trace (line numbers) I assume you are using NiFi 1.2.0.
If so, I think NullPointer can be thrown if your custom processor
returns null from its getRelationships() method. A list of possible
relationships from a processor should be returned, if the processor
doesn't have outgoing relationship, then return
Collections.emptySet().

Otherwise, please let us know which version you are using.

Thanks,
Koji

On Sat, May 20, 2017 at 12:41 AM, Yuri Krysko <yuri.kry...@mcdean.com> wrote:
> Hello Devs,
>
> I am working on a custom processor for NiFi and I am getting the below 
> exception in the nifi-user.log when I try to add the processor to the flow 
> canvas. Could you please advise what could I be doing wrong. No errors in the 
> nifi-app.log though. Thanks!
>
>
> 2017-05-19 11:31:07,983 ERROR [NiFi Web Server-22] 
> o.a.nifi.web.api.config.ThrowableMapper An unexpected error has occurred: 
> java.lang.NullPointerException. Returning Internal Server Error response.
>
> java.lang.NullPointerException: null
>
> at 
> org.apache.nifi.web.api.dto.DtoFactory.createProcessorDto(DtoFactory.java:2226)
>
> at org.apache.nifi.web.api.dto.DtoFactory.createFlowDto(DtoFactory.java:1884)
>
> at 
> org.apache.nifi.web.api.dto.DtoFactory.createProcessGroupFlowDto(DtoFactory.java:1726)
>
> at 
> org.apache.nifi.web.StandardNiFiServiceFacade.getProcessGroupFlow(StandardNiFiServiceFacade.java:3055)
>
> at 
> org.apache.nifi.web.StandardNiFiServiceFacade$$FastClassBySpringCGLIB$$358780e0.invoke()
>
> at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204)
>
> at 
> org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:720)
>
> at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
>
> at 
> org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:85)
>
> at 
> org.apache.nifi.web.NiFiServiceFacadeLock.proceedWithReadLock(NiFiServiceFacadeLock.java:137)
>
> at 
> org.apache.nifi.web.NiFiServiceFacadeLock.getLock(NiFiServiceFacadeLock.java:108)
>
> at sun.reflect.GeneratedMethodAccessor79.invoke(Unknown Source)
>
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:498)
>
> at 
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
>
> at 
> org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
>
> at 
> org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:68)
>
> at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
>
> at 
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
>
> at 
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
>
> at 
> org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
>
> at 
> org.apache.nifi.web.StandardNiFiServiceFacade$$EnhancerBySpringCGLIB$$c6bbb5bc.getProcessGroupFlow()
>
> at org.apache.nifi.web.api.FlowResource.getFlow(FlowResource.java:375)
>
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:498)
>
> at 
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
>
> at 
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
>
> at 
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
>
> at 
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
>
> at 
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
>
> at 
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
>
> at 
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
>
> at 
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.jav

Re: Prepopulate dynamic properties in custom processor?

[Joe Witt]

Russ,

I'd recommend having some highly common examples as optional
properties and then use docs to describe how they can add additional
ones of the same naming pattern, etc..  The purpose and tooling around
dynamic properties isnt' really meant that they'd be added by default.
But optional properties are a thing and should get you there.

Thanks

On Thu, Apr 27, 2017 at 9:54 AM, Russell Bateman <r...@windofkeltia.com> wrote:
> Is there a way to prepopulate dynamic properties in a custom processor?
>
> I'm writing a new processor and my users could benefit from some suggested,
> in most cases it would be spot-on, configuration. Being able to prepopulate
> dynamic properties that could be amended or deleted seems the best way.
> Incidentally, the configuration I'm talking about was to have been done via
> dynamic properties anyway.
>
> Yes, I could resort to additional details, but my guys are in a hurry when
> they use this, so cutting and pasting doesn't seem like the best help I
> could give if it's possible to do what I'm asking.
>
> Thanks,
>
> Russ

Prepopulate dynamic properties in custom processor?

[Russell Bateman]

Is there a way to prepopulate dynamic properties in a custom processor?

I'm writing a new processor and my users could benefit from some 
suggested, in most cases it would be spot-on, configuration. Being able 
to prepopulate dynamic properties that could be amended or deleted seems 
the best way. Incidentally, the configuration I'm talking about was to 
have been done via dynamic properties anyway.


Yes, I could resort to additional details, but my guys are in a hurry 
when they use this, so cutting and pasting doesn't seem like the best 
help I could give if it's possible to do what I'm asking.


Thanks,

Russ

Re: Need help with Custom Processor

[Andrew Psaltis]

Hi Pushkar,
By chance is the "A" dependency marked with a scope in "B"'s pom, such as
test, compile, or provided? If so then you would want to include "A" in
your pom as a dependency. As long as "A" is not marked with a scope that
excludes it from "B"'s final packaging or it is not excluded by one of the
maven packaging plugins then it should pick up the dependency and include
it in your NAR when you package your processor. If "A" is indeed marked
with a scope that prevents it from being included in the final NAR, you
would either want to modify B's pom or explicitly add it as a dependency to
your pom file.

Thanks,
Andrew

On Tue, Feb 28, 2017 at 2:56 PM, Pushkara R  wrote:

> Hi Andrew,
>
> The particular class that isn't being found is part of an artifact (let's
> call it A) which is a dependency of another artifact B that my processors
> depend on. A is in B's pom file and B is in my processors' pom file. Yet I
> am getting the class not found issue.
>
> I am currently working by adding A's jar in the lib folder as I am only
> doing a few benchmarks on NiFi. But please do let me know how I could fix
> this problem.
>
> Regards
> Pushkar
>
> On Tue, Feb 28, 2017 at 12:40 PM, Andrew Psaltis  >
> wrote:
>
> > Hi Pushkar,
> > The preferred approach is to add the external dependency to the pom file
> > for your processor. After doing that when you build your nar bundle, the
> > resulting nar file will contain all of the 3rd party dependencies
> required
> > by your processor. Can you double check that you are indeed including it
> as
> > a maven dependency?
> >
> > Thanks,
> > Andrew
> >
> >
> > On Tue, Feb 28, 2017 at 12:31 PM, Pushkara R 
> > wrote:
> >
> > > Hi,
> > >
> > > I'm writing a new processor for NiFi which has a dependency on an
> > external
> > > library. When I build the NAR and add it to the lib folder and restart
> > > NiFi, the external library classes that my processor depends on fail
> > > getting loaded with a "ClassNotFound" exception.
> > >
> > > How do I go about fixing this issue? Do I add the jar for the external
> > > library separately in the lib folder of NiFi? Or is there something
> else
> > to
> > > it?
> > >
> > > Thanks
> > > Pushkar
> > >
> >
> >
> >
> > --
> > Thanks,
> > Andrew
> >
> > Subscribe to my book: Streaming Data 
> > 
> > twiiter: @itmdata 
> >
>



-- 
Thanks,
Andrew

Subscribe to my book: Streaming Data 

twiiter: @itmdata 

Re: Need help with Custom Processor

[Pushkara R]

Hi Andrew,

The particular class that isn't being found is part of an artifact (let's
call it A) which is a dependency of another artifact B that my processors
depend on. A is in B's pom file and B is in my processors' pom file. Yet I
am getting the class not found issue.

I am currently working by adding A's jar in the lib folder as I am only
doing a few benchmarks on NiFi. But please do let me know how I could fix
this problem.

Regards
Pushkar

On Tue, Feb 28, 2017 at 12:40 PM, Andrew Psaltis 
wrote:

> Hi Pushkar,
> The preferred approach is to add the external dependency to the pom file
> for your processor. After doing that when you build your nar bundle, the
> resulting nar file will contain all of the 3rd party dependencies required
> by your processor. Can you double check that you are indeed including it as
> a maven dependency?
>
> Thanks,
> Andrew
>
>
> On Tue, Feb 28, 2017 at 12:31 PM, Pushkara R 
> wrote:
>
> > Hi,
> >
> > I'm writing a new processor for NiFi which has a dependency on an
> external
> > library. When I build the NAR and add it to the lib folder and restart
> > NiFi, the external library classes that my processor depends on fail
> > getting loaded with a "ClassNotFound" exception.
> >
> > How do I go about fixing this issue? Do I add the jar for the external
> > library separately in the lib folder of NiFi? Or is there something else
> to
> > it?
> >
> > Thanks
> > Pushkar
> >
>
>
>
> --
> Thanks,
> Andrew
>
> Subscribe to my book: Streaming Data 
> 
> twiiter: @itmdata 
>

Re: Need help with Custom Processor

[Andrew Psaltis]

Hi Pushkar,
The preferred approach is to add the external dependency to the pom file
for your processor. After doing that when you build your nar bundle, the
resulting nar file will contain all of the 3rd party dependencies required
by your processor. Can you double check that you are indeed including it as
a maven dependency?

Thanks,
Andrew


On Tue, Feb 28, 2017 at 12:31 PM, Pushkara R  wrote:

> Hi,
>
> I'm writing a new processor for NiFi which has a dependency on an external
> library. When I build the NAR and add it to the lib folder and restart
> NiFi, the external library classes that my processor depends on fail
> getting loaded with a "ClassNotFound" exception.
>
> How do I go about fixing this issue? Do I add the jar for the external
> library separately in the lib folder of NiFi? Or is there something else to
> it?
>
> Thanks
> Pushkar
>



-- 
Thanks,
Andrew

Subscribe to my book: Streaming Data 

twiiter: @itmdata 

Need help with Custom Processor

[Pushkara R]

Hi,

I'm writing a new processor for NiFi which has a dependency on an external
library. When I build the NAR and add it to the lib folder and restart
NiFi, the external library classes that my processor depends on fail
getting loaded with a "ClassNotFound" exception.

How do I go about fixing this issue? Do I add the jar for the external
library separately in the lib folder of NiFi? Or is there something else to
it?

Thanks
Pushkar

Re: Questions on Topology and custom-processor deployment on NiFi and MiNiFi

[Aldrin Piri]

Uwe, great points and information about the current state of interacting
with MiNiFi instances.

Pushkar,

As part of the process to aid in updating instances there is also a feature
proposal detailing some of the high level aims of better orchestration and
management to MiNiFi instances through command and control (C2).  If you
have not seen this yet, there is a write up of some of the items collected
through various community discussions available in the MiNiFi Wiki [1].

The first attempt at updates with C2 will likely necessitate a full
restart.  I think there is certainly a possibility to perhaps work around
this with some constraints and caveats about one's deployed dataflow but
will likely need to be an exercise once we have some more foundational
components and functionality in place.

Please feel free to comment on any of the materials currently available and
let us know if there are any additional questions we can answer or comments
you would like to discuss.

--aldrin

[1]
https://cwiki.apache.org/confluence/display/MINIFI/MiNiFi+Command+and+Control


On Tue, Jan 31, 2017 at 1:56 PM, u...@moosheimer.com 
wrote:

> Hi Pushkar,
>
> you can automatically update the MiniFi configs using one of three methods:
>
>   * FileChangeIngestor
>   * RestChangeIngestor
>   * PullHttpChangeIngestor
>
> You have to configure the MiNiFi bootstrap.conf.
> Here you can read how to do this:
> https://nifi.apache.org/minifi/system-admin-guide.html
>
> To update the MiNiFi config you have to create a Process Group (at NiFi
> UI) where you define the MiNiFi workflow.
> Save the Process Group as template.
>
> Export the template and convert it to MiNiFi config.yml by using the
> MiNiFi Toolkit (https://nifi.apache.org/minifi/download.html).
>
> Hot to transform the template to config.yml?
> Change to MINIFI_TOOLKIT_HOME and type: /bin/config.sh transform
> [PATH_TO_TEMPLATE_FILE/TEMPLATE_FILE_NAME]
> [PATH_TO_CONFIG_GFILE/CONFIG_FILE_NAME]
>
> You don't need to stop anything. If MiNiFi gets a new config then it
> checks validity and restarts itself.
>
> Best Regards,
> Uwe Moosheimer
>
> Am 31.01.2017 um 19:00 schrieb Pushkara R:
> > Hi,
> >
> > I am a part of a Research Lab in Indian Institute of Science and we are
> > currently evaluating MiNiFi as candidate for out IoT deployments. There
> are
> > two things I wanted get clarified as I could not find relavant
> > documentation.
> >
> > 1. It is mentioned in multiple places that using NiFi we can update the
> > processor Topology of multiple instances of MiNiFi that are connected to
> > it. However, I am not able to figure out how. I assume it would involve
> > communication with the MiNiFi bootstrap and passing it the relavant yml
> > file, but I don't know how to do it. Could someone please elaborate?
> >
> > 2. On NiFi, the processor topology can be updated in the UI without
> > stopping the entire topology. Only the concerned processors need to be
> > stopped. Though MiNiFi does not have a UI, is it necessary that every
> minor
> > edit to the topology needs a restart? Is there a way to go around the
> > necessity to restart MiNiFi entirely?
> >
> > Regards
> > Pushkar
> >
>
>

Re: Questions on Topology and custom-processor deployment on NiFi and MiNiFi

[u...@moosheimer.com]

Hi Pushkar,

you can automatically update the MiniFi configs using one of three methods:

  * FileChangeIngestor
  * RestChangeIngestor
  * PullHttpChangeIngestor

You have to configure the MiNiFi bootstrap.conf.
Here you can read how to do this:
https://nifi.apache.org/minifi/system-admin-guide.html

To update the MiNiFi config you have to create a Process Group (at NiFi
UI) where you define the MiNiFi workflow.
Save the Process Group as template.

Export the template and convert it to MiNiFi config.yml by using the
MiNiFi Toolkit (https://nifi.apache.org/minifi/download.html).

Hot to transform the template to config.yml?
Change to MINIFI_TOOLKIT_HOME and type: /bin/config.sh transform
[PATH_TO_TEMPLATE_FILE/TEMPLATE_FILE_NAME]
[PATH_TO_CONFIG_GFILE/CONFIG_FILE_NAME]

You don't need to stop anything. If MiNiFi gets a new config then it
checks validity and restarts itself.

Best Regards,
Uwe Moosheimer

Am 31.01.2017 um 19:00 schrieb Pushkara R:
> Hi,
>
> I am a part of a Research Lab in Indian Institute of Science and we are
> currently evaluating MiNiFi as candidate for out IoT deployments. There are
> two things I wanted get clarified as I could not find relavant
> documentation.
>
> 1. It is mentioned in multiple places that using NiFi we can update the
> processor Topology of multiple instances of MiNiFi that are connected to
> it. However, I am not able to figure out how. I assume it would involve
> communication with the MiNiFi bootstrap and passing it the relavant yml
> file, but I don't know how to do it. Could someone please elaborate?
>
> 2. On NiFi, the processor topology can be updated in the UI without
> stopping the entire topology. Only the concerned processors need to be
> stopped. Though MiNiFi does not have a UI, is it necessary that every minor
> edit to the topology needs a restart? Is there a way to go around the
> necessity to restart MiNiFi entirely?
>
> Regards
> Pushkar
>

Questions on Topology and custom-processor deployment on NiFi and MiNiFi

[Pushkara R]

Hi,

I am a part of a Research Lab in Indian Institute of Science and we are
currently evaluating MiNiFi as candidate for out IoT deployments. There are
two things I wanted get clarified as I could not find relavant
documentation.

1. It is mentioned in multiple places that using NiFi we can update the
processor Topology of multiple instances of MiNiFi that are connected to
it. However, I am not able to figure out how. I assume it would involve
communication with the MiNiFi bootstrap and passing it the relavant yml
file, but I don't know how to do it. Could someone please elaborate?

2. On NiFi, the processor topology can be updated in the UI without
stopping the entire topology. Only the concerned processors need to be
stopped. Though MiNiFi does not have a UI, is it necessary that every minor
edit to the topology needs a restart? Is there a way to go around the
necessity to restart MiNiFi entirely?

Regards
Pushkar

Re: Custom Processor

[Matt Burgess]

I answered on the users list but will copy here in case there are
folks on dev but not users:

Are your values Strings or numbers? Meaning does the JSON look like:

{ "a": "2.1234567891E10" }
 or
{ "a" : 2.1234567891E10 }

If the latter, would the output field ("a" or "new_a" or whatever)
have to remain a number, or is a String ok?  I think most
applications/libraries will default the "printed" version of a number
to Java's Number.toString() method, so if you wanted any number to be
displayed with the thousands separator, then you really are doing
string manipulation vs math, so what you propose should work fine.
Adding a thousands separator is itself a string operation as there is
no real concept of a separator when it comes to dealing with numbers
as numbers.

If you just want to change the "manageable" numbers to human readable
format (and have them remain Numbers), you could use ExecuteScript
with Groovy and just read in and write out the JSON object, Groovy
will represent some of the example values in their more human-readable
form.  For example, I started with this JSON:

{
  "a": 1,
  "b": 2.23456789E6,
  "c": 2.1234567891E10,
  "d": 15.123456789E7,
  "e": 2.1234567891E23
}


And used this Groovy script in ExecuteScript:

import groovy.json.*
import org.apache.commons.io.IOUtils
import java.nio.charset.StandardCharsets

def flowFile = session.get()
if(!flowFile) return
flowFile = session.write(flowFile, { inputStream, outputStream ->
   def json = new
JsonSlurper().parseText(IOUtils.toString(inputStream,
StandardCharsets.UTF_8))
   
outputStream.write(JsonOutput.prettyPrint(JsonOutput.toJson(json)).getBytes(StandardCharsets.UTF_8))
} as StreamCallback)
session.transfer(flowFile, REL_SUCCESS)


And got this result:
{
"a": 1,
"b": 2234567.89,
"c": 21234567891,
"d": 151234567.89,
"e": 2.1234567891E+23
}

Your original example values ("a" through "d") have been "expanded",
where the one I added ("e", with a large exponent) has remained in
scientific notation.

Alternatively, if you have a flat JSON object and don't mind that the
numeric values will be replaced by strings, you can add the following
to the script before the outputStream.write() line:

json.each {k,v ->
  json[k] = java.text.NumberFormat.getNumberInstance(Locale.US).format(v)
}


to get this output:
{
"a": "10,000",
"b": "2,234,567.89",
"c": "21,234,567,891",
"d": "151,234,567.89",
"e": "212,345,678,910,000,000,000,000"
}

If your JSON is not flat but you know which fields contain the numbers
you wish to transform, you can refer to those fields in dot notation
(vs doing the json.each() to get each top-level key/value pair) as


they are just Maps once the JSON has been "slurped" [1].

Regards,
Matt

[1] http://groovy-lang.org/json.html


On Thu, Jan 26, 2017 at 10:53 AM, Sven Davison <svendavi...@gmail.com> wrote:
> I have a json object the SOMETIMES contains a scientific notation for the
> value. I want to have a nice or "human readable" number w/ thousands
> separator.
>
> example values:
> 1
> 2.23456789E6
> 2.1234567891E10
> 15.123456789E7
>
> what i thought of doing was checking for the existance of E. then forking
> it in the flow. if it has an E.. get the value to the right of the E. split
> the string into an array (including the .). finding the index location of
> the "." then moving it X positions per value indicated on the right of the
> E. once thats done, go back and add the thousands seperator "," for easy
> reading.
>
> this is string manipulation rather than "math".. Can anyone recomend an
> easier way?
>
> I've never built a custom processor.. this might be the time for it.
> Setting a flowfileAttribute to the putput value would work for me.
>
> i need to read up and find some documentation/tutorials on how to make your
> own processor...

Custom Processor

[Sven Davison]

I have a json object the SOMETIMES contains a scientific notation for the
value. I want to have a nice or "human readable" number w/ thousands
separator.

example values:
1
2.23456789E6
2.1234567891E10
15.123456789E7

what i thought of doing was checking for the existance of E. then forking
it in the flow. if it has an E.. get the value to the right of the E. split
the string into an array (including the .). finding the index location of
the "." then moving it X positions per value indicated on the right of the
E. once thats done, go back and add the thousands seperator "," for easy
reading.

this is string manipulation rather than "math".. Can anyone recomend an
easier way?

I've never built a custom processor.. this might be the time for it.
Setting a flowfileAttribute to the putput value would work for me.

i need to read up and find some documentation/tutorials on how to make your
own processor...

Re: Global property for custom processor

[Andy LoPresto]

 to shut off such items.  Spitballing ideas here but
>>> what about giving you the ability to tag components with some label
>>> and then be able to do global execution of some task
>>> (stop/start/disable/delete/etc..) against components that you're
>>> authorized to and which have those labels.
>>> 
>>> Do you think this would be a typical use case or do you feel this is
>>> useful because you're testing right now?  Does the above idea make
>>> sense or do you have other suggestions?
>>> 
>>> Thanks
>>> Joe
>>> 
>>> On Wed, Nov 30, 2016 at 3:14 PM, Russell Bateman <r...@windofkeltia.com 
>>> <mailto:r...@windofkeltia.com>> wrote:
>>>> I've written a custom processor for some trivial profiling, time-stamping,
>>>> time-since, histogram-generating, etc., but would like the ability to turn
>>>> all instances completely off without having to visit each instance in the
>>>> UI. If it works out, I might consider even leaving some instances in
>>>> production- or at least staging-environment flows.
>>>> 
>>>> 1. I know that the NiFi Expression Language has access to various system- 
>>>> or
>>>> NiFi properties or settings, but what would someone suggest as best 
>>>> practice
>>>> for this? (Don't invade /conf/nifi.properties/, etc.)
>>>> 
>>>> 2. I guess I'd add a property to configure in my processor and check 
>>>> whether
>>>> it evaluates true/false/etc. based on the source data (whatever that will
>>>> be--see previous paragraph)?
>>>> 
>>>> 3. Last, if this processor is thereby reduced merely to
>>>> 
>>>>session.transfer( flowfile, SUCCESS );
>>>> 
>>>> there isn't any handling even more minimal or faster than that in the sense
>>>> of turning a processor off, right?
>>>> 
>>>> Thanks for any suggestions,
>>>> 
>>>> Russ
>> 
> 



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Global property for custom processor

[Andy LoPresto]

Hi Russ,

Could you use the cluster state manager [1] to hold this boolean trigger value 
which each instance of your customer processor checks before execution, and 
then use a simple ExecuteScript processor which simply toggles/explicitly 
writes that value? In this way, the ExecuteScript processor is like the light 
switch. You can manually start/stop that processor to trigger or stop all the 
others, or use the REST API to do the same, or even make the ExecuteScript 
processor read from a system/environment variable or the absence/presence/value 
of a file on disk to get the desired state value.

The ExecuteScript processor might have to abuse the StandardStateManager by 
first enumerating all instances of the desired “controllable” components (this 
could be achieved by dynamically querying a containing process group for 
processors by type or manually populating a list of component IDs in a static 
list, which the ExecuteScript processor could then store in its own 
StateManager) and then manually instantiating a StateManager containing the 
local/cluster StateProvider [2] for each component ID and setting the state.

Not sure if I explained that well, but Mark Payne would be your guy for a 
better explanation and possibly a cleaner solution.

[1] 
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#state_management
[2] 
https://github.com/apache/nifi/blob/master/nifi-framework-api/src/main/java/org/apache/nifi/components/state/StateProvider.java#L66
 
<https://github.com/apache/nifi/blob/master/nifi-framework-api/src/main/java/org/apache/nifi/components/state/StateProvider.java#L66>


Andy LoPresto
alopre...@apache.org
alopresto.apa...@gmail.com
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4  BACE 3C6E F65B 2F7D EF69

> On Nov 30, 2016, at 2:00 PM, Russell Bateman <r...@windofkeltia.com> wrote:
> 
> Our usage, for ETL, is controlled very up-close and personal right now. Our 
> ETL of medical documents is pretty involved, changes radically from customer 
> to customer, and must be baby-sat closely. Anything we're able to do for our 
> implementation folk to enable them to pin down waste of resources including 
> and especially time to ingest horrendous quantities of information is going 
> to serve us for a long time to come. User access to an easy processor like 
> the one I've written, though what it does is pretty trivial, makes their life 
> so much easier and they can talk back to us about where time (in particular) 
> is being spent, in which processor (we have lots of custom processors that do 
> very out-of-the-ordinary things), across which subflow, etc.
> 
> Except that we anticipate moving to a clustered implementation soon, I 
> thought about merely looking for a system environment variable or even the 
> presence of a file, then setting static state inside the processor to halt 
> doing anything. Conversely, a change to that state might start the processor 
> back up again (time-stamping, histogramming, etc.). I think this naïve 
> control strategy falls apart as soon we go to a cluster.
> 
> It's taking me a while to get into the NiFi culture, I think. However, I also 
> think that NiFi folk use NiFi in wildly different ways so maybe how I'm 
> looking to do something isn't always so un-NiFi, but that others just haven't 
> tackled it yet.
> 
> Yeah, if NiFi gave us some kind of modifiable, global state, especially if 
> less static than /conf/nifi.properties/, but even if requiring a bounce to 
> engage it (so, /conf/flow.properties/ or /conf/flow.conf/), that would solve 
> our problem pretty elegantly. However, I haven't thought about what problems 
> it also creates for you or others.
> 
> Russ
> 
> 
> On 11/30/2016 02:42 PM, Joe Witt wrote:
>> Russ
>> 
>> I don't think we provide anything particularly helpful here to do this
>> conveniently.  You could of course script this external to NiFi to
>> make HTTP calls to shut off such items.  Spitballing ideas here but
>> what about giving you the ability to tag components with some label
>> and then be able to do global execution of some task
>> (stop/start/disable/delete/etc..) against components that you're
>> authorized to and which have those labels.
>> 
>> Do you think this would be a typical use case or do you feel this is
>> useful because you're testing right now?  Does the above idea make
>> sense or do you have other suggestions?
>> 
>> Thanks
>> Joe
>> 
>> On Wed, Nov 30, 2016 at 3:14 PM, Russell Bateman <r...@windofkeltia.com> 
>> wrote:
>>> I've written a custom processor for some trivial profiling, time-stamping,
>>> time-since, histogram-generating, etc., but would like the ability to turn
>>> all instances completely off without having to visit ea

Re: Global property for custom processor

[Russell Bateman]

Our usage, for ETL, is controlled very up-close and personal right now. 
Our ETL of medical documents is pretty involved, changes radically from 
customer to customer, and must be baby-sat closely. Anything we're able 
to do for our implementation folk to enable them to pin down waste of 
resources including and especially time to ingest horrendous quantities 
of information is going to serve us for a long time to come. User access 
to an easy processor like the one I've written, though what it does is 
pretty trivial, makes their life so much easier and they can talk back 
to us about where time (in particular) is being spent, in which 
processor (we have lots of custom processors that do very 
out-of-the-ordinary things), across which subflow, etc.


Except that we anticipate moving to a clustered implementation soon, I 
thought about merely looking for a system environment variable or even 
the presence of a file, then setting static state inside the processor 
to halt doing anything. Conversely, a change to that state might start 
the processor back up again (time-stamping, histogramming, etc.). I 
think this naïve control strategy falls apart as soon we go to a cluster.


It's taking me a while to get into the NiFi culture, I think. However, I 
also think that NiFi folk use NiFi in wildly different ways so maybe how 
I'm looking to do something isn't always so un-NiFi, but that others 
just haven't tackled it yet.


Yeah, if NiFi gave us some kind of modifiable, global state, especially 
if less static than /conf/nifi.properties/, but even if requiring a 
bounce to engage it (so, /conf/flow.properties/ or /conf/flow.conf/), 
that would solve our problem pretty elegantly. However, I haven't 
thought about what problems it also creates for you or others.


Russ


On 11/30/2016 02:42 PM, Joe Witt wrote:

Russ

I don't think we provide anything particularly helpful here to do this
conveniently.  You could of course script this external to NiFi to
make HTTP calls to shut off such items.  Spitballing ideas here but
what about giving you the ability to tag components with some label
and then be able to do global execution of some task
(stop/start/disable/delete/etc..) against components that you're
authorized to and which have those labels.

Do you think this would be a typical use case or do you feel this is
useful because you're testing right now?  Does the above idea make
sense or do you have other suggestions?

Thanks
Joe

On Wed, Nov 30, 2016 at 3:14 PM, Russell Bateman <r...@windofkeltia.com> wrote:

I've written a custom processor for some trivial profiling, time-stamping,
time-since, histogram-generating, etc., but would like the ability to turn
all instances completely off without having to visit each instance in the
UI. If it works out, I might consider even leaving some instances in
production- or at least staging-environment flows.

1. I know that the NiFi Expression Language has access to various system- or
NiFi properties or settings, but what would someone suggest as best practice
for this? (Don't invade /conf/nifi.properties/, etc.)

2. I guess I'd add a property to configure in my processor and check whether
it evaluates true/false/etc. based on the source data (whatever that will
be--see previous paragraph)?

3. Last, if this processor is thereby reduced merely to

session.transfer( flowfile, SUCCESS );

there isn't any handling even more minimal or faster than that in the sense
of turning a processor off, right?

Thanks for any suggestions,

Russ

Re: Global property for custom processor

[Joe Witt]

Russ

I don't think we provide anything particularly helpful here to do this
conveniently.  You could of course script this external to NiFi to
make HTTP calls to shut off such items.  Spitballing ideas here but
what about giving you the ability to tag components with some label
and then be able to do global execution of some task
(stop/start/disable/delete/etc..) against components that you're
authorized to and which have those labels.

Do you think this would be a typical use case or do you feel this is
useful because you're testing right now?  Does the above idea make
sense or do you have other suggestions?

Thanks
Joe

On Wed, Nov 30, 2016 at 3:14 PM, Russell Bateman <r...@windofkeltia.com> wrote:
> I've written a custom processor for some trivial profiling, time-stamping,
> time-since, histogram-generating, etc., but would like the ability to turn
> all instances completely off without having to visit each instance in the
> UI. If it works out, I might consider even leaving some instances in
> production- or at least staging-environment flows.
>
> 1. I know that the NiFi Expression Language has access to various system- or
> NiFi properties or settings, but what would someone suggest as best practice
> for this? (Don't invade /conf/nifi.properties/, etc.)
>
> 2. I guess I'd add a property to configure in my processor and check whether
> it evaluates true/false/etc. based on the source data (whatever that will
> be--see previous paragraph)?
>
> 3. Last, if this processor is thereby reduced merely to
>
>session.transfer( flowfile, SUCCESS );
>
> there isn't any handling even more minimal or faster than that in the sense
> of turning a processor off, right?
>
> Thanks for any suggestions,
>
> Russ

Global property for custom processor

[Russell Bateman]

I've written a custom processor for some trivial profiling, 
time-stamping, time-since, histogram-generating, etc., but would like 
the ability to turn all instances completely off without having to visit 
each instance in the UI. If it works out, I might consider even leaving 
some instances in production- or at least staging-environment flows.


1. I know that the NiFi Expression Language has access to various 
system- or NiFi properties or settings, but what would someone suggest 
as best practice for this? (Don't invade /conf/nifi.properties/, etc.)


2. I guess I'd add a property to configure in my processor and check 
whether it evaluates true/false/etc. based on the source data (whatever 
that will be--see previous paragraph)?


3. Last, if this processor is thereby reduced merely to

   session.transfer( flowfile, SUCCESS );

there isn't any handling even more minimal or faster than that in the 
sense of turning a processor off, right?


Thanks for any suggestions,

Russ

Re: Issue on creating Custom Processor - Salesforce

[Jeremy Dyer]

Shankhamajumdar - I'd be glad to help you out but how about you email me
personally outside of this thread since this project isn't formally part of
the Apache community. Look forward to hearing from you

On Fri, Oct 14, 2016 at 8:29 AM, shankhamajumdar <
shankha.majum...@lexmark.com> wrote:

> Hi Jeremy,
>
> There is one more issue. Actually I did the build inside
> D:\nifi-addons-master\Services\nifi-salesforce-
> service\nifi-salesforce-api-nar
> directory successfully. But ideally I should build inside
> D:\nifi-addons-master\Services directory which is failing.
>
> [ERROR] [ERROR] Some problems were encountered while processing the POMs:
> [ERROR] Child module D:\nifi-addons-master\Services\nifi-google-auth-
> service
> of
> D:\nifi-addons-master\Services\pom.xml does not exist @
>  @
>
> I do not see nifi-google-auth-service as well in the
> D:\nifi-addons-master\Services directory.
>
> Regards,
> Shankha
>
>
>
>
>
> --
> View this message in context: http://apache-nifi-developer-
> list.39713.n7.nabble.com/Issue-on-creating-Custom-Processor-Salesforce-
> tp13603p13610.html
> Sent from the Apache NiFi Developer List mailing list archive at
> Nabble.com.
>

Re: Issue on creating Custom Processor - Salesforce

[shankhamajumdar]

Hi Jeremy,

There is one more issue. Actually I did the build inside
D:\nifi-addons-master\Services\nifi-salesforce-service\nifi-salesforce-api-nar
directory successfully. But ideally I should build inside
D:\nifi-addons-master\Services directory which is failing.

[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[ERROR] Child module D:\nifi-addons-master\Services\nifi-google-auth-service
of
D:\nifi-addons-master\Services\pom.xml does not exist @
 @

I do not see nifi-google-auth-service as well in the
D:\nifi-addons-master\Services directory.

Regards,
Shankha





--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/Issue-on-creating-Custom-Processor-Salesforce-tp13603p13610.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: Issue on creating Custom Processor - Salesforce

[shankhamajumdar]

Thanks Jeremy, now the build is successful.

Regards,
Shankha



--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/Issue-on-creating-Custom-Processor-Salesforce-tp13603p13605.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Re: Issue on creating Custom Processor - Salesforce

[Jeremy Dyer]

Shankhamajumdar - You are seeing this because the Salesforce controller 
services haven't been built yet. If you navigate to the {nifi_addons}/services 
directory and run a mvn clean install and then reattempt to build the 
processors the error should go away. 

- Jeremy Dyer

> On Oct 14, 2016, at 2:03 AM, shankhamajumdar <shankha.majum...@lexmark.com> 
> wrote:
> 
> Hi,
> 
> I am trying to create the Custom Salesforce Processor for Nifi but getting
> the below error while doing the maven build.
> 
> [ERROR] Failed to execute goal on project nifi-salesforce-processors: Could
> not
> resolve dependencies for project
> com.jeremydyer.nifi:nifi-salesforce-processors:
> jar:1.0.0: Could not find artifact
> com.jeremydyer.nifi:nifi-salesforce-api:jar:1
> .0.0 in central (https://repo1.maven.org/maven2) -> [Help 1]
> 
> The maven build for other Custom Processors are working fine.
> 
> Regards,
> Shankha
> 
> 
> 
> 
> 
> --
> View this message in context: 
> http://apache-nifi-developer-list.39713.n7.nabble.com/Issue-on-creating-Custom-Processor-Salesforce-tp13603.html
> Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.

Issue on creating Custom Processor - Salesforce

[shankhamajumdar]

Hi,

I am trying to create the Custom Salesforce Processor for Nifi but getting
the below error while doing the maven build.

[ERROR] Failed to execute goal on project nifi-salesforce-processors: Could
not
resolve dependencies for project
com.jeremydyer.nifi:nifi-salesforce-processors:
jar:1.0.0: Could not find artifact
com.jeremydyer.nifi:nifi-salesforce-api:jar:1
.0.0 in central (https://repo1.maven.org/maven2) -> [Help 1]

The maven build for other Custom Processors are working fine.

Regards,
Shankha





--
View this message in context: 
http://apache-nifi-developer-list.39713.n7.nabble.com/Issue-on-creating-Custom-Processor-Salesforce-tp13603.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.