Re: IMPORTANT -- Lunch and Learn - NiFi

[Devin Fisher]

Sorry, this was for some company nifi training. Did not mean to send to
nifi dev mailing list.

Sorry again.

Devin

On Wed, Oct 19, 2016 at 11:43 AM, Joe Percivall <
joeperciv...@yahoo.com.invalid> wrote:

> Hello Devin,
>
> I'm a bit confused but I don't think you meant to send this to the Apache
> dev mailing list.
>
> Joe
>  - - - - - -
> Joseph Percivall
> linkedin.com/in/Percivall
> e: joeperciv...@yahoo.com
>
>
>
>
> On Wednesday, October 19, 2016 1:40 PM, Devin Fisher <devin.fisher@
> perfectsearchcorp.com> wrote:
> New like for Template: (fixed a typo)
>
> https://drive.google.com/open?id=0B_BTD18gUE3CeHAxSExfd3Q2T2s
>
>
> On Wed, Oct 19, 2016 at 11:23 AM, Devin Fisher <
> devin.fis...@perfectsearchcorp.com> wrote:
>
> > This week Info: (Hopefully this one is correct)
> >
> > <https://drive.google.com/open?id=0B_BTD18gUE3CZkNpZ3RiRVhHZ2M>
> > Nifi HW Template
> > <https://drive.google.com/open?id=0B_BTD18gUE3CZkNpZ3RiRVhHZ2M> -- You
> > will need this for the Homework.
> > Slide Show Presentation
> > <https://drive.google.com/open?id=1M01jTuDBey-
> 8t6xovAbkqo9CQDoy7BISTTjzsV0pkIA>
> >
>

Re: IMPORTANT -- Lunch and Learn - NiFi

[Devin Fisher]

New like for Template: (fixed a typo)

https://drive.google.com/open?id=0B_BTD18gUE3CeHAxSExfd3Q2T2s

On Wed, Oct 19, 2016 at 11:23 AM, Devin Fisher <
devin.fis...@perfectsearchcorp.com> wrote:

> This week Info: (Hopefully this one is correct)
>
> <https://drive.google.com/open?id=0B_BTD18gUE3CZkNpZ3RiRVhHZ2M>
> Nifi HW Template
> <https://drive.google.com/open?id=0B_BTD18gUE3CZkNpZ3RiRVhHZ2M> -- You
> will need this for the Homework.
> Slide Show Presentation
> <https://drive.google.com/open?id=1M01jTuDBey-8t6xovAbkqo9CQDoy7BISTTjzsV0pkIA>
>

Re: IMPORTANT -- Lunch and Learn - NiFi

[Devin Fisher]

This week Info: (Hopefully this one is correct)


Nifi HW Template
 -- You will
need this for the Homework.
Slide Show Presentation

Re: IMPORTANT -- Lunch and Learn - NiFi

[Devin Fisher]

This week Info:

Nifi HW Template
<https://drive.google.com/open?id=0B_BTD18gUE3CZkNpZ3RiRVhHZ2M> -- You will
need this for the Homework.
Slide Show Presentation
<https://drive.google.com/open?id=1M01jTuDBey-8t6xovAbkqo9CQDoy7BISTTjzsV0pkIA>

On Wed, Oct 12, 2016 at 11:37 AM, Lara Leavitt <
lara.leav...@perfectsearchcorp.com> wrote:

> Hi,
>
> I've installed nifi locally, but spreadsheet is locked for me to mark it.
>
> -Lara
>
> On Mon, Oct 10, 2016 at 1:06 PM, Devin Fisher <devin.fisher@
> perfectsearchcorp.com> wrote:
>
>> This week (on Wednesday during lunch and learn) we as a team are going to
>> be learning about our ETL tool NIFI.
>>
>> For this training, everyone will need to do a few things before we get
>> into the training. I've broken these prerequisites into two categories.
>>
>> *Please DO RIGHT NOW:*
>>
>>1. Go to the HW spreadsheet
>>
>> <https://docs.google.com/a/perfectsearchcorp.com/spreadsheets/d/1R03MxW4xgWRuYMmjN_tyY1EqrlJz0IitTv1jj5ypGm0/edit?usp=sharing>
>>and make sure your name is on the list. (if not please add it)
>>
>> (if your name was not on the list, you may consider checking the
>> employee directory
>> <https://docs.google.com/a/perfectsearchcorp.com/spreadsheets/d/1QcrtdshBzi9_hjWj6j-oGoFX9Zv2m7SeR-UBLOXXv58/edit?usp=sharing>
>> and make sure you are on that list)
>>
>> Please *Do Before Wednesday:*
>>
>>1. Have a IMAT Nifi Instance available for your use. This can take
>>the following forms:
>>   - A fully installed IMAT appliance (installed and configured).
>>   - ETL dev sandbox (mostly applies to Russ and I).
>>   - Local Installed using instructions below.
>>2. Once you have an IMAT Nifi Instance for you and only you to use.
>>Please revisit the HW spreadsheet
>>
>> <https://docs.google.com/a/perfectsearchcorp.com/spreadsheets/d/1R03MxW4xgWRuYMmjN_tyY1EqrlJz0IitTv1jj5ypGm0/edit?usp=sharing>
>>  and
>>mark your name as having Nifi Available.
>>
>>
>> Local Install of Nifi:
>>
>>1. Make sure you have Java 7 or greater installed. Test: java -version
>>2. Download Nifi with IMAT
>><http://10.10.10.240/data/user/devin.fisher/lunchLearn/nifi-0.7.0-bin.zip>
>>extensions from fortandy. (http://10.10.10.240/data/user
>>/devin.fisher/lunchLearn/nifi-0.7.0-bin.zip
>><http://10.10.10.240/data/user/devin.fisher/lunchLearn/nifi-0.7.0-bin.zip>
>>)
>>3. Extract Nifi zip to any directory.
>>4. In */nifi-0.7.0/bin* and run:
>>   1. Windows: *run-nifi.bat*
>>   2. Linux/macOS: *./nifi.sh start*
>>5. Allow time for nifi to start up (up to 3 mins)
>>6. In browser go to http://127.0.0.1:8799/nifi
>>
>> If you have any issue installing please let me know.
>>
>
>
>
> --
> Lara Leavitt
> User Interface Engineer
> Perfect Search Corporation
>

Re: Error: Self-suppression not permitted

[Devin Fisher]

Thanks, I'll see if I can do that. Might be difficult since we don't build
our own nifi binaries. I could use the 0.7.x RC when it comes out.

But could you tell me what processor you were using when you had that issue
because the error looks different? Also, if that processor was an
in-house one, did it like the ticket mention write the content more than
once?  I don't believe our processors are writing more than once the
content of the flowfile.

Devin

On Wed, Oct 12, 2016 at 4:07 PM, Michael Moser <moser...@gmail.com> wrote:

> Devin,
>
> This sounds a lot like the issue documented in NIFI-2551 [1].  Since you
> are using NiFi 0.7.0, you might try to build the latest code in the 0.x
> branch and see if this version resolves your problem.
>
> -- Mike
>
> [1] - https://issues.apache.org/jira/browse/NIFI-2551
>
>
> On Wed, Oct 12, 2016 at 6:01 PM, Devin Fisher <
> devin.fis...@perfectsearchcorp.com> wrote:
>
> > I'm having an issue with an in-house processor (not for email inboxes).
> > Only seems to happen periodically on heavy loads, not in dev
> environments.
> > Based on the stack trace the Illegal self-suppression is happening in the
> > framework code and it seems to be masking the original exception that was
> > thrown (maybe?).
> >
> > Going to be continuing to trace this one down but was wondering if anyone
> > has seen something like and what might be the likely cause.
> >
> > version: 0.7.0
> > os: centOS 7
> > java: openjdk version "1.8.0_101"
> >
> >
> >
> > 71088 2016-10-12 15:07:19,503 ERROR [Timer-Driven Process Thread-16]
> > c.imatsolutions.nifi.processor.GetInbox
> > GetInbox[id=cb72b2ca-02da-4508-b619-8884713db8cb]
> > GetInbox[id=cb72b2ca-02da-4508-b619-8884713db8cb] f
> >   ailed to process due to java.lang.IllegalArgumentException:
> > Self-suppression not permitted; rolling back session:
> > java.lang.IllegalArgumentException: Self-suppression not permitted
> > 71089 2016-10-12 15:07:19,505 ERROR [Timer-Driven Process Thread-16]
> > c.imatsolutions.nifi.processor.GetInbox
> > 71090 java.lang.IllegalArgumentException: Self-suppression not permitted
> > 71091 at java.lang.Throwable.addSuppressed(Throwable.java:1043)
> > ~[na:1.8.0_101]
> > 71092 at
> > org.apache.nifi.controller.repository.StandardProcessSession.write(
> > StandardProcessSession.java:1999)
> > ~[na:na]
> > 71093 at com.imatsolutions.nifi.processor.GetInbox.processInboxItem(
> > GetInbox.java:219) [legacy-1.0.0.jar:na]
> > 71094 at com.imatsolutions.nifi.processor.GetInbox.
> processInboxBatch(
> > GetInbox.java:118) [legacy-1.0.0.jar:na]
> > 71095 at com.imatsolutions.nifi.processor.GetInbox.onTrigger(
> > GetInbox.java:76)
> > [legacy-1.0.0.jar:na]
> > 71096 at
> > org.apache.nifi.controller.StandardProcessorNode.onTrigger(
> > StandardProcessorNode.java:1054)
> > [nifi-framework-core-0.7.0.jar:0.7.0]
> > 71097 at
> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(
> > ContinuallyRunProcessorTask.java:136)
> > [nifi-framework-core-0.7.0.jar:0.7.0]
> > 71098 at
> > org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(
> > ContinuallyRunProcessorTask.java:47)
> > [nifi-framework-core-0.7.0.jar:0.7.0]
> > 71099 at
> > org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(
> > TimerDrivenSchedulingAgent.java:127)
> > [nifi-framework-core-0.7.0.jar:0.7.0]
> > 71100 at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > [na:1.8.0_101]
> > 71101 at
> > java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> > [na:1.8.0_101]
> > 71102 at
> > java.util.concurrent.ScheduledThreadPoolExecutor$
> > ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> > [na:1.8.0_101]
> > 71103 at
> > java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.run(
> > ScheduledThreadPoolExecutor.java:294)
> > [na:1.8.0_101]
> > 71104 at
> > java.util.concurrent.ThreadPoolExecutor.runWorker(
> > ThreadPoolExecutor.java:1142)
> > [na:1.8.0_101]
> > 71105 at
> > java.util.concurrent.ThreadPoolExecutor$Worker.run(
> > ThreadPoolExecutor.java:617)
> > [na:1.8.0_101]
> > 71106 at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
> > 71107 Suppressed: java.lang.NullPointerException: null
> > 71108 Caused by: java.lang.NullPointerException: null
> >
>

Error: Self-suppression not permitted

[Devin Fisher]

I'm having an issue with an in-house processor (not for email inboxes).
Only seems to happen periodically on heavy loads, not in dev environments.
Based on the stack trace the Illegal self-suppression is happening in the
framework code and it seems to be masking the original exception that was
thrown (maybe?).

Going to be continuing to trace this one down but was wondering if anyone
has seen something like and what might be the likely cause.

version: 0.7.0
os: centOS 7
java: openjdk version "1.8.0_101"



71088 2016-10-12 15:07:19,503 ERROR [Timer-Driven Process Thread-16]
c.imatsolutions.nifi.processor.GetInbox
GetInbox[id=cb72b2ca-02da-4508-b619-8884713db8cb]
GetInbox[id=cb72b2ca-02da-4508-b619-8884713db8cb] f
  ailed to process due to java.lang.IllegalArgumentException:
Self-suppression not permitted; rolling back session:
java.lang.IllegalArgumentException: Self-suppression not permitted
71089 2016-10-12 15:07:19,505 ERROR [Timer-Driven Process Thread-16]
c.imatsolutions.nifi.processor.GetInbox
71090 java.lang.IllegalArgumentException: Self-suppression not permitted
71091 at java.lang.Throwable.addSuppressed(Throwable.java:1043)
~[na:1.8.0_101]
71092 at
org.apache.nifi.controller.repository.StandardProcessSession.write(StandardProcessSession.java:1999)
~[na:na]
71093 at com.imatsolutions.nifi.processor.GetInbox.processInboxItem(
GetInbox.java:219) [legacy-1.0.0.jar:na]
71094 at com.imatsolutions.nifi.processor.GetInbox.processInboxBatch(
GetInbox.java:118) [legacy-1.0.0.jar:na]
71095 at 
com.imatsolutions.nifi.processor.GetInbox.onTrigger(GetInbox.java:76)
[legacy-1.0.0.jar:na]
71096 at
org.apache.nifi.controller.StandardProcessorNode.onTrigger(StandardProcessorNode.java:1054)
[nifi-framework-core-0.7.0.jar:0.7.0]
71097 at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:136)
[nifi-framework-core-0.7.0.jar:0.7.0]
71098 at
org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call(ContinuallyRunProcessorTask.java:47)
[nifi-framework-core-0.7.0.jar:0.7.0]
71099 at
org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent$1.run(TimerDrivenSchedulingAgent.java:127)
[nifi-framework-core-0.7.0.jar:0.7.0]
71100 at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[na:1.8.0_101]
71101 at
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
[na:1.8.0_101]
71102 at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
[na:1.8.0_101]
71103 at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
[na:1.8.0_101]
71104 at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[na:1.8.0_101]
71105 at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_101]
71106 at java.lang.Thread.run(Thread.java:745) [na:1.8.0_101]
71107 Suppressed: java.lang.NullPointerException: null
71108 Caused by: java.lang.NullPointerException: null

Nifi 1.0.0 breaking changes?

[Devin Fisher]

Based on Version Scheme and API Compatibility[1] the recently released 1.0
(congratulations to all) could have some breaking changes for code
written against 0.* releases.  I was wondering if there are any resources
about what has broken with the new release? I looked at the release notes
but there is nothing there that looked like it would break anything.

[1] :
https://cwiki.apache.org/confluence/display/NIFI/Version+Scheme+and+API+Compatibility

nifi.sensitive.props.key from external source

[Devin Fisher]

We are looking to deploy nifi as part of an appliance. Normally, we use
etckeeper [1] to maintain, track and backup our configuration across the
board. Etckeeper puts configuration files into a git repo. I would like to
store nifi configuration in the same way but I'm concerned about storing
the sensitive properties key there along with the flow.xml.gz. I would like
to store that key somewhere else and load it in at start up time.  Any
thoughts on how that could be done.  Ultimately, I just want is to not back
up the key with the flow.xml.gz (which has the encrypted data). That way if
someone gets a hold of the backup it would not trivial to decrypt the
sensitive data in flow.xml.gz.

I thought I might be able to do this by adding a custom java.arg to the
bootstrap.conf that would point to an environment variable.

Something like:
java.arg.99=-Dnifi.sensitive.props.key=$NIFI_SENSITIVE_PROPS_KEY

But I'm not sure if System properties can stand in for nifi.perperies
values and if the boot loader launches nifi in such a way to use
environment variables.


[1] https://github.com/joeyh/etckeeper

Re: escape functions in Expression Language

[Devin Fisher]

Done. Excited to hear any feedback when someone has time.

Devin

On Tue, Aug 2, 2016 at 9:16 PM, Bryan Bende <bbe...@gmail.com> wrote:

> Hi Devin,
>
> Glad to hear you were able to implement the functionality you were looking
> for. Yes submitting a pull request against the master branch is
> probably the easiest way to submit a contribution. Right now the community
> is working towards closing out tickets for an upcoming 1.0 release, so it
> may take some time before PMCs/committers have time to review, but it is
> still good to get it posted and available.
>
> Thanks,
>
> Bryan
>
> On Tuesday, August 2, 2016, Devin Fisher <
> devin.fis...@perfectsearchcorp.com>
> wrote:
>
> > Forgot to mention that I made a JIRA ticket too.
> >
> > https://issues.apache.org/jira/browse/NIFI-2460
> >
> > Devin
> >
> > On Tue, Aug 2, 2016 at 10:18 AM, Devin Fisher <
> > devin.fis...@perfectsearchcorp.com <javascript:;>> wrote:
> >
> > > I've added the functions following the instructions Bryan point me to
> and
> > > ready to at least have someone else look at it.
> > >
> > > Do I do a pull request to the master or some other branch?
> > >
> > > Devin
> > >
> > > On Fri, Jun 24, 2016 at 11:33 AM, Devin Fisher <
> > > devin.fis...@perfectsearchcorp.com <javascript:;>> wrote:
> > >
> > >> Thanks for point out the README (did not find it in my searching).
> I'll
> > >> have a look at it.
> > >>
> > >> Devin
> > >>
> > >> On Fri, Jun 24, 2016 at 7:24 PM, Bryan Bende <bbe...@gmail.com
> > <javascript:;>> wrote:
> > >>
> > >>> Hi Devin,
> > >>>
> > >>> I'm not aware of a built in escape function in NiFi's EL, although
> you
> > >>> might be able to do some of it with the replace functions.
> > >>>
> > >>> NiFi's expression language is in this module:
> > >>>
> > >>>
> >
> https://github.com/apache/nifi/tree/master/nifi-commons/nifi-expression-language
> > >>>
> > >>> The README has some good details about how to make changes. Let us
> know
> > >>> if
> > >>> it isn't clear.
> > >>>
> > >>> Thanks,
> > >>>
> > >>> Bryan
> > >>>
> > >>>
> > >>> On Fri, Jun 24, 2016 at 1:15 PM, Devin Fisher <
> > >>> devin.fis...@perfectsearchcorp.com <javascript:;>> wrote:
> > >>>
> > >>> > I'm looking to create a bit of JSON using Nifi Expression Language
> so
> > >>> that
> > >>> > I can send it to a web service. But It does not look like Nifi
> > >>> Expression
> > >>> > Language have a function for escaping text to put into JSON.  I've
> > >>> looked
> > >>> > around in the wiki and I don't see any documentation now how to
> > extend
> > >>> > the Expression Language (not even sure if it is possible).
> > >>> >
> > >>> > Looks like nifi makes use of Apache commons-lang which has some
> > >>> functions
> > >>> > that would do the heavy lifting. [1]
> > >>> >
> > >>> > So if someone can help me understand the Expression Language end
> > >>> points I
> > >>> > could do the work fairly easily I think. And add functions for the
> > >>> other
> > >>> > formats supported by StringEscapeUtils (XML, csv, html).
> > >>> >
> > >>> > I think this could be useful.
> > >>> >
> > >>> > Devin
> > >>> >
> > >>> > [1]
> > >>> >
> > >>> >
> > >>>
> >
> https://commons.apache.org/proper/commons-lang/apidocs/org/apache/commons/lang3/StringEscapeUtils.html
> > >>> >
> > >>>
> > >>
> > >>
> > >
> >
>
>
> --
> Sent from Gmail Mobile
>

Re: escape functions in Expression Language

[Devin Fisher]

Forgot to mention that I made a JIRA ticket too.

https://issues.apache.org/jira/browse/NIFI-2460

Devin

On Tue, Aug 2, 2016 at 10:18 AM, Devin Fisher <
devin.fis...@perfectsearchcorp.com> wrote:

> I've added the functions following the instructions Bryan point me to and
> ready to at least have someone else look at it.
>
> Do I do a pull request to the master or some other branch?
>
> Devin
>
> On Fri, Jun 24, 2016 at 11:33 AM, Devin Fisher <
> devin.fis...@perfectsearchcorp.com> wrote:
>
>> Thanks for point out the README (did not find it in my searching). I'll
>> have a look at it.
>>
>> Devin
>>
>> On Fri, Jun 24, 2016 at 7:24 PM, Bryan Bende <bbe...@gmail.com> wrote:
>>
>>> Hi Devin,
>>>
>>> I'm not aware of a built in escape function in NiFi's EL, although you
>>> might be able to do some of it with the replace functions.
>>>
>>> NiFi's expression language is in this module:
>>>
>>> https://github.com/apache/nifi/tree/master/nifi-commons/nifi-expression-language
>>>
>>> The README has some good details about how to make changes. Let us know
>>> if
>>> it isn't clear.
>>>
>>> Thanks,
>>>
>>> Bryan
>>>
>>>
>>> On Fri, Jun 24, 2016 at 1:15 PM, Devin Fisher <
>>> devin.fis...@perfectsearchcorp.com> wrote:
>>>
>>> > I'm looking to create a bit of JSON using Nifi Expression Language so
>>> that
>>> > I can send it to a web service. But It does not look like Nifi
>>> Expression
>>> > Language have a function for escaping text to put into JSON.  I've
>>> looked
>>> > around in the wiki and I don't see any documentation now how to extend
>>> > the Expression Language (not even sure if it is possible).
>>> >
>>> > Looks like nifi makes use of Apache commons-lang which has some
>>> functions
>>> > that would do the heavy lifting. [1]
>>> >
>>> > So if someone can help me understand the Expression Language end
>>> points I
>>> > could do the work fairly easily I think. And add functions for the
>>> other
>>> > formats supported by StringEscapeUtils (XML, csv, html).
>>> >
>>> > I think this could be useful.
>>> >
>>> > Devin
>>> >
>>> > [1]
>>> >
>>> >
>>> https://commons.apache.org/proper/commons-lang/apidocs/org/apache/commons/lang3/StringEscapeUtils.html
>>> >
>>>
>>
>>
>

Re: escape functions in Expression Language

[Devin Fisher]

I've added the functions following the instructions Bryan point me to and
ready to at least have someone else look at it.

Do I do a pull request to the master or some other branch?

Devin

On Fri, Jun 24, 2016 at 11:33 AM, Devin Fisher <
devin.fis...@perfectsearchcorp.com> wrote:

> Thanks for point out the README (did not find it in my searching). I'll
> have a look at it.
>
> Devin
>
> On Fri, Jun 24, 2016 at 7:24 PM, Bryan Bende <bbe...@gmail.com> wrote:
>
>> Hi Devin,
>>
>> I'm not aware of a built in escape function in NiFi's EL, although you
>> might be able to do some of it with the replace functions.
>>
>> NiFi's expression language is in this module:
>>
>> https://github.com/apache/nifi/tree/master/nifi-commons/nifi-expression-language
>>
>> The README has some good details about how to make changes. Let us know if
>> it isn't clear.
>>
>> Thanks,
>>
>> Bryan
>>
>>
>> On Fri, Jun 24, 2016 at 1:15 PM, Devin Fisher <
>> devin.fis...@perfectsearchcorp.com> wrote:
>>
>> > I'm looking to create a bit of JSON using Nifi Expression Language so
>> that
>> > I can send it to a web service. But It does not look like Nifi
>> Expression
>> > Language have a function for escaping text to put into JSON.  I've
>> looked
>> > around in the wiki and I don't see any documentation now how to extend
>> > the Expression Language (not even sure if it is possible).
>> >
>> > Looks like nifi makes use of Apache commons-lang which has some
>> functions
>> > that would do the heavy lifting. [1]
>> >
>> > So if someone can help me understand the Expression Language end points
>> I
>> > could do the work fairly easily I think. And add functions for the other
>> > formats supported by StringEscapeUtils (XML, csv, html).
>> >
>> > I think this could be useful.
>> >
>> > Devin
>> >
>> > [1]
>> >
>> >
>> https://commons.apache.org/proper/commons-lang/apidocs/org/apache/commons/lang3/StringEscapeUtils.html
>> >
>>
>
>

escape functions in Expression Language

[Devin Fisher]

I'm looking to create a bit of JSON using Nifi Expression Language so that
I can send it to a web service. But It does not look like Nifi Expression
Language have a function for escaping text to put into JSON.  I've looked
around in the wiki and I don't see any documentation now how to extend
the Expression Language (not even sure if it is possible).

Looks like nifi makes use of Apache commons-lang which has some functions
that would do the heavy lifting. [1]

So if someone can help me understand the Expression Language end points I
could do the work fairly easily I think. And add functions for the other
formats supported by StringEscapeUtils (XML, csv, html).

I think this could be useful.

Devin

[1]
https://commons.apache.org/proper/commons-lang/apidocs/org/apache/commons/lang3/StringEscapeUtils.html

[GitHub] nifi pull request: NIFI-1560 - Fixing a copy and paste error

[devin-fisher]

GitHub user devin-fisher opened a pull request:

https://github.com/apache/nifi/pull/402

NIFI-1560 - Fixing a copy and paste error

Looks like when the original coder copied code from AuthenticationStrategy 
for the ReferralStrategy and did not change this reference for the error case.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/devin-fisher/nifi NIFI-1560

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/nifi/pull/402.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #402


commit 931c7c32b56268d46b4ae4e0caf0e8920d28e83e
Author: Devin Fisher <dev.fis...@gmail.com>
Date:   2016-05-02T10:26:13Z

Fixing a copy and paste error

Looks like when the original coder copied code from AuthenticationStrategy 
for the ReferralStrategy and did not change this reference for the error case.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

InvokeHTTP is a Final Class

[Devin Fisher]

I was looking at using InvokeHTTP as the base for a class that would
interact with our API.  I wanted to to have the processors make some
assumptions that are true with our API but not HTTP in general (Mostly our
method of Authentication). This class looked well suited for this (the
class is well decomposed with methods that deal with different aspects of
HTTP Protocol). But the InvokeHTTP is a final class which prevents me from
using it this manner.  I have looked over the StandardProcessors and this
is not only one that is a final class. So I was wondering if there is a
reason for this. And if not if it could maybe be changed.

I'd be happy to make the change. I have been meaning to figurate out how
contributions are made in the Nifi community. This might be good chance to
figure that out.

Devin

Re: Rollbacks

[Devin Fisher]

Cool, thanks. I'll take a look at that Processor.  I've learned a lot from
looking at the processor in the main project.

Devin

On Thu, Mar 17, 2016 at 10:46 AM, Joe Witt <joe.w...@gmail.com> wrote:

> Devin,
>
> Good stuff.  Unless the api call to change the content completes
> normally the transformation effectively did not happen.  Take a look
> at CompressContent [1] for an example.
>
> It reads in the original content and streams out new content.  If that
> process of transformation fails it routes the flow file to failure.
> What gets routed is the original representation of the data.
>
> Go ahead and give a really simple form of your case a try.  Do
> something like always throw an exception in your callback to test it
> and then you can validate the behavior meets your needs.  In fact,
> that could be a great test case.
>
> [1]
> https://github.com/apache/nifi/blob/master/nifi-nar-bundles/nifi-standard-bundle/nifi-standard-processors/src/main/java/org/apache/nifi/processors/standard/CompressContent.java
>
> Thanks
> Joe
>
> On Thu, Mar 17, 2016 at 12:36 PM, Devin Fisher
> <devin.fis...@perfectsearchcorp.com> wrote:
> > Thanks for the detailed response. The details about the batching were
> very
> > helpful. I've changed my processors to take advantage of the batching
> that
> > framework provides since my use case fits what you described to a tee.
> >
> > But I have a question about the failure relationship. Maybe a concrete
> > example might help. So let's say I have an XML document that I'm
> processing
> > with a Processor that converts XML to JSON. This Processor makes use an
> SAX
> > parser so it doesn't have to read the whole XML document into memory. The
> > SAX parser read from an input stream provided by Nifi and calls SAX
> > callbacks base on the XML it encounters. These callbacks write the JSON
> to
> > an output stream provided by Nifi. So as soon as the Processor starts it
> > has written to the current Session. If towards the end of the XML
> document
> > it hits a syntax error, let say there is a text element with an '&' that
> > has not been encoded as an entity. The SAX parser will fail and will not
> be
> > able to complete successfully. (the parser is very strict) But the
> > processor will have already written most of the resulting JSON but not
> all.
> > At this point if I transfer it the failed relationship without rolling
> back
> > what will I get? Will I get the incomplete JSON or will I get the invalid
> > but complete XML? I'm guessing that it will be half written JSON. What I
> > would think you would want is the complete XML so that a processor that
> > fixes the '&' entity issue could handle the issue and route it back. Or a
> > different processor could log it or something else.
> >
> > Anyway, I hope that example explains the confusion that I still have
> about
> > handling error cases. Rolling back here would not be helpful from what I
> > understand because it will just get processed again by the same Processor
> > with the SAX parser that can't handle the '&' over and over again. The
> > penalty will make sure that other documents get process before it is
> tried
> > again but that one malformed XML document will never progress because it
> > will always fail in the same way each time and be rollback again.
> >
> > Again thanks for the reply. The info was really useful and helped my
> > understanding.
> >
> > Devin
> >
> > On Tue, Mar 15, 2016 at 7:27 PM, Joe Witt <joe.w...@gmail.com> wrote:
> >
> >> Devin,
> >>
> >> So the session follows the unit of work pattern that Martin Fowler
> >> describes [1].  The idea here is that whether you access one flow file
> >> or many flow files or anything in between you are doing some session
> >> of work.  You can commit all the things you do on that session or
> >> rollback all the things you do on that session.
> >>
> >> This pattern/concept provides a nice and safe environment in which the
> >> contract between the developer and framework is well understood.
> >> Generally, rollback is only necessary when some unplanned or
> >> unexpected exception occurs and it is largely there so that the
> >> framework can ensure all things are returned to a safe state.  It can
> >> also do things like penalize that processor/extension so that if it is
> >> some programming error that it will reduce its impact on the system
> >> overall.
> >>
> >> So, with that said there are two ways to think about your case.  It

Re: Rollbacks

[Devin Fisher]

.  For the scenario of failures that you can predict such as
> invalid data or invalid state of something you actually want to have a
> failure relationship on that processor and simply route things there.
> From a 'developer' perspective this is not a rollback case.  "Failure"
> then is as planned for and expected as "success".  So you go ahead and
> route the flowfile to failure and call commit.  All good.  It is the
> person designing this loosely coupled and highly cohesive set of
> components together in a flow that gets to decide what failure means
> for their context.
>
> Lots of info here and probably not well written or with big gaps.
> You're asking the right questions so just keep asking.  Lots of folks
> here that want to help.
>
> [1] http://martinfowler.com/eaaCatalog/unitOfWork.html
> [2]
> https://nifi.apache.org/docs/nifi-docs/html/developer-guide.html#session-rollback
>
> Thanks
> Joe
>
> On Tue, Mar 15, 2016 at 7:25 PM, Devin Fisher
> <devin.fis...@perfectsearchcorp.com> wrote:
> > Thanks for your reply. I'm sorry if my question seems confusing. I'm
> still
> > learning how nifi works. I don't have any understand about how the
> > framework works on the back end and incomplete understanding of the
> exposed
> > interface. From my point view (an external process developer) asking to
> > rollback the one flow file that failed (I don't want changes made to it
> > incompletely) and lets the other n flowfiles move on seems reasonable.
> But
> > I don't know what is happening in the session on the back end.
> >
> > I likely don't really understand what happens on a rollback. Reading the
> > developer's guide I got the impression that rollback disregards all
> changes
> > made the session include transfers. It then returns the flowfiles to the
> > queue. It would seem that a session is really finished and not usable
> after
> > a rollback. So, I then don't understand how I can do my use case. I want
> to
> > rollback (undo changes to a single flow file that failed) and then
> transfer
> > it to the Failed relationship unchanged (or add the discard.reason to the
> > attributes).
> >
> > I assume you mean "Run duration" when you refer to the 'scheduling' tab.
> I
> > would love to understand better how that works. In the documentation, I
> > only see a note about it in the User guide.  But the developer's guide is
> > silent. I don't see how that slider is enforced in the processor code. It
> > seems that once the framework has ceded control to the processor it can
> run
> > for as long as it wants. So more information about this would be great.
> >
> > Thanks again for the response. The information is always useful and
> > enlighting.
> > Devin
> >
> > On Tue, Mar 15, 2016 at 4:26 PM, Andrew Grande <agra...@hortonworks.com>
> > wrote:
> >
> >> Devin,
> >>
> >> What you're asking for is a contradicting requirement. One trades
> >> individual message transactional control (and necessary overhead) for
> the
> >> higher throughput with micro-batching (but lesser control). In short,
> you
> >> can't expect to rollback a message and not affect the whole batch.
> >>
> >> However, if you 'commit' this batch as received by your processor, and
> >> take on the responsibility of storing, tracking and commit/rollback of
> >> those yourself for downstream connection But then, why?
> >>
> >> In general, one should leverage NiFi 'Scheduling' tab and have the
> >> micro-batching aspect controlled via the framework. Unless you really
> >> really have a very good reason to do it yourself.
> >>
> >> Hope this helps,
> >> Andrew
> >>
> >>
> >>
> >>
> >> On 3/7/16, 5:00 PM, "Devin Fisher" <devin.fis...@perfectsearchcorp.com>
> >> wrote:
> >>
> >> >Question about rollbacks. I have a processor that is grabbing a list of
> >> >FlowFiles from session.get(100). It will then process each flow file
> one
> >> at
> >> >a time.  I want to then be able if there is an error with a single
> >> FlowFile
> >> >to roll it back (and only this failed FlowFile) and transfer it to the
> >> >FAILED relationship. But reading the javadoc for ProcessSession I don't
> >> get
> >> >the sense that I can do that.
> >> >
> >> >Is my workflow wrong, should I only get one at a time from the session
> and
> >> >commit after each one?
> >> >
> >> >Devin
> >>
>

Re: Cross NAR Controller Services

[Devin Fisher]

Thanks Bryan. That should give me plenty to work on tomorrow. I'll write
back if I can't figure it out.

Devin

On Tue, Mar 15, 2016 at 6:11 PM, Bryan Bende <bbe...@gmail.com> wrote:

> Devin,
>
> This WIki page shows how to create the appropriate dependencies between
> your NAR and the ControllerService:
>
>
> https://cwiki.apache.org/confluence/display/NIFI/Maven+Projects+for+Extensions#MavenProjectsforExtensions-LinkingProcessorsandControllerServices
>
> I also created an example project on GitHub to show a working example:
>   https://github.com/bbende/nifi-dependency-example
>
> Hope that helps.
>
> -Bryan
>
> On Tue, Mar 15, 2016 at 7:33 PM, Oleg Zhurakousky <
> ozhurakou...@hortonworks.com> wrote:
>
> > Devin
> >
> > Your problem is most likely in your NAR poms where you may satisfy
> compile
> > dependency but not NAR to participate in class loader runtime
> inheritance.
> > Is there a way to look at your poms and also the general structure of the
> > project?
> >
> > Oleg
> >
> > Sent from my iPhone
> >
> > > On Mar 15, 2016, at 18:51, Devin Fisher <
> > devin.fis...@perfectsearchcorp.com> wrote:
> > >
> > > I'm having issues using a standard controller service
> > (DBCPConnectionPool)
> > > that is provided by nifi-dbcp-service-nar. But I'm having issues with
> my
> > > nar. I have included a dependency on nifi-dbcp-service-api in my maven
> > pom
> > > and have used the property description that is the same as ExecuteSQL
> for
> > > the DBCP_SERVICE property. When I load my processor in nifi I don't
> get a
> > > list of DBCPConnectionPool controller service like I expect. I have an
> > > ExecuteSQL processor in the same flow (for testing) and it list the
> > > controller service I created just fine and uses it just fine.
> > >
> > > The problem seems to me (I don't have a development environment to
> > confirm)
> > > that the DBCPService.class that I use in my processor is not seen as
> the
> > > same class object (because of the isolation features of NAR) as the one
> > > that DBCPCOnnectionPool implements. I think I have mostly confirmed
> this
> > by
> > > implementing a dummy controller service that implements DBCPService in
> > the
> > > same NAR as my processor and my processor is able to list it just fine.
> > But
> > > the ExecuteSQL don't list my dummy controller service. So they seem to
> be
> > > considered different classes.
> > >
> > > I think I'm doing something wrong because ExecuteSQL is not in the same
> > nar
> > > as DBCPConnectionPool. So they play nice together somehow but I don't
> see
> > > what I need to do so that my nar works the same way.
> > >
> > > I'm enjoying developing against nifi and sorry if this is a rookie
> > mistake.
> > >
> > > Devin
> >
>

Re: Rollbacks

[Devin Fisher]

Thanks for your reply. I'm sorry if my question seems confusing. I'm still
learning how nifi works. I don't have any understand about how the
framework works on the back end and incomplete understanding of the exposed
interface. From my point view (an external process developer) asking to
rollback the one flow file that failed (I don't want changes made to it
incompletely) and lets the other n flowfiles move on seems reasonable. But
I don't know what is happening in the session on the back end.

I likely don't really understand what happens on a rollback. Reading the
developer's guide I got the impression that rollback disregards all changes
made the session include transfers. It then returns the flowfiles to the
queue. It would seem that a session is really finished and not usable after
a rollback. So, I then don't understand how I can do my use case. I want to
rollback (undo changes to a single flow file that failed) and then transfer
it to the Failed relationship unchanged (or add the discard.reason to the
attributes).

I assume you mean "Run duration" when you refer to the 'scheduling' tab. I
would love to understand better how that works. In the documentation, I
only see a note about it in the User guide.  But the developer's guide is
silent. I don't see how that slider is enforced in the processor code. It
seems that once the framework has ceded control to the processor it can run
for as long as it wants. So more information about this would be great.

Thanks again for the response. The information is always useful and
enlighting.
Devin

On Tue, Mar 15, 2016 at 4:26 PM, Andrew Grande <agra...@hortonworks.com>
wrote:

> Devin,
>
> What you're asking for is a contradicting requirement. One trades
> individual message transactional control (and necessary overhead) for the
> higher throughput with micro-batching (but lesser control). In short, you
> can't expect to rollback a message and not affect the whole batch.
>
> However, if you 'commit' this batch as received by your processor, and
> take on the responsibility of storing, tracking and commit/rollback of
> those yourself for downstream connection But then, why?
>
> In general, one should leverage NiFi 'Scheduling' tab and have the
> micro-batching aspect controlled via the framework. Unless you really
> really have a very good reason to do it yourself.
>
> Hope this helps,
> Andrew
>
>
>
>
> On 3/7/16, 5:00 PM, "Devin Fisher" <devin.fis...@perfectsearchcorp.com>
> wrote:
>
> >Question about rollbacks. I have a processor that is grabbing a list of
> >FlowFiles from session.get(100). It will then process each flow file one
> at
> >a time.  I want to then be able if there is an error with a single
> FlowFile
> >to roll it back (and only this failed FlowFile) and transfer it to the
> >FAILED relationship. But reading the javadoc for ProcessSession I don't
> get
> >the sense that I can do that.
> >
> >Is my workflow wrong, should I only get one at a time from the session and
> >commit after each one?
> >
> >Devin
>

Cross NAR Controller Services

[Devin Fisher]

I'm having issues using a standard controller service (DBCPConnectionPool)
that is provided by nifi-dbcp-service-nar. But I'm having issues with my
nar. I have included a dependency on nifi-dbcp-service-api in my maven pom
and have used the property description that is the same as ExecuteSQL for
the DBCP_SERVICE property. When I load my processor in nifi I don't get a
list of DBCPConnectionPool controller service like I expect. I have an
ExecuteSQL processor in the same flow (for testing) and it list the
controller service I created just fine and uses it just fine.

The problem seems to me (I don't have a development environment to confirm)
that the DBCPService.class that I use in my processor is not seen as the
same class object (because of the isolation features of NAR) as the one
that DBCPCOnnectionPool implements. I think I have mostly confirmed this by
implementing a dummy controller service that implements DBCPService in the
same NAR as my processor and my processor is able to list it just fine. But
the ExecuteSQL don't list my dummy controller service. So they seem to be
considered different classes.

I think I'm doing something wrong because ExecuteSQL is not in the same nar
as DBCPConnectionPool. So they play nice together somehow but I don't see
what I need to do so that my nar works the same way.

I'm enjoying developing against nifi and sorry if this is a rookie mistake.

Devin

Re: Split Content (One-to-Many) early commit

[Devin Fisher]

Thanks for the response. It gave my additional context such that I can
proceed with more confidence. I've been playing with creating additional
sessions and I believe that it will work for my use case. But I will keep
the duplicate flowfile potential issue in mind, though.

Thanks again.

Devin

On Sun, Mar 13, 2016 at 12:40 PM, Mark Payne <marka...@hotmail.com> wrote:

> Devin,
>
> We do realize that we have some work to do in order to make it so that
> a single Processor can buffer up hundreds of thousands or more FlowFiles.
> The SplitText processor is very popular and suffers from this exact same
> problem.
> We want to have a mechanism for swapping those out of the Java Heap,
> similar
> to how we do when we have millions of FlowFiles sitting in a queue. There
> is a ticket
> here [1] to address this. However, this has turned out to be very time
> consuming, and
> not quite a straight-forward as we had hoped, so it's not been finished up
> yet.
>
> In the meantime, you can use the approach that you described, using two
> different
> Process Sessions, by extending AbstractSessionFactoryProcessor instead of
> AbstractProcessor. The downside to this approach, though, is that when
> NiFi is restarted,
> you could potentially have a lot of data duplication.
>
> As an example, let's imagine that you create a ProcessSession and use it
> to create 10,000 FlowFiles
> and then commit the session and create a new one. If you have an incoming
> FlowFiles that has
> 1 million rows in it, you may create 800,000 FlowFiles and send them out
> and then NiFi gets restarted.
> In this case, you will pick up the original FlowFile and begin processing
> it again. But you've already sent
> out those 800,000 FlowFiles. Depending on your requirements, this may or
> may not be acceptable.
>
> One option that you could use is just to document that this behavior
> exists and that SplitText should be
> used ahead of you Processor in order to split the content into 10,000 line
> chunks. This would avoid the
> heap exhaustion.
>
> Another possible solution that you could use, though it's not as pretty as
> I'd like: Process up to 10,000 FlowFiles
> from an input FlowFile. Then, add an attribute to the input FlowFile
> indicating your progress (for instance,
> add an attribute named "rows.converted" and then do
> "session.transfer(flowFile);" This will transfer the FlowFile
> back into its input queue. You can then commit the session. Then, when you
> call session.get() to get an input
> FlowFile again, you can check for that attribute and skip that many rows.
> This way, you won't end up with
> data duplication. The downside here is that you would end up reading the
> first N rows each time and ignoring
> the content which can be expensive. A more optimized approach would be to
> wrap the InputStream in
> a ByteCountingInputStream and record the number of bytes consumed and use
> that as an attribute, and then
> for each subsequent iteration use StreamUtils.skip() to skip the
> appropriate number of bytes.
>
> I know there's a lot of info here. Let me know if anything doesn't make
> sense.
>
> I hope this helps!
> -Mark
>
>
> [1] https://issues.apache.org/jira/browse/NIFI-1008 <
> https://issues.apache.org/jira/browse/NIFI-1008>
>
>
> > On Mar 11, 2016, at 5:29 PM, Devin Fisher <
> devin.fis...@perfectsearchcorp.com> wrote:
> >
> > I'm creating a processor that will read a customer csv and will create a
> > new flowfile for each line in the form of XML. The CSV file will be quite
> > large (100s of thousands of lines). I would like to commit a reasonable
> > amount from time to time so that they can flow down to other processors.
> > But looking at similar processors SplitText and SplitXml they save up all
> > the created flowfiles and release them all at the end.  In some trials,
> I'm
> > running out of memory doing that. But I can't commit the session early
> > because I'm still reading the original CSV file.  Is there a workflow
> where
> > I can read the incoming CSV flowfile but still release created flowfiles?
> > I'm thinking of not using AbstractProcessor and instead
> > use AbstractSessionFactoryProcessor and create two different sessions but
> > is that advisable or possible?
> >
> > Devin
>
>

Rollbacks

[Devin Fisher]

Question about rollbacks. I have a processor that is grabbing a list of
FlowFiles from session.get(100). It will then process each flow file one at
a time.  I want to then be able if there is an error with a single FlowFile
to roll it back (and only this failed FlowFile) and transfer it to the
FAILED relationship. But reading the javadoc for ProcessSession I don't get
the sense that I can do that.

Is my workflow wrong, should I only get one at a time from the session and
commit after each one?

Devin

Re: Logging issue in Jira

[Devin Fisher]

Thanks, that got me what I needed. Not super familiar with Jira. The Issue
has been logged (NIFI-1560 <https://issues.apache.org/jira/browse/NIFI-1560>
).
Devin

On Mon, Feb 22, 2016 at 1:08 PM, Matt Gilman <matt.c.gil...@gmail.com>
wrote:

> Devin,
>
> Thanks for reporting and showing interest in NiFi!
>
> There should be a button in the toolbar at the top that says Create. I
> believe this button is only visible if you are logged in. Have you created
> a JIRA account?
>
> Matt
>
> On Mon, Feb 22, 2016 at 3:02 PM, Devin Fisher <
> devin.fis...@perfectsearchcorp.com> wrote:
>
> > I noticed a small copy and past error in LdapProvider class. (the error
> > message is not using the right variable) I tried to create a ticket in
> Jira
> > but don't see how to do so. Look through the Contributor Guide and did
> not
> > see info on how to log a bug. What is the best approach to log an issue?
> >
> > Devin
> >
>

Logging issue in Jira

[Devin Fisher]

I noticed a small copy and past error in LdapProvider class. (the error
message is not using the right variable) I tried to create a ticket in Jira
but don't see how to do so. Look through the Contributor Guide and did not
see info on how to log a bug. What is the best approach to log an issue?

Devin

Javadoc

[Devin Fisher]

Might be a silly question but I can't find a link the Javadocs anywhere on
your website. I've googled for it and only find the reference to the
Javadocs in the NiFi Developer’s Guide
. I'm hopeful that I'm just
not seeing it. If not, what is the recommended way to get a hold of it?

Also, thanks in advanced.

Devin

Re: Testing Custom Validators

[Devin Fisher]

Thanks, I'm guessing that subject is not to0 important to the validation
workflow but provides useful info for the user in the context of the
property that they are setting.

The validator I'm writing is not using the context so it should be fine. If
I need to do more I guess I'll figure out how the mock the rest then.

Devin

On Fri, Dec 18, 2015 at 1:16 PM, Mark Payne <marka...@hotmail.com> wrote:

> Devin,
>
> Mockito should be find for most cases. If you validator itself is calling
> into the ValidationContext, for instance to create a new PropertyValue
> object, then you may need to mock out some methods. Otherwise, it should
> be fine.
>
> When validate() is called, it is given the 'input' (which is the value to
> valid) and the 'subject' (which is a description of what is being
> validated).
> So generally, the 'subject' is the name of the Property that is being
> validated.
>
> Thanks
> -Mark
>
> > On Dec 18, 2015, at 3:11 PM, Devin Fisher <
> devin.fis...@perfectsearchcorp.com> wrote:
> >
> > I'm trying to create some tests for some Validators that I'm creating.
> But
> > I can't figure out an easy way to create MockValidationContext. I don't
> > want to create the whole environment that I would for Processors.
> >
> > I looked
> > at
> nifi/nifi-commons/nifi-processor-utilities/src/test/java/org/apache/nifi/processor/util/TestStandardValidators.java
> > and it looks that it just uses Mockito to mock it.
> >
> > So, is that the recommended way to do it. Will that allow me to control
> the
> > ValidationContext?
> >
> > Also, not sure if asking two questions in one email is allowed, what is
> the
> > subject parameter in the validate method?
> >
> > Devin
>
>