Re: NIFI connecting to Activemq

[Christopher Hamm]

Thanks for the update. Do you have suggestion what I should set my
processor to?
On Jan 9, 2016 8:25 PM, "Joe Witt" <joe.w...@gmail.com> wrote:

> Chris,
>
> Thanks for sending the screenshot.  The NullPointerException is
> trivially reproduced using your settings.  I didn't even need a JMS
> server to cause it.
>
> Have created a JIRA for this:
> https://issues.apache.org/jira/browse/NIFI-1378
>
> Thanks!
> Joe
>
>
>
> On Sat, Jan 9, 2016 at 3:21 PM, Oleg Zhurakousky
> <ozhurakou...@hortonworks.com> wrote:
> > Well, you URL is malformed, since it should be of form
> protocol://host:port.
> > Just look at any AMQ sample out there on the web. The part that worries
> me
> > is the NPE and nothing in the logs. Did you try Joe's suggestion and
> enable
> > Debug level logging?
> >
> > Sent from my iPhone
> >
> > On Jan 9, 2016, at 14:06, Christopher Hamm <em...@christopherhamm.com>
> > wrote:
> >
> > Those logs came from a completely new restart with cleared logs.
> >
> > If have an ActiveMQ topic setup. What should each field have in it if it
> is
> > running locally and using port 61613? I think maybe the NPE is related to
> > bad network connection with my activeMQ. Can't find examples out there.
> > 
> >
> > On Sat, Jan 9, 2016 at 9:13 AM, Oleg Zhurakousky
> > <ozhurakou...@hortonworks.com> wrote:
> >>
> >> Chris
> >>
> >> Are you sure you are providing the correct logs? I can’t see a single
> >> mention of JMS nor any stack traces which would definitely be there
> given
> >> what you see in UI.
> >> Also, the fact that you see NPE is definitely a bug that we have to fix
> >> (user’s should never see NPE), so that can be filed. What I am trying to
> >> figure out is the condition that triggers it. May be if you shut down
> NIFI,
> >> delete all the logs and restart so you can get a fresh data. . .
> >>
> >> Cheers
> >> Oleg
> >>
> >> On Jan 9, 2016, at 1:42 AM, Christopher Hamm <em...@christopherhamm.com
> >
> >> wrote:
> >>
> >> Here are the logs
> >>
> >> On Wed, Jan 6, 2016 at 11:32 PM, Joe Witt <joe.w...@gmail.com> wrote:
> >>>
> >>> Chris,
> >>>
> >>> Ok.  Can you take a look in the /logs/nifi-app.log and see if
> there
> >>> is a stack trace for the NullPointerException included?
> >>>
> >>> If not please add the following to your logback.xml and after 30
> seconds
> >>> or so it should start giving you the stack traces.  A stacktrace for
> the NPE
> >>> would be really useful in pinpointing the likely issue.  Also please
> share
> >>> the config details of GetJMSQueue processor that you can.
> >>>
> >>>  >>> level="DEBUG"/>
> >>>
> >>> Thanks
> >>> Joe
> >>>
> >>> On Wed, Jan 6, 2016 at 11:16 PM, Christopher Hamm
> >>> <em...@christopherhamm.com> wrote:
> >>>>
> >>>> Couldnt copy the text but here it is.
> >>>> 
> >>>>
> >>>>
> >>>>
> >>>> On Wed, Jan 6, 2016 at 5:28 PM, Joe Witt <joe.w...@gmail.com> wrote:
> >>>>>
> >>>>> Christopher,
> >>>>>
> >>>>> Is there any error/feedback showing up in the UI or in the logs?
> >>>>>
> >>>>> Thanks
> >>>>> Joe
> >>>>>
> >>>>> On Wed, Jan 6, 2016 at 5:19 PM, Christopher Hamm
> >>>>> <em...@christopherhamm.com> wrote:
> >>>>> > What am I doing wrong with hooking up my activemq jms get
> template? I
> >>>>> > put
> >>>>> > stuff into the activeMQ and NIFI wont get it. Using 0.4.1.
> >>>>> >
> >>>>> > --
> >>>>> > Sincerely,
> >>>>> > Chris Hamm
> >>>>> > (E) ceham...@gmail.com
> >>>>> > (Twitter) http://twitter.com/webhamm
> >>>>> > (Linkedin) http://www.linkedin.com/in/chrishamm
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> --
> >>>> Sincerely,
> >>>> Chris Hamm
> >>>> (E) ceham...@gmail.com
> >>>> (Twitter) http://twitter.com/webhamm
> >>>> (Linkedin) http://www.linkedin.com/in/chrishamm
> >>>
> >>>
> >>
> >>
> >>
> >> --
> >> Sincerely,
> >> Chris Hamm
> >> (E) ceham...@gmail.com
> >> (Twitter) http://twitter.com/webhamm
> >> (Linkedin) http://www.linkedin.com/in/chrishamm
> >> 
> >>
> >>
> >
> >
> >
> > --
> > Sincerely,
> > Chris Hamm
> > (E) ceham...@gmail.com
> > (Twitter) http://twitter.com/webhamm
> > (Linkedin) http://www.linkedin.com/in/chrishamm
>

NIFI connecting to Activemq

[Christopher Hamm]

What am I doing wrong with hooking up my activemq jms get template? I put
stuff into the activeMQ and NIFI wont get it. Using 0.4.1.

-- 
Sincerely,
Chris Hamm
(E) ceham...@gmail.com
(Twitter) http://twitter.com/webhamm
(Linkedin) http://www.linkedin.com/in/chrishamm
 jms template56f8e76e-e731-41c1-8f29-277cfbbc09c051cb06fa-c473-4fc3-befa-0ec229c5ff870 MB0622.2763671875194.97129821777344622.2763671875244.9712982177734451cb06fa-c473-4fc3-befa-0ec229c5ff87071c2d7c-875e-4a1f-ad49-f874b5ff5605PROCESSOR0 sec1activity.restoredinactivesuccess51cb06fa-c473-4fc3-befa-0ec229c5ff87071c2d7c-875e-4a1f-ad49-f874b5ff5605PROCESSOR0b8564aac-b081-4349-a94d-4cfbff43538051cb06fa-c473-4fc3-befa-0ec229c5ff870 MB551cb06fa-c473-4fc3-befa-0ec229c5ff87820558a7-95d9-47ec-a2aa-b637cabf3c97PROCESSOR5 sec1success51cb06fa-c473-4fc3-befa-0ec229c5ff87f7dd9023-171a-483f-861f-eb4db68e03d4PROCESSOR0b922b9da-f1f3-423d-86f3-03f4977f516351cb06fa-c473-4fc3-befa-0ec229c5ff870 MB10051cb06fa-c473-4fc3-befa-0ec229c5ff87071c2d7c-875e-4a1f-ad49-f874b5ff5605PROCESSOR5 sec1success51cb06fa-c473-4fc3-befa-0ec229c5ff87820558a7-95d9-47ec-a2aa-b637cabf3c97PROCESSOR0f7dd9023-171a-483f-861f-eb4db68e03d451cb06fa-c473-4fc3-befa-0ec229c5ff87279.19517517089844-153.80633544921875WARN1TIMER_DRIVEN1EVENT_DRIVEN0CRON_DRIVEN1TIMER_DRIVEN0 secCRON_DRIVEN* * * * * ?JMS ProviderActiveMQActiveMQActiveMQThe Provider used for the JMS ServerJMS ProviderfalseJMS ProvidertruefalsefalseURLThe URL of the JMS ServerURLfalseURLtruefalsefalseDestination NameThe name of the JMS Topic or queue to useDestination NamefalseDestination NametruefalsefalseCommunications Timeout30 secThe amount of time to wait when attempting to receive a message before giving up and assuming failureCommunications TimeoutfalseCommunications TimeouttruefalsefalseMessage Batch Size10The number of messages to pull/push in a single iteration of the processorMessage Batch SizefalseMessage Batch SizetruefalsefalseUsernameUsername used for authentication and authorizationUsernamefalseUsernamefalsefalsefalsePasswordPassword used for authentication and authorizationPasswordfalsePasswordfalsetruefalseSSL Context ServiceThe Controller Service to use in order to obtain an SSL Context.SSL Context Servicefalseorg.apache.nifi.ssl.SSLContextServiceSSL Context ServicefalsefalsefalseAcknowledgement ModeClient AcknowledgeClient AcknowledgeAuto AcknowledgeAuto AcknowledgeClient AcknowledgeThe JMS Acknowledgement Mode. Using Auto Acknowledge can cause messages to be lost on restart of NiFi but may provide better performance than Client Acknowledge.Acknowledgement ModefalseAcknowledgement ModetruefalsefalseMessage SelectorThe JMS Message Selector to use in order to narrow the messages that are pulledMessage SelectorfalseMessage SelectorfalsefalsefalseCopy JMS Properties to AttributestruetruefalsefalsetrueWhether or not the JMS Message Properties should be copied to the FlowFile Attributes; if so, the attribute name will be jms.XXX, where XXX is the JMS Property nameCopy JMS Properties to AttributesfalseCopy JMS Properties to AttributestruefalsefalseClient ID PrefixA human-readable ID that can be used to associate connections with yourself so that the maintainers of the JMS Server know who to contact if problems ariseClient ID PrefixfalseClient ID Prefixfalsefalsefalsefalse30 secJMS ProviderURLlocalhostDestination NamefaaCommunications TimeoutMessage Batch SizeUsernamePasswordSSL Context ServiceAcknowledgement ModeMessage SelectorCopy JMS Properties to AttributesClient ID Prefix00 secTIMER_DRIVEN1 secGetJMSQueuefalseAll FlowFiles are routed to successsuccessSTOPPEDfalsetrueorg.apache.nifi.processors.standard.GetJMSQueue820558a7-95d9-47ec-a2aa-b637cabf3c9751cb06fa-c473-4fc3-befa-0ec229c5ff87789.7225952148438-5.361641883850098WARN1TIMER_DRIVEN1EVENT_DRIVEN0CRON_DRIVEN1TIMER_DRIVEN0 secCRON_DRIVEN* * * * * ?Attributes ListComma separated list of attributes to be included in the resulting JSON. If this value is left empty then all existing Attributes will be included. This list of attributes is case sensitive. If an attribute specified in the list is not found it will be be emitted to the resulting JSON with an empty string or NULL value.Attributes ListfalseAttributes ListfalsefalsefalseDestinationflowfile-attributeflowfile-attributeflowfile-contentflowfile-contentflowfile-attributeControl if JSON value is written as a new flowfile attribute 'JSONAttributes' or written in the flowfile content. Writing to flowfile content will overwrite any existing flowfile content.DestinationfalseDestinationtruefalsefalseInclude Core AttributestruetruefalsefalsetrueDetermines if the FlowFile org.apache.nifi.flowfile.attributes.CoreAttributes which are contained in every FlowFile should be included in the final JSON value generated.Include Core AttributesfalseInclude Core AttributestruefalsefalseNull ValuetruetruefalsefalsefalseIf true a non existing or empty attribute will be NULL in the resulting JSON. If false an empty string will be placed in the 

Re: Upgrade 0.3.0 to 0.4.1

[Christopher Hamm]

when does brew update?
On Jan 4, 2016 9:08 PM, "Joe Witt"  wrote:

> Hello Chakri,
>
> Take a look here
>   https://cwiki.apache.org/confluence/display/NIFI/Upgrading+NiFi
>
> And here
>   https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance
>
> Thanks
> Joe
>
> On Mon, Jan 4, 2016 at 8:02 PM, Chakrader Dewaragatla
>  wrote:
> > Hi – What is the best way to upgrade from 0.3.0 to 0.4.1  ? We are using
> > binaries to deploy nifi.
> >
> > Thanks,
> > -Chakri
> > 
> > The information contained in this transmission may contain privileged and
> > confidential information. It is intended only for the use of the
> person(s)
> > named above. If you are not the intended recipient, you are hereby
> notified
> > that any review, dissemination, distribution or duplication of this
> > communication is strictly prohibited. If you are not the intended
> recipient,
> > please contact the sender by reply email and destroy all copies of the
> > original message.
> > 
>

Re: Upgrade 0.3.0 to 0.4.1

[Christopher Hamm]

brew update and brew doctor is my friend.
On Jan 4, 2016 10:11 PM, "Aldrin Piri" <aldrinp...@gmail.com> wrote:

> This occurs as shortly after the release is official and the formula
> update gets incorporated into the main homebrew repository by their group
> of committers.
>
> 0.4.1 is available in the main repo, currently.
>
> Some older versions are also available in the homebrew-versions tap.
> These will likely be scoped to the latest patch/bugfix version for a minor
> release as per homebrew's guidelines.  e.g. - 0.3.1 will be available but
> 0.3.0 will not.
>
> On Mon, Jan 4, 2016 at 9:53 PM, Christopher Hamm <ceham...@gmail.com>
> wrote:
>
>> when does brew update?
>> On Jan 4, 2016 9:08 PM, "Joe Witt" <joe.w...@gmail.com> wrote:
>>
>>> Hello Chakri,
>>>
>>> Take a look here
>>>   https://cwiki.apache.org/confluence/display/NIFI/Upgrading+NiFi
>>>
>>> And here
>>>   https://cwiki.apache.org/confluence/display/NIFI/Migration+Guidance
>>>
>>> Thanks
>>> Joe
>>>
>>> On Mon, Jan 4, 2016 at 8:02 PM, Chakrader Dewaragatla
>>> <chakrader.dewaraga...@lifelock.com> wrote:
>>> > Hi – What is the best way to upgrade from 0.3.0 to 0.4.1  ? We are
>>> using
>>> > binaries to deploy nifi.
>>> >
>>> > Thanks,
>>> > -Chakri
>>> > 
>>> > The information contained in this transmission may contain privileged
>>> and
>>> > confidential information. It is intended only for the use of the
>>> person(s)
>>> > named above. If you are not the intended recipient, you are hereby
>>> notified
>>> > that any review, dissemination, distribution or duplication of this
>>> > communication is strictly prohibited. If you are not the intended
>>> recipient,
>>> > please contact the sender by reply email and destroy all copies of the
>>> > original message.
>>> > 
>>>
>>
>

NIFI template work needed

[Christopher Hamm]

I finally got requirements nailed down for my project which means I can now
start building stuff out.

  Honestly, all I need to find is a template that fits the needs of the
project that works well in NIFI. If I had a boilerplate template with the
following criteria, I could easily form it with much more details.

What I am basically looking for:

1) Initial input will come from outside JMS (*java messaging system) based
technology.
2) Parsing of message from JMS to extract text and output clean textTake
clean final values
3) create new or update (*if main id exists already) field record values of
a MySql DB

For more exact template I am happy to pay someone to do the work, I just
need to know the rate.

Many thanks

If you would like to talk more let me know your availability.

Looking to pay for template setup to spec

[Christopher Hamm]

Not sure if against the rules but
..
Is there anyone who is willing to setup and send me the exact template I
need to move forward with some work for compensation?

I would provide the exact specifications and would keep revisions to
minimal. I would mainly use template as a jumping off point.

I don't have full specifications yet due to awaiting information, but
should have what I need soon.

Anyone interested can contact me via em...@christopherhamm.com

Re: Queue wasn't emptying

[Christopher Hamm]

I will look, but how do I just kill it with fire. What can I delete?
On Nov 12, 2015 7:42 AM, "Mark Payne" <marka...@hotmail.com> wrote:

> Chris / Elli,
>
> What version of NiFi are you running? I am wondering if you may have been
> bitten by
> this bug [1]. This should affect only the 'master' branch, not any
> formally released
> versions of NiFi. If you built the application from source code, rather
> than using an
> already-released version, this may well be what is happening.
>
> Thanks
> -Mark
>
>
> [1] https://issues.apache.org/jira/browse/NIFI-1155
>
>
> On Nov 12, 2015, at 2:32 AM, Christopher Hamm <em...@christopherhamm.com>
> wrote:
>
> where does it dump? I cant get mine to expire fast enough. It is dev test
> I want to blow away but cant delete.
>
> On Fri, Nov 6, 2015 at 11:33 AM, Joe Witt <joe.w...@gmail.com> wrote:
>
>> Elli,
>>
>> Can you share your flow configuration that we could possibly use to
>> replicate this?  Perhaps turn this into a JIRA and attach that.
>>
>> Also anytime you see something that appears 'stuck' please try to get
>> a stack dump (bin/nifi.sh dump)  As if there is truly a stuck thread
>> we'll see it there and what it is blocked on.
>>
>> Thanks
>> Joe
>>
>> On Fri, Nov 6, 2015 at 11:25 AM, Elli Schwarz <eliezer_schw...@yahoo.com>
>> wrote:
>> > I had a queue that built up overnight with several thousand flowfiles.
>> The
>> > queue was pointing to a RouteOnAttribute processor, which was running.
>> For
>> > some reason, the RouteOnAttribute processor wasn't emptying the queue
>> so the
>> > queue just built up. Stopping and starting the processor didn't help.
>> > However, simply restarting Nifi got the flow moving again. I can't
>> figure
>> > out why the queue was stuck.
>> >
>> > What could cause a queue to build up like that? How would restarting
>> Nifi
>> > get it going again? There weren't any errors in the log.  I did have
>> > backpressure on the relationship to the processor after the
>> RouteOnAttribute
>> > set to 1, but that queue was empty. There was no backpressure on the
>> queue
>> > heading in to the RouteOnAttribute processor. The RouteOnAttribute
>> processor
>> > has a Penalty Duration and a Yield of 1 sec. The relationships all used
>> the
>> > PriorityAttributeAnalyzer as the prioritizer.
>> >
>> > The interesting thing is I had the exact same problem on two different
>> nifis
>> > (one was version 0.2.0 and one 0.3.0). The same type of flow files were
>> > going from one to the other using site-to-site (I was doing this for
>> load
>> > balancing). On the other Nifi, instead of restarting, I added another
>> > RouteOnAttribute processor and rerouted the stuck queue to the new one
>> > instead, and that also got things moving again. (I have a backpressure
>> since
>> > the processor after the RouteOnAttribute is a ControlRate processor,
>> and I
>> > only want to allow 1 flowfile through per second so as not to overwhelm
>> the
>> > system downstream. The backpressure is to assist with load balancing,
>> so if
>> > one pathway fills up, I have the other pathway which uses the
>> site-to-site
>> > to route the flow to the other nifi).
>> >
>> > Is there some combination of backpressure, penalty or yield along with
>> > prioritization that could cause a kind of deadlock-like situation? Any
>> ideas
>> > as to how I can prevent this from occurring?
>> >
>> > Thanks!
>> > -Elli
>> >
>>
>
>
>
> --
> Sincerely,
> Chris Hamm
> (E) ceham...@gmail.com
> (Twitter) http://twitter.com/webhamm
> (Linkedin) http://www.linkedin.com/in/chrishamm
>
>
>

Re: Queue wasn't emptying

[Christopher Hamm]

that isn't clearing them out really. it gradually decreases but doesn't
clear out 500k items.
On Nov 12, 2015 8:22 AM, "Juan Sequeiros" <juan.sequei...@onyxpoint.com>
wrote:

> To answer Christopher,
>
> You can clear out flowfiles on a queue by setting expiration on
> relationship line.
>
> On Thu, Nov 12, 2015 at 7:53 AM, Christopher Hamm <ceham...@gmail.com>
> wrote:
>
>> I will look, but how do I just kill it with fire. What can I delete?
>> On Nov 12, 2015 7:42 AM, "Mark Payne" <marka...@hotmail.com> wrote:
>>
>>> Chris / Elli,
>>>
>>> What version of NiFi are you running? I am wondering if you may have
>>> been bitten by
>>> this bug [1]. This should affect only the 'master' branch, not any
>>> formally released
>>> versions of NiFi. If you built the application from source code, rather
>>> than using an
>>> already-released version, this may well be what is happening.
>>>
>>> Thanks
>>> -Mark
>>>
>>>
>>> [1] https://issues.apache.org/jira/browse/NIFI-1155
>>>
>>>
>>> On Nov 12, 2015, at 2:32 AM, Christopher Hamm <em...@christopherhamm.com>
>>> wrote:
>>>
>>> where does it dump? I cant get mine to expire fast enough. It is dev
>>> test I want to blow away but cant delete.
>>>
>>> On Fri, Nov 6, 2015 at 11:33 AM, Joe Witt <joe.w...@gmail.com> wrote:
>>>
>>>> Elli,
>>>>
>>>> Can you share your flow configuration that we could possibly use to
>>>> replicate this?  Perhaps turn this into a JIRA and attach that.
>>>>
>>>> Also anytime you see something that appears 'stuck' please try to get
>>>> a stack dump (bin/nifi.sh dump)  As if there is truly a stuck thread
>>>> we'll see it there and what it is blocked on.
>>>>
>>>> Thanks
>>>> Joe
>>>>
>>>> On Fri, Nov 6, 2015 at 11:25 AM, Elli Schwarz <
>>>> eliezer_schw...@yahoo.com> wrote:
>>>> > I had a queue that built up overnight with several thousand
>>>> flowfiles. The
>>>> > queue was pointing to a RouteOnAttribute processor, which was
>>>> running. For
>>>> > some reason, the RouteOnAttribute processor wasn't emptying the queue
>>>> so the
>>>> > queue just built up. Stopping and starting the processor didn't help.
>>>> > However, simply restarting Nifi got the flow moving again. I can't
>>>> figure
>>>> > out why the queue was stuck.
>>>> >
>>>> > What could cause a queue to build up like that? How would restarting
>>>> Nifi
>>>> > get it going again? There weren't any errors in the log.  I did have
>>>> > backpressure on the relationship to the processor after the
>>>> RouteOnAttribute
>>>> > set to 1, but that queue was empty. There was no backpressure on the
>>>> queue
>>>> > heading in to the RouteOnAttribute processor. The RouteOnAttribute
>>>> processor
>>>> > has a Penalty Duration and a Yield of 1 sec. The relationships all
>>>> used the
>>>> > PriorityAttributeAnalyzer as the prioritizer.
>>>> >
>>>> > The interesting thing is I had the exact same problem on two
>>>> different nifis
>>>> > (one was version 0.2.0 and one 0.3.0). The same type of flow files
>>>> were
>>>> > going from one to the other using site-to-site (I was doing this for
>>>> load
>>>> > balancing). On the other Nifi, instead of restarting, I added another
>>>> > RouteOnAttribute processor and rerouted the stuck queue to the new one
>>>> > instead, and that also got things moving again. (I have a
>>>> backpressure since
>>>> > the processor after the RouteOnAttribute is a ControlRate processor,
>>>> and I
>>>> > only want to allow 1 flowfile through per second so as not to
>>>> overwhelm the
>>>> > system downstream. The backpressure is to assist with load balancing,
>>>> so if
>>>> > one pathway fills up, I have the other pathway which uses the
>>>> site-to-site
>>>> > to route the flow to the other nifi).
>>>> >
>>>> > Is there some combination of backpressure, penalty or yield along with
>>>> > prioritization that could cause a kind of deadlock-like situation?
>>>> Any ideas
>>>> > as to how I can prevent this from occurring?
>>>> >
>>>> > Thanks!
>>>> > -Elli
>>>> >
>>>>
>>>
>>>
>>>
>>> --
>>> Sincerely,
>>> Chris Hamm
>>> (E) ceham...@gmail.com
>>> (Twitter) http://twitter.com/webhamm
>>> (Linkedin) http://www.linkedin.com/in/chrishamm
>>>
>>>
>>>
>

Re: Queue wasn't emptying

[Christopher Hamm]

yes and thank you
On Nov 12, 2015 8:30 AM, "Mark Payne" <marka...@hotmail.com> wrote:

> Chris,
>
> If this is a test system (which I believe you said it is) and you're okay
> losing all data in the system, you can shutdown
> and delete your content and flowfile repositories (by default they are
> located at
> ./flowfile_repository and ./content_repository) and then restart.
>
> Thanks
> -Mark
>
>
> On Nov 12, 2015, at 8:25 AM, Christopher Hamm <ceham...@gmail.com> wrote:
>
> that isn't clearing them out really. it gradually decreases but doesn't
> clear out 500k items.
> On Nov 12, 2015 8:22 AM, "Juan Sequeiros" <juan.sequei...@onyxpoint.com>
> wrote:
>
>> To answer Christopher,
>>
>> You can clear out flowfiles on a queue by setting expiration on
>> relationship line.
>>
>> On Thu, Nov 12, 2015 at 7:53 AM, Christopher Hamm <ceham...@gmail.com>
>> wrote:
>>
>>> I will look, but how do I just kill it with fire. What can I delete?
>>> On Nov 12, 2015 7:42 AM, "Mark Payne" <marka...@hotmail.com> wrote:
>>>
>>>> Chris / Elli,
>>>>
>>>> What version of NiFi are you running? I am wondering if you may have
>>>> been bitten by
>>>> this bug [1]. This should affect only the 'master' branch, not any
>>>> formally released
>>>> versions of NiFi. If you built the application from source code, rather
>>>> than using an
>>>> already-released version, this may well be what is happening.
>>>>
>>>> Thanks
>>>> -Mark
>>>>
>>>>
>>>> [1] https://issues.apache.org/jira/browse/NIFI-1155
>>>>
>>>>
>>>> On Nov 12, 2015, at 2:32 AM, Christopher Hamm <
>>>> em...@christopherhamm.com> wrote:
>>>>
>>>> where does it dump? I cant get mine to expire fast enough. It is dev
>>>> test I want to blow away but cant delete.
>>>>
>>>> On Fri, Nov 6, 2015 at 11:33 AM, Joe Witt <joe.w...@gmail.com> wrote:
>>>>
>>>>> Elli,
>>>>>
>>>>> Can you share your flow configuration that we could possibly use to
>>>>> replicate this?  Perhaps turn this into a JIRA and attach that.
>>>>>
>>>>> Also anytime you see something that appears 'stuck' please try to get
>>>>> a stack dump (bin/nifi.sh dump)  As if there is truly a stuck thread
>>>>> we'll see it there and what it is blocked on.
>>>>>
>>>>> Thanks
>>>>> Joe
>>>>>
>>>>> On Fri, Nov 6, 2015 at 11:25 AM, Elli Schwarz <
>>>>> eliezer_schw...@yahoo.com> wrote:
>>>>> > I had a queue that built up overnight with several thousand
>>>>> flowfiles. The
>>>>> > queue was pointing to a RouteOnAttribute processor, which was
>>>>> running. For
>>>>> > some reason, the RouteOnAttribute processor wasn't emptying the
>>>>> queue so the
>>>>> > queue just built up. Stopping and starting the processor didn't help.
>>>>> > However, simply restarting Nifi got the flow moving again. I can't
>>>>> figure
>>>>> > out why the queue was stuck.
>>>>> >
>>>>> > What could cause a queue to build up like that? How would restarting
>>>>> Nifi
>>>>> > get it going again? There weren't any errors in the log.  I did have
>>>>> > backpressure on the relationship to the processor after the
>>>>> RouteOnAttribute
>>>>> > set to 1, but that queue was empty. There was no backpressure on the
>>>>> queue
>>>>> > heading in to the RouteOnAttribute processor. The RouteOnAttribute
>>>>> processor
>>>>> > has a Penalty Duration and a Yield of 1 sec. The relationships all
>>>>> used the
>>>>> > PriorityAttributeAnalyzer as the prioritizer.
>>>>> >
>>>>> > The interesting thing is I had the exact same problem on two
>>>>> different nifis
>>>>> > (one was version 0.2.0 and one 0.3.0). The same type of flow files
>>>>> were
>>>>> > going from one to the other using site-to-site (I was doing this for
>>>>> load
>>>>> > balancing). On the other Nifi, instead of restarting, I added another
>>>>> > RouteOnAttribute processor and rerouted the stuck queue to the new
>>>>> one
>>>>> > instead, and that also got things moving again. (I have a
>>>>> backpressure since
>>>>> > the processor after the RouteOnAttribute is a ControlRate processor,
>>>>> and I
>>>>> > only want to allow 1 flowfile through per second so as not to
>>>>> overwhelm the
>>>>> > system downstream. The backpressure is to assist with load
>>>>> balancing, so if
>>>>> > one pathway fills up, I have the other pathway which uses the
>>>>> site-to-site
>>>>> > to route the flow to the other nifi).
>>>>> >
>>>>> > Is there some combination of backpressure, penalty or yield along
>>>>> with
>>>>> > prioritization that could cause a kind of deadlock-like situation?
>>>>> Any ideas
>>>>> > as to how I can prevent this from occurring?
>>>>> >
>>>>> > Thanks!
>>>>> > -Elli
>>>>> >
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Sincerely,
>>>> Chris Hamm
>>>> (E) ceham...@gmail.com
>>>> (Twitter) http://twitter.com/webhamm
>>>> (Linkedin) http://www.linkedin.com/in/chrishamm
>>>>
>>>>
>>>>
>>
>

template repo for learning

[Christopher Hamm]

Is there a repo of nifi templates with advanced features that use lots of
expression language expecially when used to make requests? I can't find
enough docs or youtube videos that really dig into it.

Re: template repo for learning

[Christopher Hamm]

Thanks Bryan. I will take a look. I might have a few questions to resolve
errors from the UI once I try again.

On Tue, Nov 3, 2015 at 10:14 AM, Bryan Bende <bbe...@gmail.com> wrote:

> If you can share a little more info about what the API that you're trying
> to interact with looks like, we can likely provide more concrete guidance.
>
> As a very basic test, to familiarize yourself with the Expression
> Language, you could create a "dummy flow" such as:
> - GenerateFlowFile to create a new FlowFile on a timer
> - UpdateAttribute to set attributes you want to be passed to your API, you
> can use expression language here to create dynamic date/time values
> - InvokeHttp to call your API
> - You could then route the "Response" relationship from InvokeHttp to some
> other processor to possibly extract information from the response for
> further use
>
> Let us know if we can help more.
>
> On Tue, Nov 3, 2015 at 9:51 AM, Christopher Hamm <ceham...@gmail.com>
> wrote:
>
>> I am trying to query api based on date/time and possibly based on results
>> from fields of another query.
>> On Nov 3, 2015 9:41 AM, "Bryan Bende" <bbe...@gmail.com> wrote:
>>
>>> Christopher,
>>>
>>> In terms of templates, the best resources we have right now are:
>>>
>>> https://cwiki.apache.org/confluence/display/NIFI/Example+Dataflow+Templates
>>> https://github.com/xmlking/nifi-examples
>>>
>>> For expression language we have the EL guide:
>>>
>>> https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html
>>>
>>> Is there a specific flow you are trying to tackle?
>>>
>>> -Bryan
>>>
>>>
>>> On Tue, Nov 3, 2015 at 9:36 AM, Christopher Hamm <
>>> em...@christopherhamm.com> wrote:
>>>
>>>> Is there a repo of nifi templates with advanced features that use lots
>>>> of expression language expecially when used to make requests? I can't find
>>>> enough docs or youtube videos that really dig into it.
>>>>
>>>
>>>
>


-- 
Sincerely,
Chris Hamm
(E) ceham...@gmail.com
(Twitter) http://twitter.com/webhamm
(Linkedin) http://www.linkedin.com/in/chrishamm