Elli, What you may be seeing could be a different thing. In 0.4.0 we've improved who large queue handling works significantly.
Thanks Joe On Thu, Nov 12, 2015 at 8:56 AM, Elli Schwarz <eliezer_schw...@yahoo.com> wrote: > I had the issue with 0.2.0 and 0.3.0, and I did not build from source. It > happens on rare occasions, and I can't figure out what causes it. I do have > backpressure but no expiration on the route that RouteOnAttribute feeds. I > haven't had a chance to try to dump. > > It seems to happen after a long period of time of the queue being full and > backpressure being applied due to a large amount of flowfiles being added > constantly. > > > > On Thursday, November 12, 2015 7:42 AM, Mark Payne <marka...@hotmail.com> > wrote: > > > > Chris / Elli, > > What version of NiFi are you running? I am wondering if you may have been > bitten by > this bug [1]. This should affect only the 'master' branch, not any formally > released > versions of NiFi. If you built the application from source code, rather than > using an > already-released version, this may well be what is happening. > > Thanks > -Mark > > > [1] https://issues.apache.org/jira/browse/NIFI-1155 > > > On Nov 12, 2015, at 2:32 AM, Christopher Hamm <em...@christopherhamm.com> > wrote: > > where does it dump? I cant get mine to expire fast enough. It is dev test I > want to blow away but cant delete. > > On Fri, Nov 6, 2015 at 11:33 AM, Joe Witt <joe.w...@gmail.com> wrote: > > Elli, > > Can you share your flow configuration that we could possibly use to > replicate this? Perhaps turn this into a JIRA and attach that. > > Also anytime you see something that appears 'stuck' please try to get > a stack dump (bin/nifi.sh dump) As if there is truly a stuck thread > we'll see it there and what it is blocked on. > > Thanks > Joe > > On Fri, Nov 6, 2015 at 11:25 AM, Elli Schwarz <eliezer_schw...@yahoo.com> > wrote: >> I had a queue that built up overnight with several thousand flowfiles. The >> queue was pointing to a RouteOnAttribute processor, which was running. For >> some reason, the RouteOnAttribute processor wasn't emptying the queue so >> the >> queue just built up. Stopping and starting the processor didn't help. >> However, simply restarting Nifi got the flow moving again. I can't figure >> out why the queue was stuck. >> >> What could cause a queue to build up like that? How would restarting Nifi >> get it going again? There weren't any errors in the log. I did have >> backpressure on the relationship to the processor after the >> RouteOnAttribute >> set to 1, but that queue was empty. There was no backpressure on the queue >> heading in to the RouteOnAttribute processor. The RouteOnAttribute >> processor >> has a Penalty Duration and a Yield of 1 sec. The relationships all used >> the >> PriorityAttributeAnalyzer as the prioritizer. >> >> The interesting thing is I had the exact same problem on two different >> nifis >> (one was version 0.2.0 and one 0.3.0). The same type of flow files were >> going from one to the other using site-to-site (I was doing this for load >> balancing). On the other Nifi, instead of restarting, I added another >> RouteOnAttribute processor and rerouted the stuck queue to the new one >> instead, and that also got things moving again. (I have a backpressure >> since >> the processor after the RouteOnAttribute is a ControlRate processor, and I >> only want to allow 1 flowfile through per second so as not to overwhelm >> the >> system downstream. The backpressure is to assist with load balancing, so >> if >> one pathway fills up, I have the other pathway which uses the site-to-site >> to route the flow to the other nifi). >> >> Is there some combination of backpressure, penalty or yield along with >> prioritization that could cause a kind of deadlock-like situation? Any >> ideas >> as to how I can prevent this from occurring? >> >> Thanks! >> -Elli >> > > > > > -- > Sincerely, > Chris Hamm > (E) ceham...@gmail.com > (Twitter) http://twitter.com/webhamm > (Linkedin) http://www.linkedin.com/in/chrishamm > > > >
