Can you show what is happening inside the first process group? Is there a
SplitText processor with line count of 1?

On Fri, Aug 30, 2019 at 4:21 AM Jeremy Pemberton-Pigott <
fuzzych...@gmail.com> wrote:

> Hi Pierre,
>
> I'm using Nifi version 1.6.0.
>
> 04/03/2018 08:16:22 UTC
>
> Tagged nifi-1.6.0-RC3
>
> From 7c0ee01 on branch NIFI-4995-RC3
> FlowFile expiration = 0
> Back pressure object threshold = 20000
> Back pressure data size threshold = 1GB
>
> The connection is just from the output port of 1 PG to the input port of
> another PG.  Inside the PG all the connections are using the same settings
> between processors.
>
> Regards,
>
> Jeremy
>
> On Fri, Aug 30, 2019 at 4:14 PM Pierre Villard <
> pierre.villard...@gmail.com> wrote:
>
>> Hi Jeremy,
>>
>> It seems very weird that you get 200M flow files in a relationship that
>> should have backpressure set at 20k flow files. While backpressure is not a
>> hard limit you should not get to such numbers. Can you give us more
>> details? What version of NiFi are you using? What's the configuration of
>> your relationship between your two process groups?
>>
>> Thanks,
>> Pierre
>>
>> Le ven. 30 août 2019 à 07:46, Jeremy Pemberton-Pigott <
>> fuzzych...@gmail.com> a écrit :
>>
>>> Hi,
>>>
>>> I have a 3 node Nifi 1.6.0 cluster.  It ran out of disk space when there
>>> was a log jam of flow files (from slow HBase lookups).  My queue is
>>> configured for 20,000 but 1 node has over 206 million flow files stuck in
>>> the queue.  I managed to clear up some disk space to get things going again
>>> but it seems that after a few mins of processing all the processors in the
>>> Log Parser process group will stop processing and show zero in/out.
>>>
>>> Is this a bug fixed in a later version?
>>>
>>> Each time I have to tear down the Docker containers running Nifi and
>>> restart it to process a few 10,000s and repeat every few mins.  Any idea
>>> what I should do to keep it processing the data (nifi-app.log doesn't show
>>> my anything unusual about the stop or delay) until the 1 node can clear the
>>> backlog?
>>>
>>> [image: image.png]
>>>
>>> Regards,
>>>
>>> Jeremy
>>>
>> --
Sent from Gmail Mobile

Reply via email to