Arnaud, Did you have any WARN or ERROR messages in the logs? I'm particular interested in anything that mentions the word "Swap" or "swap" (i.e., regardless of case). Is it possible that the FlowFile Repository could have run out of disk space?
Thanks -Mark On May 16, 2017, at 3:34 AM, Arnaud G <[email protected]<mailto:[email protected]>> wrote: Hi Matt, Thanks for your reply! I finally solved the problem by deleting all the content in the flowfile directory, but here are my observations: 1) The problem was coming from one of the cluster node, when this node was out of the cluster, the queue were reporting 0 flowfile. 2) The first time I restarted this node, about 20'000 flowfile reappeared and were treated, every time I subsequently restarted this node about 20-30k flowfiles were again treated (I was only specifically monitoring one queue, but it happened for multiple other queues) 3) After 3-4 reboots of this node the queue reported 90K elements and remained in this state despite multiple other restart. 4) The flowfile directory on this node contained 200 MB of data 5) I tried to setup the flowfile expiration but it didn't do anything to the queue status 6) I tried to change the backpressure threshold without any effect. 7) During the problem the queue was operating normally on the cluster, and flowfiles were flowing through it without any issue. Arnaud On Mon, May 15, 2017 at 10:39 PM, Matt Gilman <[email protected]<mailto:[email protected]>> wrote: Sorry for the delayed response. Similar behavior has been reported by some other users [1]. Does the connection have any back pressure threshold configured? Can new flowfiles be enqueued? Do the expiration settings have any affect? Lastly, if you restart the cluster does it claim the connection still has flowfiles enqueued? Matt [1] https://issues.apache.org/jira/browse/NIFI-3897 On Fri, May 12, 2017 at 5:47 AM, Arnaud G <[email protected]<mailto:[email protected]>> wrote: Hi again! I currently have another issue with incoherent queue status. Following the upgrade to 1.2 of a cluster, I have a couple of queues that display through the GUI a high number of flowfiles. As the queue were no emptying despite tuning, I tried to list the content of the queue. This action returns that the queue contains no flowfile, which is not the expected as the GUI displays another value. If I try to empty the queue, I receive a message: 0 FlowFiles (0 bytes) out of 210'000 (92.71MB) were removed from the queue. And of course I cannot delete the queue as this action reports me that the queue is not empty. So somehow it seems that the queue are empty but that the current display of the queue don't reflect it (it is very likely that some data were lost during the upgrade procedure as we had to reboot a few node to change the heap property) What will be the best method to restore a proper state and be able to edit the flow file again? Thank you! Arnaud
