[
https://issues.apache.org/jira/browse/NIFI-10226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17566755#comment-17566755
]
Josef Zahner commented on NIFI-10226:
-------------------------------------
Hi [~joewitt] ,
I did some research and it looks like during that period, I don't know why, the
emptying of the queue was abnormally slow. I can see that based on the
following log messages:
{code:java}
2022-07-13 12:08:39,987 INFO [Drop FlowFiles for Connection
592c3c09-4b56-1155-ffff-ffffaaae5907] o.a.n.c.r.WriteAheadFlowFileRepository
Repository updated to reflect that 10000 FlowFiles were swapped in to
FlowFileQueue[id=592c3c09-4b56-1155-ffff-ffffaaae5907, Load Balance
Strategy=DO_NOT_LOAD_BALANCE, size=QueueSize[FlowFiles=1536640, ContentSize=0
Bytes]] {code}
The time between the next 10'000 drop message was sometimes at about 3s. Please
check my splunk screenshot below. For the first flowfiles it was fast, but then
the time between the next message dropped to 3s. So it took multiple minutes to
drop all 1'500'000 flowfiles.
!image-2022-07-14-12-23-15-980.png!
After a bit more than 1minute, we saw the first socket timeouts:
!image-2022-07-14-12-28-58-329.png!
To sum it up, please ensure that a slow dropping doesn't cause
SocketTimeoutExceptions and please extend the GUI to not just show the 0% and
100% as today. The log shows 10'000 of flowfiles dropping, why not the GUI? We
had no idea when the dropping will be finished...
Hope this helps
> Empty Queue 1'500'000 flowfiles - NiFi GUI HTTP Error 503
> ---------------------------------------------------------
>
> Key: NIFI-10226
> URL: https://issues.apache.org/jira/browse/NIFI-10226
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core UI
> Affects Versions: 1.15.3
> Environment: 8-Node cluster
> Reporter: Josef Zahner
> Priority: Major
> Attachments: Screenshot 2022-07-13 at 13.46.23.png,
> image-2022-07-14-12-23-15-980.png, image-2022-07-14-12-28-58-329.png
>
>
> We have a ListSFTP processor which produces 1'500'000 flowfiles as output.
> When we try to "Empty Queue" it takes multiple minutes and the GUI shows a
> HTTP 503 error during that period. We can't open the GUI again until the
> deletion is complete. Those flowfiles are all on the primary node and not
> load balanced within the cluster. Below is the final GUI message. The
> intermediate message looks similar but without the removed string.
> !Screenshot 2022-07-13 at 13.46.23.png!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
