Joe, 1.8 million FlowFiles is not a concern. But when you say “Should I reduce the queue sizes?” it makes me wonder if they’re all in a single queue? Generally, you should leave the backpressure threshold at the default 10,000 FlowFile max. Increasing this can lead to huge amounts of swapping, which will drastically reduce performance and increase disk utilization very significantly.
Also from the diagnostics, it looks like you’ve got a lot of CPU cores, but you’re not using much. And based on the amount of disk space available and the fact that you’re seeing 100% utilization, I’m wondering if you’re using spinning disks, rather than SSDs? I would highly recommend always running NiFi with ssd/nvme drives. Absent that, if you have multiple disk drives, you could also configure the content repository to span multiple disks, in order to spread that load. Thanks -Mark On Mar 22, 2023, at 10:41 AM, Joe Obernberger <[email protected]> wrote: Thank you. Was able to get in. Currently there are 1.8 million flow files and 3.2G. Is this too much for a 3 node cluster with mutliple spindles each (SATA drives)? Should I reduce the queue sizes? -Joe On 3/22/2023 10:23 AM, Phillip Lord wrote: Joe, If you need the UI to come back up, try setting the autoresume setting in nifi.properties to false and restart node(s). This will bring up every component/controllerService up stopped/disabled and may provide some breathing room for the UI to become available again. Phil On Mar 22, 2023 at 10:20 AM -0400, Joe Obernberger <[email protected]><mailto:[email protected]>, wrote: atop shows the disk as being all red with IO - 100% utilization. There are a lot of flowfiles currently trying to run through, but I can't monitor it because....UI wont' load. -Joe On 3/22/2023 10:16 AM, Mark Payne wrote: Joe, I’d recommend taking a look at garbage collection. It is far more likely the culprit than disk I/O. Thanks -Mark On Mar 22, 2023, at 10:12 AM, Joe Obernberger <[email protected]><mailto:[email protected]> wrote: I'm getting "java.net.SocketTimeoutException: timeout" from the user interface of NiFi when load is heavy. This is 1.18.0 running on a 3 node cluster. Disk IO is high and when that happens, I can't get into the UI to stop any of the processors. Any ideas? I have put the flowfile repository and content repository on different disks on the 3 nodes, but disk usage is still so high that I can't get in. Thank you! -Joe -- This email has been checked for viruses by AVG antivirus software. www.avg.com<http://www.avg.com/> [https://s-install.avcdn.net/ipm/preview/icons/icon-envelope-tick-green-avg-v1.png]<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> Virus-free.www.avg.com<http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> <x-msg://111/#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2>
