Another thing is you want to check the actual disk usage. If it's above a threshold limit defined by your NiFi Properties the system will stall seemingly randomly. See this thread for more details: https://lists.apache.org/thread.html/r70dcc6387dccd062cbae09cfa6470c8d7443db63a495be35450e2b00%40%3Cusers.nifi.apache.org%3E
-Eric On Sun, Feb 7, 2021 at 11:57 PM Pierre Villard <[email protected]> wrote: > Hi Asmath, > > In addition to excellent answers from Nathan and Jorge, I just want to add > that NiFi should be seen as a framework enabling you to build efficient > flows. But just like any framework and you can do pretty much anything you > want, it can be something good but it can also be something bad. It's > really hard to provide generic answers about debugging flows/processors > given the very wide set of components NiFi is providing. > > I recommend anyone using NiFi to look at the 4 anti-patterns videos made > by Mark as it gives awesome insights: > Anti-Patterns in NiFi - Part 1: > https://www.youtube.com/watch?v=RjWstt7nRVY > Anti-Patterns in NiFi - Part 2 : Flow layout: > https://www.youtube.com/watch?v=v1CoQk730qs > Anti-Patterns in NiFi - Part 3 : Load balanced connections: > https://www.youtube.com/watch?v=by9P0Zi8Dk8 > Anti-Patterns in NiFi - Part 4 : Scheduling: > https://www.youtube.com/watch?v=pZq0EbfDBy4 > > Pierre > > Le lun. 8 févr. 2021 à 11:04, Jorge Machado <[email protected]> a écrit : > >> Another thing to look would be to check if you are creating two much flow >> files as this hammers the disks. If you see very high memory usage it could >> be you are having to much data in attributes and not in the content of the >> flow file. >> >> On 8. Feb 2021, at 07:07, [email protected] wrote: >> >> Hi Asmath, >> >> In terms of performance tuning, I’ve found a lot that needs to be >> considered. >> >> I’ve done tuning it’s mainly been caused large queues building due to the >> read-write queues on the content repo disks being very high. I’ve also had >> previous threads on the JOLT default transformation, which should be >> avoided at all costs as it’s not performant. >> >> If you are confident the performance bottleneck isn’t the hardware in its >> cluster itself, I would look at of tuning threads on the flow. I can >> recommend looking at Mark Payne’s NiFi Notes Videos on Youtube: >> https://www.youtube.com/channel/UCcNsK56MZRXzPbRfe8sODWw >> >> Bear in mind, that you may need to go round the loop of tuning the >> hardware and the flow to meet your requirements. >> >> Kind Regards, >> >> Nathan >> *From:* KhajaAsmath Mohammed [mailto:[email protected] >> <[email protected]>] >> *Sent:* 08 February 2021 08:22 >> *To:* [email protected] >> *Subject:* NIFI - Performance issues >> >> Hi, >> >> I am looking for some information on how to improve performance of our >> flows. Any suggestions? >> >> 1. how to troubleshoot which processor in the nifi is causing an issue? >> 2. What is the best way to increase the threads or tasks for a particular >> processor? >> 3. How to debug the flow when there is an issue on a particular processor >> group? >> >> Thanks, >> Asmath >> >> >>
