Hi Asmath, In addition to excellent answers from Nathan and Jorge, I just want to add that NiFi should be seen as a framework enabling you to build efficient flows. But just like any framework and you can do pretty much anything you want, it can be something good but it can also be something bad. It's really hard to provide generic answers about debugging flows/processors given the very wide set of components NiFi is providing.
I recommend anyone using NiFi to look at the 4 anti-patterns videos made by Mark as it gives awesome insights: Anti-Patterns in NiFi - Part 1: https://www.youtube.com/watch?v=RjWstt7nRVY Anti-Patterns in NiFi - Part 2 : Flow layout: https://www.youtube.com/watch?v=v1CoQk730qs Anti-Patterns in NiFi - Part 3 : Load balanced connections: https://www.youtube.com/watch?v=by9P0Zi8Dk8 Anti-Patterns in NiFi - Part 4 : Scheduling: https://www.youtube.com/watch?v=pZq0EbfDBy4 Pierre Le lun. 8 févr. 2021 à 11:04, Jorge Machado <[email protected]> a écrit : > Another thing to look would be to check if you are creating two much flow > files as this hammers the disks. If you see very high memory usage it could > be you are having to much data in attributes and not in the content of the > flow file. > > On 8. Feb 2021, at 07:07, [email protected] wrote: > > Hi Asmath, > > In terms of performance tuning, I’ve found a lot that needs to be > considered. > > I’ve done tuning it’s mainly been caused large queues building due to the > read-write queues on the content repo disks being very high. I’ve also had > previous threads on the JOLT default transformation, which should be > avoided at all costs as it’s not performant. > > If you are confident the performance bottleneck isn’t the hardware in its > cluster itself, I would look at of tuning threads on the flow. I can > recommend looking at Mark Payne’s NiFi Notes Videos on Youtube: > https://www.youtube.com/channel/UCcNsK56MZRXzPbRfe8sODWw > > Bear in mind, that you may need to go round the loop of tuning the > hardware and the flow to meet your requirements. > > Kind Regards, > > Nathan > *From:* KhajaAsmath Mohammed [mailto:[email protected] > <[email protected]>] > *Sent:* 08 February 2021 08:22 > *To:* [email protected] > *Subject:* NIFI - Performance issues > > Hi, > > I am looking for some information on how to improve performance of our > flows. Any suggestions? > > 1. how to troubleshoot which processor in the nifi is causing an issue? > 2. What is the best way to increase the threads or tasks for a particular > processor? > 3. How to debug the flow when there is an issue on a particular processor > group? > > Thanks, > Asmath > > >
