Hi Ali, How many FlowFile’s are you expecting per minute? My experience with NiFi has been that if you have 1 FlowFile that is several GB’s in size you can frequently process it faster than 100k FlowFile’s at a fraction the size. There is a lot of overhead in managing the life cycle of a FlowFile when you start to have lots of them.
--Peter From: Ali Nazemian [mailto:[email protected]] Sent: Wednesday, August 23, 2017 2:49 PM To: [email protected] Subject: [EXT] High-performance Nifi decoder Hi all, I am investigating the right approach for implementing an high-performance network packet decoder in Nifi. I am trying to achieve about 1 Gbps throughput on a specific decoder. (with 4 Nifi instances) I was wondering which option would perform better. 1) Implementing my decoder in a separate Nifi processor using NAR mechanism; 2) Implementing my decoder in C++ and use "ExecuteStreamCommand" to run that decoder. Basically, if ExecuteStreamCommand involves Disk for the sake of sending messages to the written script, it doesn't matter how fast the decoder is, disk would be the main bottleneck. Regards, Ali
