Hello guys, Ever since I tried Apache Nifi, I was amazed with its powerful features and perfomance. Every day, I would like to understand and learn more of Nifi and I try to improve my knowledge of by reading articles and following on the forums. The applications of nifi have simplied a lot of work as ordinary user and developer [ I have yet to explore the advance use cases, and i will soon]
Nifi is doing a fantastic job and I am wondering if there anything which I can do to improve its performance from my current settings. I have thousands and thousands [ more than many 20million+] files which need ETL and I am wondering what are the two cents of valuable advice would advice me on how i can improve the performance [ meaning transform more files per day/ hour]. The files size ranges from 1MB to 60MB. My set up is very simple. Everything reside in the same location [ location]. However, i need to know if there is anything i can do to improve the performance. I am sure there one or two twiks I can do achieve the best performance. Please kindly do share some valuable tips for me. <code> openjdk version "1.8.0_131" OpenJDK Runtime Environment (build 1.8.0_131-8u131-b11-0ubuntu1.16.04.2-b11) OpenJDK 64-Bit Server VM (build 25.131-b11, mixed mode) Nifi 1.3 Server RAM 128 GB </code> File open limits.. /etc/security/limits.conf <code> * hard nofile 500000 * soft nofile 500000 root hard nofile 500000 root soft nofile 500000 session required pam_limits.so </code> Anything in particular i am doing wrong in particular..? Please find the attached files in mail. -- NOEL ALEX MAKUMULI
nifi.properties
Description: Binary data
bootstrap.conf
Description: Binary data
