It's very hard to give you any advice without any knowledge of what your data looks like and what processors you're using.
On Thu, Aug 17, 2017 at 7:05 PM, Noel Alex Makumuli <[email protected]> wrote: > Hello guys, > > Ever since I tried Apache Nifi, I was amazed with its powerful features > and > perfomance. > Every day, I would like to understand and learn more of Nifi and I try to > improve my knowledge of by reading articles and following on the forums. > The applications of nifi have simplied a lot of work as ordinary user and > developer [ I have yet to explore the advance use cases, and i will soon] > > Nifi is doing a fantastic job and I am wondering if there anything which > I can > do to improve its performance from my current settings. > > I have thousands and thousands [ more than many 20million+] files which > need > ETL and I am wondering what are the two cents of valuable advice would > advice > me on how i can improve the performance [ meaning transform more files per > day/ hour]. The files size ranges from 1MB to 60MB. > > My set up is very simple. Everything reside in the same location [ > location]. > However, i need to know if there is anything i can do to improve the > performance. I am sure there one or two twiks I can do achieve the best > performance. Please kindly do share some valuable tips for me. > > <code> > openjdk version "1.8.0_131" > OpenJDK Runtime Environment (build 1.8.0_131-8u131-b11-0ubuntu1.1 > 6.04.2-b11) > OpenJDK 64-Bit Server VM (build 25.131-b11, mixed mode) > > Nifi 1.3 > > Server RAM 128 GB > </code> > File open limits.. > /etc/security/limits.conf > <code> > * hard nofile 500000 > * soft nofile 500000 > root hard nofile 500000 > root soft nofile 500000 > > session required pam_limits.so > </code> > Anything in particular i am doing wrong in particular..? > > Please find the attached files in mail. > -- > NOEL ALEX MAKUMULI >
