RE: Taking Huge Time for Connect all Nodes to NIFI cluster

Hi Vijay, Thanks for your inputs. Actually we found that there is approximation formula for this setting. Recommended configuration for this 2-4 times than number of cores on machine. We have 48 cores on machine (multithreading enabled), hence configured 100 Max Timer Driven Threads at this poi

RE: Taking Huge Time for Connect all Nodes to NIFI cluster

Hi Joe Witt, Yes, first we want to identify actual underlying problem before splitting up NIFI cluster. Recently observed below scenario as well in the same cluster? We have too many small flow files all the time, on average 5M-10M flow files. NIFI UI/REST API response became too slow & one no

Re: Taking Huge Time for Connect all Nodes to NIFI cluster

Avinash, Did you try to tune Max Timer Driven Thread : 100 Set this value to (number of cores per node) * (number of nodes) i.e. 48 *13 = 624 https://docs.cloudera.com/cfm/2.1.1/nifi-tuning/topics/cfm-tuning-recommendations.html

RE: Taking Huge Time for Connect all Nodes to NIFI cluster

Hi Joe, Thanks for your feedback. It’s deployed by using Hortonworks stack, so if we have to upgrade our NIFI then we might have to upgrade entire stack. Also we have some integrations done to NIFI using REST API along with some custom processors. We have plans to upgrade down the lane, but not

Re: Taking Huge Time for Connect all Nodes to NIFI cluster

I dont recommend splitting the clusters for this. Lets figure out what is happening for your case. On Tue, Jun 29, 2021 at 10:25 PM Modepalli Venkata Avinash < modepalli.avin...@subex.com> wrote: > Hi Joe, > > > > Thanks for your feedback. > > Yes, working on to split into multiple clusters. But

RE: Taking Huge Time for Connect all Nodes to NIFI cluster

Hi Joe, Thanks for your feedback. Yes, working on to split into multiple clusters. But it looks like this problem will be still there while restarting NIFI cluster if flow.xml.gz size is grows in 10’s of MBs (When more number of components/processors deployed on root canvas increases). Thanks

Re: NiFi configuration files changes

The physical files will get synchronized to the reference state from a central config management source (CM). There's no point watching them on the file system. If you need a change log for config files, I'd look into CM api to fetch those instead. On Tue, Jun 29, 2021, 8:30 AM Tomislav Novosel <

NiFi configuration files changes

Hi to all, Is there a good way how to capture NiFi configuration files changes (nifi.properties, authorizers.xml,...etc) and to forward that changes (or just to notify) some other system or app? Can I do it with NiFi itself? The question is in the context of Cloudera platform - CFM. Thanks, Rega

Re: Taking Huge Time for Connect all Nodes to NIFI cluster

Hello A cluster that size should be fine. We did make various improvement to cluster behavior and startup times though. What prevents you from moving to 1.13? How many flowfiles are in the repo when restarting is taking that long? thanks On Tue, Jun 29, 2021 at 7:38 AM Joe Obernberger < joseph

Re: Taking Huge Time for Connect all Nodes to NIFI cluster

Impressive cluster size!  I do not have an answer for you, but could you change your architecture so that instead of one large NiFi cluster you have 2 or 3 smaller clusters?  Very curious on the answer here as I have also noticed UI slow-downs as the number of nodes increases. -Joe On 6/29/20

RE: Taking Huge Time for Connect all Nodes to NIFI cluster

Hi List, We have 13 Nodes NIFI cluster in production & it's taking huge time for completing NIFI restart. According to our analysis, flow election & flow validation from other nodes with coordinator is taking more time, approx. ~30 hours. Even after all 13 nodes gets connected, NIFI UI responds