Hi Thắng,
Also, is your concern more on the amount of processors on canvas and how
it may slow down the UI?
If that's case, you probably can think about creating process groups and
pushing relevant processors into them.
regards,
Ravi Nallappan
On 13/10/21 6:18 pm, Edward Armes wrote:
Hi Thắng,
It feels like to me you may need more nifi nodes in your cluster, as
sounds like he current load is not distributed enough across the
cluster. Would you be able share a few additional pieces of
information to help the community help you? Specificly what version of
Nifi you are running, what version of Java your using this will help
us give you more specific advice.
In general a few things to think about that might help you improve the
performance of you Nifi cluster:
- Look at adjusting the scheduling of your less active processors so
that they are concidered for running less often by the framework.
- Look at setting up a reporting task and using that to capture the
metrics from your cluster to external system like Prometheus to give
you an idea of what processors are not performing as expected and
where your bottlenecks are.
- If you are using custom processors check that they are not doing
anything in the background that may result in additional memory
consumption.
- Try to reduce the ammount of information you keep in the flow-files
as these are normally kept in memory, where as the content is always
kept on disk (except be used in a processor)
- Investigate can be done with things like queue limits and back
preasure to reduce the ammount of WIP in the cluster.
Hope that helps
Edward
On Tue, 12 Oct 2021, 10:07 Thắng Nguyễn Đình, <[email protected]
<mailto:[email protected]>> wrote:
Hi everyone,
I have a question about the way a NiFi cluster is scaled when we
have more new data Flows.
We are having a NiFi cluster with 3 nodes running on production
for about 2 years.
Currently, there are 3000 processors running. The UI interaction
will slow down when more processors are added and more JVM memory
required.
If we add more nodes to the cluster, the UI interaction will be
slow as the replication of Flow configuration and the required JVM
memory will not decrease as the number of processors remains the same.
Should we set up a separate cluster for new data Flows?
Could you please give me a suggestion? Thank you so much!
*Our server specs:*
Intel(R) Xeon(R) CPU E5-2620 v3 @ 2.40GHz
CPU(s): 24
Mem: 62 gb (total), 24 gb (used), 7 gb(free), ... cached
*NiFi instance specs:*
# Secure setup
# JVM memory settings
java.arg.2=-Xms24g
java.arg.3=-Xmx24g
# Repos
50 % of 1.7 Tb HDD disk
# Network: 10Gb/s bandwidth
Thanks & Regards
Thắng, Nguyễn Đình
