Hello,
In my application, I need to run several 'topologies' in parallel.
I understand I can have a number of 'slots', which can be assigned. Once the
number of slots available are used, no topology actually runs, even if they can
load.
If i wanted to run many topologies, I need to set a high number of executors.
I also see that each time a topology is started, a new JVM is started, with its
own memory allocation etc...
So I have a few questions:
- what is the rule of thumb for slots / executors per CPU? How far can it go?-
how much memory does the JVM require each time a topology is ran? I assume as I
spin many topologies, I will end up saturating the memory and it will be the
bottleneck even if the topologies don't need much CPU time.
I'm trying to devise a strategy to figure what is the best way to deploy many
topologies: uber-jar with many topologies defined into one larger one to cut
down on memory usage, with less flexibility to change / run the separate
topologies.
What is the recommended way to proceed in such use case?
Any help greatly appreciated.
Best regards
