Thank you for replying! I am trying to do this on a single machine in fact. Since it has 64 cores, it would be interesting to look at the performance in that regard.
> How many machines are you using for this? > > The fact that you are giving 64 slots to each TaskManager means that a > single TaskManager may end up executing all 64 pipelines. That would heavily > overload that TaskManager and cause heavy degradation. Does it make sense if I run multiple TaskManagers on a single machine if 64 slots are too many for a TaskManager? > If, for example, you use 16 machines, then give each machine 4 task slots > (total of 64 slots across all machines) > That way, the final run (parallelism 64) will be guaranteed to be spread > across all machines. My intention for the experiment at the moment is to try to scale the application up on a single machine to its maximum before moving on to run the experiment on multiple machines. Thank you again! With best regards, Shinhyung Yang
