Hi, I'm afraid no one read your email carefully. You indeed have one very big machine with 64 physical CPU cores and 120 GB of RAM, correct?
In that case, the amount of RAM you give to the TaskManager seems to low. Could you try re-running your experiments with: jobmanager.heap.mb: 5000 taskmanager.heap.mb: 100000 So, about 100 GB of RAM for the TaskManager. Cheers, Aljoscha On Mon, 4 Apr 2016 at 10:32 Robert Metzger <rmetz...@apache.org> wrote: > Hi, > > usually it doesn't make sense to run multiple task managers on a single > machine to get more slots. > Your machine has only 4 CPU cores, so you are just putting a lot of > pressure on the cpu scheduler.. > > On Thu, Mar 31, 2016 at 7:16 PM, Shinhyung Yang <shinhyung.y...@gmail.com> > wrote: > >> Thank you for replying! >> >> I am trying to do this on a single machine in fact. Since it has 64 >> cores, it would be interesting to look at the performance in that >> regard. >> >> > How many machines are you using for this? >> > >> > The fact that you are giving 64 slots to each TaskManager means that a >> > single TaskManager may end up executing all 64 pipelines. That would >> heavily >> > overload that TaskManager and cause heavy degradation. >> >> Does it make sense if I run multiple TaskManagers on a single machine >> if 64 slots are too many for a TaskManager? >> >> > If, for example, you use 16 machines, then give each machine 4 task >> slots >> > (total of 64 slots across all machines) >> > That way, the final run (parallelism 64) will be guaranteed to be spread >> > across all machines. >> >> My intention for the experiment at the moment is to try to scale the >> application up on a single machine to its maximum before moving on to >> run the experiment on multiple machines. >> >> Thank you again! >> With best regards, >> Shinhyung Yang >> > >
