The TM's request the buffers in batches, so you 384 were requested, but only 200 were left in the pool. This means your overall pool size is too small. Here is the relevant section from the documentation:
https://ci.apache.org/projects/flink/flink-docs-release-0.10/setup/config.html#configuring-the-network-buffers On Thu, May 5, 2016 at 1:30 PM, Punit Naik <naik.puni...@gmail.com> wrote: > Yes I followed it and changed it to 298 but again it said the same thing. > The only change was that it now said "required 298, but only 200 available". > > Why did it say that? > > On Thu, May 5, 2016 at 4:50 PM, Robert Metzger <rmetz...@apache.org> > wrote: > >> Hi, >> >> I think you've chosen a good initial value for the parallelism. >> The higher the parallelism, the more network buffers are needed. I would >> follow the recommendation from the exception and increase the number of >> network buffers. >> >> On Thu, May 5, 2016 at 11:23 AM, Punit Naik <naik.puni...@gmail.com> >> wrote: >> >>> Hello >>> >>> I was running a program with 'parallelism.default' of 384 as I read in >>> the documentation on Flink's official page that 'parallelism.default' is >>> "the total number of CPUs in the cluster". I have four machines with 96 >>> cores on each of them. So 96*4=384. But the program thew an error saying: >>> >>> Caused by: java.io.IOException: Insufficient number of network buffers: >>> required 384, but only 298 available. The total number of network buffers >>> is currently set to 2048. You can increase this number by setting the >>> configuration key 'taskmanager.network.numberOfBuffers'. >>> >>> What does this mean? And how to choose a proper value for parallelism? >>> >>> -- >>> Thank You >>> >>> Regards >>> >>> Punit Naik >>> >> >> > > > -- > Thank You > > Regards > > Punit Naik > -- My GPG Key ID: 336E2680
