Re: Insufficient number of network buffers- what does Total mean on the Flink Dashboard

2020-06-12 Thread Xintong Song
> > Assuming no -Xmx is set, the doc above says 1/4 of physical memory i.e > 29GB will be used. > This is true. So, if I can set env.java.opts: "-Xmx102g" in flink-conf.yaml, I am > assuming the heap max of 102Gb will be used in the N/w mem calculation. > Is that the right way to set

Re: Insufficient number of network buffers- what does Total mean on the Flink Dashboard

2020-06-12 Thread Vijay Balakrishnan
Hi Xintong, Just to be clear. I haven't set any -Xmx -i will check our scripts again. Assuming no -Xmx is set, the doc above says 1/4 of physical memory i.e 29GB will be used. So, if I can set env.java.opts: "-Xmx102g" in flink-conf.yaml, I am assuming the heap max of 102Gb will be used in the

Re: Insufficient number of network buffers- what does Total mean on the Flink Dashboard

2020-06-12 Thread Xintong Song
Flink should have calculated the heap size and set the -Xms, according to the equations I mentioned. So if you haven't set an customized -Xmx that overwrites this, it should not use the default 1.4 of physical memory. > > >- Standalone: jvmHeap = total * (1 - networkFraction) = 102 GB * (1 - >

Re: Insufficient number of network buffers- what does Total mean on the Flink Dashboard

2020-06-12 Thread Vijay Balakrishnan
Thx, Xintong for a great answer. Much appreciated. https://ci.apache.org/projects/flink/flink-docs-release-1.9/ops/mem_setup.html#jvm-heap Max heap: if -Xmx is set then it is its value else ΒΌ of physical machine memory estimated by the JVM No -Xmx is set.So, 1/4 of 102GB = 25.5GB but not sure

Re: Insufficient number of network buffers- what does Total mean on the Flink Dashboard

2020-06-11 Thread Xintong Song
Hi Vijay, The memory configurations in Flink 1.9 and previous versions are indeed complicated and confusing. That is why we made significant changes to it in Flink 1.10. If possible, I would suggest upgrading to Flink 1.10, or the upcoming Flink 1.11 which is very likely to be released in this