I like this approach and would suggest to make the ratio configurable. The default could be 50/50 or 60/40 (op heap / net heap)
On Mon, Feb 2, 2015 at 6:45 PM, Ufuk Celebi <u.cel...@fu-berlin.de> wrote: > Currently, the memory configuration of a task manager encompasses two > things: > > 1) NETWORK buffers: Fixed amount of memory for the network buffer pool > (default: 2048 buffers, each 32 KB => 64 MB) > > 2) OPERATOR buffers: A configurable fraction of the available heap memory > (default 0.7) for the memory manager used by internal operators like sort > or hash > > With the recently added supported for intermediate results, intermediate > results live in the network stack and use buffers from the network buffer > pool. Currently, our memory management is not a problem, because we only > support ephemeral intermediate results, which are directly consumed in a > pipelined fashion (i.e. the buffer pools are short-lived). > > But with the upcoming support for persistent intermediate results ( > https://github.com/apache/flink/pull/356) and fine-grained fault > tolerance, we need to rethink how we configure/divide the available memory > between the network stack and the operators as the network buffer pools > will live longer. > > I would suggest the following: > > 1) Remove the configuration for network buffers > > 2) Keep the fraction configuration, but internally divide it between > network stack and operators in a 50:50 fashion (in the future with dynamic > memory management, we will not even need to statically divide the memory > between network stack and operators). > > > What do you think? Is this reasonable? Should the division be configurable > as well? > > – Ufuk
