Oh yes, good point! The documentation needs to be updated to something like:
"The amount of memory (in megabytes) that the task manager reserves on-heap or off-heap (depending on taskmanager.memory.off-heap) for sorting, hash tables, and caching of intermediate results. If unspecified (-1), the memory manager will take a fixed ratio with respect to the size of the task manager JVM as specified by taskmanager.memory.fraction." I will open a JIRA to fix the documentation. If you do the following taskmanager.heap.mb: 4096 taskmanager.memory.size: 2048 taskmanager.memory.off-heap: true The TaskManager will be started with a 2GB (4096MB-2048MB) JVM and allocate (2048MB) as off-heap memory. Hence, the overall process size will be roughly 4GB. The parameter name "taskmanager.heap.mb" is a bit confusing in case of off-heap memory usage, because it does not define this size of the heap but of the overall process. Hope this helps, Fabian 2016-03-16 12:40 GMT+01:00 Ovidiu-Cristian MARCU < ovidiu-cristian.ma...@inria.fr>: > Thanks! > I will try this one: taskmanager.memory.size. So I should expect this > will be the off-heap memory size, right? > I am using taskmanager.heap.mb=some value, taskmanager.memory.off-heap: > true > Memory usage goes up to 99%. > > The documentation is confusing: > > taskmanager.memory.size: The amount of memory (in megabytes) that *the > task manager reserves on the JVM’s heap* space for sorting, hash tables, > and caching of intermediate results. If unspecified (-1), the memory > manager will take a fixed ratio of the heap memory available to the JVM, as > specified by taskmanager.memory.fraction. [1] > > [1] > https://ci.apache.org/projects/flink/flink-docs-release-1.0/setup/config.html#managed-memory > > Best, > Ovidiu > > On 16 Mar 2016, at 12:13, Fabian Hueske <fhue...@gmail.com> wrote: > > Hi Ovidiu, > > the parameters to configure the amount of managed memory > (taskmanager.memory.size, > taskmanager.memory.fraction) are valid for on and off-heap memory. > > Have you tried these parameters and didn't they work as expected? > > Best, Fabian > > > 2016-03-16 11:43 GMT+01:00 Ovidiu-Cristian MARCU < > ovidiu-cristian.ma...@inria.fr>: > >> Hi, >> >> Is it possible to add a parameter off-heap.size for the task manager >> off-heap memory [1]? >> >> It is not possible to limit the off-heap memory size, at least I found >> nothing in the documentation. >> >> [1] >> https://ci.apache.org/projects/flink/flink-docs-release-1.0/setup/config.html#managed-memory >> >> Best, >> Ovidiu >> > > >
