xintongsong commented on a change in pull request #11947:
URL: https://github.com/apache/flink/pull/11947#discussion_r420517791
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -30,104 +30,127 @@ Flink allows both high level and fine-grained tuning of
memory allocation within
* toc
{:toc}
-The further described memory configuration is applicable starting with the
release version *1.10*. If you upgrade Flink
-from earlier versions, check the [migration guide](mem_migration.html) because
many changes were introduced with the *1.10* release.
-
-<span class="label label-info">Note</span> This memory setup guide is relevant
<strong>only for task executors</strong>!
-Check [job manager related configuration
options](../config.html#jobmanager-heap-size) for the memory setup of job
manager.
+The further described memory configuration is applicable starting with the
release version *1.10* for task executors and
+*1.11* for job executors. If you upgrade Flink from earlier versions, check
the [migration guide](mem_migration.html)
+because many changes were introduced with the *1.10* and *1.11* releases.
## Configure Total Memory
-The *total process memory* of Flink JVM processes consists of memory consumed
by Flink application (*total Flink memory*)
-and by the JVM to run the process. The *total Flink memory* consumption
includes usage of JVM heap,
-*managed memory* (managed by Flink) and other direct (or native) memory.
+The *total process memory* of Flink JVM processes consists of memory consumed
by the Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption
includes usage of *JVM Heap* and *Off-heap*
+(*Direct* or *Native*) memory.
<center>
- <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px"
alt="Simple memory model" usemap="#simple-mem-model">
+ <img src="{{ site.baseurl }}/fig/process_mem_model.svg" width="300px"
alt="Flink's process memory model" usemap="#process-mem-model">
</center>
<br />
-If you run Flink locally (e.g. from your IDE) without creating a cluster, then
only a subset of the memory configuration
-options are relevant, see also [local
execution](mem_detail.html#local-execution) for more details.
+The simplest way to setup memory in Flink is to configure either of the two
following options:
+
+| **Component** | **Option for task
executor** | **Option
for job executor** |
Review comment:
```suggestion
| **Component** | **Option for task
executor** | **Option
for job manager** |
```
Is this intentional? I'm not sure about the phrase "job executor". I don't
see it used in any other docs.
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -30,104 +30,127 @@ Flink allows both high level and fine-grained tuning of
memory allocation within
* toc
{:toc}
-The further described memory configuration is applicable starting with the
release version *1.10*. If you upgrade Flink
-from earlier versions, check the [migration guide](mem_migration.html) because
many changes were introduced with the *1.10* release.
-
-<span class="label label-info">Note</span> This memory setup guide is relevant
<strong>only for task executors</strong>!
-Check [job manager related configuration
options](../config.html#jobmanager-heap-size) for the memory setup of job
manager.
+The further described memory configuration is applicable starting with the
release version *1.10* for task executors and
+*1.11* for job executors. If you upgrade Flink from earlier versions, check
the [migration guide](mem_migration.html)
Review comment:
```suggestion
*1.11* for job managers. If you upgrade Flink from earlier versions, check
the [migration guide](mem_migration.html)
```
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -30,104 +30,127 @@ Flink allows both high level and fine-grained tuning of
memory allocation within
* toc
{:toc}
-The further described memory configuration is applicable starting with the
release version *1.10*. If you upgrade Flink
-from earlier versions, check the [migration guide](mem_migration.html) because
many changes were introduced with the *1.10* release.
-
-<span class="label label-info">Note</span> This memory setup guide is relevant
<strong>only for task executors</strong>!
-Check [job manager related configuration
options](../config.html#jobmanager-heap-size) for the memory setup of job
manager.
+The further described memory configuration is applicable starting with the
release version *1.10* for task executors and
+*1.11* for job executors. If you upgrade Flink from earlier versions, check
the [migration guide](mem_migration.html)
+because many changes were introduced with the *1.10* and *1.11* releases.
## Configure Total Memory
-The *total process memory* of Flink JVM processes consists of memory consumed
by Flink application (*total Flink memory*)
-and by the JVM to run the process. The *total Flink memory* consumption
includes usage of JVM heap,
-*managed memory* (managed by Flink) and other direct (or native) memory.
+The *total process memory* of Flink JVM processes consists of memory consumed
by the Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption
includes usage of *JVM Heap* and *Off-heap*
+(*Direct* or *Native*) memory.
<center>
- <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px"
alt="Simple memory model" usemap="#simple-mem-model">
+ <img src="{{ site.baseurl }}/fig/process_mem_model.svg" width="300px"
alt="Flink's process memory model" usemap="#process-mem-model">
</center>
<br />
-If you run Flink locally (e.g. from your IDE) without creating a cluster, then
only a subset of the memory configuration
-options are relevant, see also [local
execution](mem_detail.html#local-execution) for more details.
+The simplest way to setup memory in Flink is to configure either of the two
following options:
+
+| **Component** | **Option for task
executor** | **Option
for job executor** |
+| :------------------------------------ |
:----------------------------------------------------------------------------------
|
:--------------------------------------------------------------------------------
|
+| Total Flink memory |
[`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)
|
[`jobmanager.memory.flink.size`](../config.html#jobmanager-memory-flink-size)
|
+| Total process memory |
[`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
|
[`jobmanager.memory.process.size`](../config.html#jobmanager-memory-process-size)
|
+{:.table-bordered}
+<br/>
-Otherwise, the simplest way to setup memory in Flink is to configure either of
the two following options:
-* Total Flink memory
([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size))
-* Total process memory
([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size))
+See also notes for the execution mode of [task
executor](mem_setup_tm.html#local-execution) and
Review comment:
Additionally, I would suggest to label this sentence as a `note`.
Currently, it's IMO a bit interrupting, making it hard to connect "the rest of
the memory components" with "either of the two following options".
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -30,104 +30,127 @@ Flink allows both high level and fine-grained tuning of
memory allocation within
* toc
{:toc}
-The further described memory configuration is applicable starting with the
release version *1.10*. If you upgrade Flink
-from earlier versions, check the [migration guide](mem_migration.html) because
many changes were introduced with the *1.10* release.
-
-<span class="label label-info">Note</span> This memory setup guide is relevant
<strong>only for task executors</strong>!
-Check [job manager related configuration
options](../config.html#jobmanager-heap-size) for the memory setup of job
manager.
+The further described memory configuration is applicable starting with the
release version *1.10* for task executors and
+*1.11* for job executors. If you upgrade Flink from earlier versions, check
the [migration guide](mem_migration.html)
+because many changes were introduced with the *1.10* and *1.11* releases.
## Configure Total Memory
-The *total process memory* of Flink JVM processes consists of memory consumed
by Flink application (*total Flink memory*)
-and by the JVM to run the process. The *total Flink memory* consumption
includes usage of JVM heap,
-*managed memory* (managed by Flink) and other direct (or native) memory.
+The *total process memory* of Flink JVM processes consists of memory consumed
by the Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption
includes usage of *JVM Heap* and *Off-heap*
+(*Direct* or *Native*) memory.
<center>
- <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px"
alt="Simple memory model" usemap="#simple-mem-model">
+ <img src="{{ site.baseurl }}/fig/process_mem_model.svg" width="300px"
alt="Flink's process memory model" usemap="#process-mem-model">
</center>
<br />
-If you run Flink locally (e.g. from your IDE) without creating a cluster, then
only a subset of the memory configuration
-options are relevant, see also [local
execution](mem_detail.html#local-execution) for more details.
+The simplest way to setup memory in Flink is to configure either of the two
following options:
+
+| **Component** | **Option for task
executor** | **Option
for job executor** |
Review comment:
If not, we'd better search for the pattern "job executor" in the
project, because there are quite some occurrences.
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -30,104 +30,127 @@ Flink allows both high level and fine-grained tuning of
memory allocation within
* toc
{:toc}
-The further described memory configuration is applicable starting with the
release version *1.10*. If you upgrade Flink
-from earlier versions, check the [migration guide](mem_migration.html) because
many changes were introduced with the *1.10* release.
-
-<span class="label label-info">Note</span> This memory setup guide is relevant
<strong>only for task executors</strong>!
-Check [job manager related configuration
options](../config.html#jobmanager-heap-size) for the memory setup of job
manager.
+The further described memory configuration is applicable starting with the
release version *1.10* for task executors and
+*1.11* for job executors. If you upgrade Flink from earlier versions, check
the [migration guide](mem_migration.html)
+because many changes were introduced with the *1.10* and *1.11* releases.
## Configure Total Memory
-The *total process memory* of Flink JVM processes consists of memory consumed
by Flink application (*total Flink memory*)
-and by the JVM to run the process. The *total Flink memory* consumption
includes usage of JVM heap,
-*managed memory* (managed by Flink) and other direct (or native) memory.
+The *total process memory* of Flink JVM processes consists of memory consumed
by the Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption
includes usage of *JVM Heap* and *Off-heap*
+(*Direct* or *Native*) memory.
<center>
- <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px"
alt="Simple memory model" usemap="#simple-mem-model">
+ <img src="{{ site.baseurl }}/fig/process_mem_model.svg" width="300px"
alt="Flink's process memory model" usemap="#process-mem-model">
</center>
<br />
-If you run Flink locally (e.g. from your IDE) without creating a cluster, then
only a subset of the memory configuration
-options are relevant, see also [local
execution](mem_detail.html#local-execution) for more details.
+The simplest way to setup memory in Flink is to configure either of the two
following options:
+
+| **Component** | **Option for task
executor** | **Option
for job executor** |
+| :------------------------------------ |
:----------------------------------------------------------------------------------
|
:--------------------------------------------------------------------------------
|
+| Total Flink memory |
[`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)
|
[`jobmanager.memory.flink.size`](../config.html#jobmanager-memory-flink-size)
|
+| Total process memory |
[`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
|
[`jobmanager.memory.process.size`](../config.html#jobmanager-memory-process-size)
|
+{:.table-bordered}
+<br/>
-Otherwise, the simplest way to setup memory in Flink is to configure either of
the two following options:
-* Total Flink memory
([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size))
-* Total process memory
([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size))
+See also notes for the execution mode of [task
executor](mem_setup_tm.html#local-execution) and
Review comment:
```suggestion
For local execution, see also notes for the execution mode of [task
executor](mem_setup_tm.html#local-execution) and
```
I think the linked notes are only relevant to local execution mode. It might
be better to mention that so people who don't care about local execution won't
bother to jump to the notes.
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -30,104 +30,127 @@ Flink allows both high level and fine-grained tuning of
memory allocation within
* toc
{:toc}
-The further described memory configuration is applicable starting with the
release version *1.10*. If you upgrade Flink
-from earlier versions, check the [migration guide](mem_migration.html) because
many changes were introduced with the *1.10* release.
-
-<span class="label label-info">Note</span> This memory setup guide is relevant
<strong>only for task executors</strong>!
-Check [job manager related configuration
options](../config.html#jobmanager-heap-size) for the memory setup of job
manager.
+The further described memory configuration is applicable starting with the
release version *1.10* for task executors and
+*1.11* for job executors. If you upgrade Flink from earlier versions, check
the [migration guide](mem_migration.html)
+because many changes were introduced with the *1.10* and *1.11* releases.
## Configure Total Memory
-The *total process memory* of Flink JVM processes consists of memory consumed
by Flink application (*total Flink memory*)
-and by the JVM to run the process. The *total Flink memory* consumption
includes usage of JVM heap,
-*managed memory* (managed by Flink) and other direct (or native) memory.
+The *total process memory* of Flink JVM processes consists of memory consumed
by the Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption
includes usage of *JVM Heap* and *Off-heap*
+(*Direct* or *Native*) memory.
<center>
- <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px"
alt="Simple memory model" usemap="#simple-mem-model">
+ <img src="{{ site.baseurl }}/fig/process_mem_model.svg" width="300px"
alt="Flink's process memory model" usemap="#process-mem-model">
</center>
<br />
-If you run Flink locally (e.g. from your IDE) without creating a cluster, then
only a subset of the memory configuration
-options are relevant, see also [local
execution](mem_detail.html#local-execution) for more details.
+The simplest way to setup memory in Flink is to configure either of the two
following options:
+
+| **Component** | **Option for task
executor** | **Option
for job executor** |
+| :------------------------------------ |
:----------------------------------------------------------------------------------
|
:--------------------------------------------------------------------------------
|
+| Total Flink memory |
[`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)
|
[`jobmanager.memory.flink.size`](../config.html#jobmanager-memory-flink-size)
|
+| Total process memory |
[`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
|
[`jobmanager.memory.process.size`](../config.html#jobmanager-memory-process-size)
|
+{:.table-bordered}
+<br/>
-Otherwise, the simplest way to setup memory in Flink is to configure either of
the two following options:
-* Total Flink memory
([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size))
-* Total process memory
([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size))
+See also notes for the execution mode of [task
executor](mem_setup_tm.html#local-execution) and
+[job executor](mem_setup_jm.html#notes-for-the-execution-mode).
The rest of the memory components will be adjusted automatically, based on
default values or additionally configured options.
-[Here](mem_detail.html) are more details about the other memory components.
+See also how to set up other components for [task executor](mem_setup_tm.html)
and [job executor](mem_setup_jm.html) memory.
-Configuring *total Flink memory* is better suited for standalone deployments
where you want to declare how much memory
-is given to Flink itself. The *total Flink memory* splits up into JVM heap,
[managed memory size](#managed-memory)
-and *direct memory*.
+Configuring *total Flink memory* is better suited for [standalone
deployments](../deployment/cluster_setup.html)
+where you want to declare how much memory is given to Flink itself. The *total
Flink memory* splits up into *JVM Heap*
+and *Off-heap* memory.
+See also [how to configure memory for standalone
deployments](mem_tuning.html#configure-memory-for-standalone-deployment).
If you configure *total process memory* you declare how much memory in total
should be assigned to the Flink *JVM process*.
For the containerized deployments it corresponds to the size of the requested
container, see also
[how to configure memory for
containers](mem_tuning.html#configure-memory-for-containers)
([Kubernetes](../deployment/kubernetes.html),
[Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
-Another way to setup the memory is to set [task
heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
-([`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size)
and
[`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size)).
-This more fine-grained approach is described in more detail
[here](#configure-heap-and-managed-memory).
-
-<span class="label label-info">Note</span> One of the three mentioned ways has
to be used to configure Flink’s memory (except for local execution), or the
Flink startup will fail.
-This means that one of the following option subsets, which do not have default
values, have to be configured explicitly:
-*
[`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)
-*
[`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
-*
[`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size)
and
[`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size)
-
-<span class="label label-info">Note</span> Explicitly configuring both *total
process memory* and *total Flink memory* is not recommended.
-It may lead to deployment failures due to potential memory configuration
conflicts. Additional configuration
-of other memory components also requires caution as it can produce further
configuration conflicts.
-
-## Configure Heap and Managed Memory
-
-As mentioned before in [total memory description](#configure-total-memory),
another way to setup memory in Flink is
-to specify explicitly both [task heap](#task-operator-heap-memory) and
[managed memory](#managed-memory).
-It gives more control over the available JVM heap to Flink’s tasks and its
[managed memory](#managed-memory).
-
-The rest of the memory components will be adjusted automatically, based on
default values or additionally configured options.
-[Here](mem_detail.html) are more details about the other memory components.
-
-<span class="label label-info">Note</span> If you have configured the task
heap and managed memory explicitly, it is recommended to set neither
-*total process memory* nor *total Flink memory*. Otherwise, it may easily lead
to memory configuration conflicts.
-
-### Task (Operator) Heap Memory
-
-If you want to guarantee that a certain amount of JVM heap is available for
your user code, you can set the *task heap memory*
-explicitly
([`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size)).
-It will be added to the JVM heap size and will be dedicated to Flink’s
operators running the user code.
-
-### Managed Memory
-
-*Managed memory* is managed by Flink and is allocated as native memory
(off-heap). The following workloads use *managed memory*:
-* Streaming jobs can use it for [RocksDB state
backend](../state/state_backends.html#the-rocksdbstatebackend).
-* [Batch jobs](../../dev/batch) can use it for sorting, hash tables, caching
of intermediate results.
-
-The size of *managed memory* can be
-* either configured explicitly via
[`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size)
-* or computed as a fraction of *total Flink memory* via
[`taskmanager.memory.managed.fraction`](../config.html#taskmanager-memory-managed-fraction).
-
-*Size* will override *fraction*, if both are set.
-If neither *size* nor *fraction* is explicitly configured, the [default
fraction](../config.html#taskmanager-memory-managed-fraction) will be used.
-
-See also [how to configure memory for state
backends](mem_tuning.html#configure-memory-for-state-backends) and [batch
jobs](mem_tuning.html#configure-memory-for-batch-jobs).
-
-## Configure Off-Heap Memory (direct or native)
-
-The off-heap memory which is allocated by user code should be accounted for in
*task off-heap memory*
-([`taskmanager.memory.task.off-heap.size`](../config.html#taskmanager-memory-task-off-heap-size)).
-
-<span class="label label-info">Note</span> You can also adjust the [framework
off-heap memory](mem_detail.html#framework-memory). This option is advanced
-and only recommended to be changed if you are sure that the Flink framework
needs more memory.
-
-Flink includes the *framework off-heap memory* and *task off-heap memory* into
the *direct memory* limit of the JVM,
-see also [JVM parameters](mem_detail.html#jvm-parameters).
-
-<span class="label label-info">Note</span> Although, native non-direct memory
usage can be accounted for as a part of the
-*framework off-heap memory* or *task off-heap memory*, it will result in a
higher JVM's *direct memory* limit in this case.
-
-<span class="label label-info">Note</span> The *network memory* is also part
of JVM *direct memory* but it is managed by Flink and guaranteed
-to never exceed its configured size. Therefore, resizing the *network memory*
will not help in this situation.
-
-See also [the detailed memory model](mem_detail.html).
+Another way to set up the memory is to configure the required internal
components of the *total Flink memory* which are
+specific to the concrete Flink process. Check how to configure them for task
executor [here](mem_setup_tm.html#configure-heap-and-managed-memory)
+and for job executor [here](mem_setup_jm.html#configure-jvm-heap).
+
+<span class="label label-info">Note</span> One of the three mentioned ways has
to be used to configure Flink’s memory
+(except for local execution), or the Flink startup will fail. This means that
one of the following option subsets,
+which do not have default values, have to be configured explicitly:
+
+| **for task executor:**
| **for job manager:**
|
+|
:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|
:--------------------------------------------------------------------------------
|
+|
[`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)
|
[`jobmanager.memory.flink.size`](../config.html#jobmanager-memory-flink-size)
|
+|
[`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
|
[`jobmanager.memory.process.size`](../config.html#jobmanager-memory-process-size)
|
+|
[`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size)
<br/> and
[`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size)
| [`jobmanager.memory.heap.size`](../config.html#jobmanager-memory-heap-size)
|
+{:.table-bordered}
+<br/>
+
+<span class="label label-info">Note</span> Explicitly configuring both *total
process memory* and *total Flink memory*
+is not recommended. It may lead to deployment failures due to potential memory
configuration conflicts. Additional
+configuration of other memory components also requires caution as it can
produce further configuration conflicts.
+
+See also [here](#capped-fractionated-components) how to configure the *JVM
Overhead*.
Review comment:
Why do we specifically mention JVM overhead here?
##########
File path: docs/ops/memory/mem_tuning.md
##########
@@ -63,13 +62,13 @@ will dictate the optimal memory configurations of your
cluster.
### Heap state backend
Review comment:
This whole section (`Configure memory for state backends`) seems to me
only relevant to TMs. Maybe mention that as well, like what we do for
`Configure memory for batch jobs`.
##########
File path: docs/ops/memory/mem_setup_jm.md
##########
@@ -0,0 +1,98 @@
+---
+title: "Set up Job Manager Memory"
+nav-parent_id: ops_mem
+nav-pos: 3
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+* toc
+{:toc}
+
+The further described memory configuration is applicable starting with the
release version *1.11*. If you upgrade Flink
+from earlier versions, check the [migration guide](mem_migration.html) because
many changes were introduced with the *1.11* release.
+
+<span class="label label-info">Note</span> This memory setup guide is relevant
<strong>only for job managers</strong>!
+The main job manager memory components have a similar but simpler structure
compared to the [task executors'](mem_setup_tm.html).
+
+## Configure Total Memory
+
+If you run the job manager process locally, you do not need to configure
memory options, they will have no effect.
+See also [notes for the execution mode](#notes-for-the-execution-mode).
Otherwise, the simplest way to set up the
+memory configuration is to configure the total memory.
+See [how to configure it for the Flink
process](mem_setup.html#configure-total-memory).
+
+## Detailed configuration
+
+The following table lists all memory components, depicted above, and
references Flink configuration options which
+affect the size of the respective components:
+
+| **Component** |
**Configuration options**
|
**Description**
|
+| :------------------------------------------------------------- |
:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|
:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|
+| [Total Process Memory](mem_setup.html#configure-total-memory) |
[`jobmanager.memory.process.size`](../config.html#jobmanager-memory-process-size)
| The *total process memory*
size for the job manager. This includes all the memory that a job manager JVM
process consumes, consisting of the *total Flink memory*, *JVM metaspace* and
*JVM overhead*.
|
+| [Total Flink Memory](mem_setup.html#configure-total-memory) |
[`jobmanager.memory.flink.size`](../config.html#jobmanager-memory-flink-size)
| The *total Flink memory*
size for the job manager. This includes all the memory that a job manager
consumes, except for *JVM metaspace* and *JVM overhead*. It consists of [JVM
Heap](#configure-jvm-heap) and [Off-heap Memory](#configure-off-heap-memory)
Memory. |
+| [JVM Heap](#configure-jvm-heap) |
[`jobmanager.memory.heap.size`](../config.html#jobmanager-memory-heap-size)
| *JVM Heap* memory size
for job manager.
|
+| [Off-heap Memory](#configure-off-heap-memory) |
[`jobmanager.memory.off-heap.size`](../config.html#jobmanager-memory-off-heap-size)
| *Off-heap* memory size for
job manager. This option covers all off-heap memory usage including direct and
native memory allocation.
|
+| [JVM metaspace](mem_setup.html#jvm-parameters) |
[`jobmanager.memory.jvm-metaspace.size`](../config.html#jobmanager-memory-jvm-metaspace-size)
| Metaspace size of the Flink JVM
process
|
+| JVM Overhead |
[`jobmanager.memory.jvm-overhead.min`](../config.html#jobmanager-memory-jvm-overhead-min)
<br/>
[`jobmanager.memory.jvm-overhead.max`](../config.html#jobmanager-memory-jvm-overhead-max)
<br/>
[`jobmanager.memory.jvm-overhead.fraction`](../config.html#jobmanager-memory-jvm-overhead-fraction)
| Native memory reserved for other JVM overhead: e.g. thread stacks, code
cache, garbage collection space etc, it is a [capped fractionated
component](mem_setup.html#capped-fractionated-components) of the [total process
memory](mem_setup.html#configure-total-memory) |
+{:.table-bordered}
+<br/>
+
+### Configure JVM Heap
+
+As mentioned before in the [total memory
description](mem_setup.html#configure-total-memory), another way to setup memory
+for job manager is to specify explicitly *JVM Heap* size
([`jobmanager.memory.heap.size`](../config.html#jobmanager-memory-heap-size)).
+It gives more control over the available *JVM Heap* to the following purposes:
+
+* Flink framework (e.g. *Job cache*)
+* User code running during job submission (e.g. for certain batch sources) or
in checkpoint completion callbacks
+
+The required size of *JVM Heap* is mostly driven by the amount of running
jobs, their structure and requirements for
+the mentioned user code.
+
+The *Job cache* resides in the *JVM Heap*. It already can be configured
currently by
+[`jobstore.cache-size`](../config.html#jobstore-cache-size) which must be less
than the configured or derived *JVM Heap*.
+
+<span class="label label-info">Note</span> If you have configured the *JVM
Heap* explicitly, it is recommended to set
+neither *total process memory* nor *total Flink memory*. Otherwise, it may
easily lead to memory configuration conflicts.
+The *JVM Heap* size is set as the corresponding JVM parameters (*-Xms* and
*-Xmx*) when the job manager process is started
+by Flink’s scripts or CLI, see also [JVM
parameters](mem_setup.html#jvm-parameters).
+
+### Configure Off-heap Memory
+
+The *Off-heap* memory component accounts for any type of *JVM direct memory*
and *native memory* usage. Therefore, it
+is also set as the corresponding JVM argument: *-XX:MaxDirectMemorySize*, see
also [JVM parameters](mem_setup.html#jvm-parameters).
+
+The size of this component can be configured by
[`jobmanager.memory.off-heap.size`](../config.html#jobmanager-memory-off-heap-size)
+option. This option can be tuned e.g. if the job manager process throws
‘OutOfMemoryError: Direct buffer memory’, see
+also [the troubleshooting
guide](mem_trouble.html#outofmemoryerror-direct-buffer-memory).
+
+There can be the following possible sources of the *Off-heap* memory
consumption in JM:
+
+* Flink framework dependencies (e.g. Akka network communication)
+* User code running during job submission (e.g. for certain batch sources) or
in checkpoint completion callbacks
+
+## Notes for the execution mode
Review comment:
I'm wondering whether this section should be titled "local execution" or
so, to align with the TM doc.
It is practically talking about local execution as an exception apart from
other execution modes.
##########
File path: docs/ops/memory/mem_migration.md
##########
@@ -144,89 +157,125 @@ It is also recommended to use these new options instead
of the legacy ones as th
See also [how to configure total memory
now](mem_setup.html#configure-total-memory).
-## JVM Heap Memory
+### JVM Heap Memory
-JVM heap memory previously consisted of the managed memory (if configured to
be on-heap) and the rest
+JVM Heap memory previously consisted of the managed memory (if configured to
be on-heap) and the rest
which included any other usages of heap memory. This rest was always
implicitly derived as the remaining part of the total memory,
see also [how to migrate managed memory](#managed-memory).
-Now, if only *total Flink memory* or *total process memory* is configured,
then the JVM heap is also derived as the rest of
+Now, if only *total Flink memory* or *total process memory* is configured,
then the JVM Heap is also derived as the rest of
what is left after subtracting all other components from the total memory, see
also [how to configure total memory](mem_setup.html#configure-total-memory).
-Additionally, you can now have more direct control over the JVM heap assigned
to the operator tasks
+Additionally, you can now have more direct control over the JVM Heap assigned
to the operator tasks
([`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size)),
-see also [Task (Operator) Heap
Memory](mem_setup.html#task-operator-heap-memory).
-The JVM heap memory is also used by the heap state backends
([MemoryStateBackend](../state/state_backends.html#the-memorystatebackend)
+see also [Task (Operator) Heap
Memory](mem_setup_tm.html#task-operator-heap-memory).
+The JVM Heap memory is also used by the heap state backends
([MemoryStateBackend](../state/state_backends.html#the-memorystatebackend)
or [FsStateBackend](../state/state_backends.html#the-fsstatebackend)) if it is
chosen for streaming jobs.
-A part of the JVM heap is now always reserved for Flink framework
+A part of the JVM Heap is now always reserved for Flink framework
([`taskmanager.memory.framework.heap.size`](../config.html#taskmanager-memory-framework-heap-size)).
-See also [Framework memory](mem_detail.html#framework-memory).
+See also [Framework memory](mem_setup_tm.html#framework-memory).
-## Managed Memory
+### Managed Memory
-See also [how to configure managed memory now](mem_setup.html#managed-memory).
+See also [how to configure managed memory
now](mem_setup_tm.html#managed-memory).
-### Explicit Size
+#### Explicit Size
The previous option to configure managed memory size
(`taskmanager.memory.size`) was renamed to
[`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size)
and deprecated.
It is recommended to use the new option because the legacy one can be removed
in future releases.
-### Fraction
+#### Fraction
If not set explicitly, the managed memory could be previously specified as a
fraction (`taskmanager.memory.fraction`)
of the total memory minus network memory and container cut-off (only for
[Yarn](../deployment/yarn_setup.html) and
[Mesos](../deployment/mesos.html) deployments). This option has been
completely removed and will have no effect if still used.
Please, use the new option
[`taskmanager.memory.managed.fraction`](../config.html#taskmanager-memory-managed-fraction)
instead.
-This new option will set the [managed memory](mem_setup.html#managed-memory)
to the specified fraction of the
+This new option will set the [managed
memory](mem_setup_tm.html#managed-memory) to the specified fraction of the
[total Flink memory](mem_setup.html#configure-total-memory) if its size is not
set explicitly by
[`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size).
-### RocksDB state
+#### RocksDB state
If the
[RocksDBStateBackend](../state/state_backends.html#the-rocksdbstatebackend) is
chosen for a streaming job,
-its native memory consumption should now be accounted for in [managed
memory](mem_setup.html#managed-memory).
-The RocksDB memory allocation is limited by the [managed
memory](mem_setup.html#managed-memory) size.
+its native memory consumption should now be accounted for in [managed
memory](mem_setup_tm.html#managed-memory).
+The RocksDB memory allocation is limited by the [managed
memory](mem_setup_tm.html#managed-memory) size.
This should prevent the killing of containers on
[Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html).
You can disable the RocksDB memory control by setting
[state.backend.rocksdb.memory.managed](../config.html#state-backend-rocksdb-memory-managed)
to `false`. See also [how to migrate container
cut-off](#container-cut-off-memory).
-### Other changes
+#### Other changes
Additionally, the following changes have been made:
-* The [managed memory](mem_setup.html#managed-memory) is always off-heap now.
The configuration option `taskmanager.memory.off-heap` is removed and will have
no effect anymore.
-* The [managed memory](mem_setup.html#managed-memory) now uses native memory
which is not direct memory. It means that the managed memory is no longer
accounted for in the JVM direct memory limit.
-* The [managed memory](mem_setup.html#managed-memory) is always lazily
allocated now. The configuration option `taskmanager.memory.preallocate` is
removed and will have no effect anymore.
+* The [managed memory](mem_setup_tm.html#managed-memory) is always off-heap
now. The configuration option `taskmanager.memory.off-heap` is removed and will
have no effect anymore.
+* The [managed memory](mem_setup_tm.html#managed-memory) now uses native
memory which is not direct memory. It means that the managed memory is no
longer accounted for in the JVM direct memory limit.
+* The [managed memory](mem_setup_tm.html#managed-memory) is always lazily
allocated now. The configuration option `taskmanager.memory.preallocate` is
removed and will have no effect anymore.
+
+## Migrate Job Manager Memory Configuration
+
+Previously, there were options responsible for the *JVM Heap* used by Flink
are:
+* `jobmanager.heap.size`
+* `jobmanager.heap.mb`
+
+Despite their naming, they represented the *JVM Heap* only for [standalone
deployments](../deployment/cluster_setup.html).
+For the containerized deployments ([Kubernetes](../deployment/kubernetes.html)
and [Yarn](../deployment/yarn_setup.html)),
+they also included other off-heap memory consumption. The size of *JVM Heap*
was additionally reduced by the container
+cut-off which has been completely removed after *1.11*.
+
+The [Mesos](../deployment/mesos.html) integration did not take into account
the mentioned legacy memory options.
+The scripts provided in Flink to start the Mesos master did not set any memory
JVM arguments. After *1.11* release,
+they are set the same way as for [standalone
deployment](../deployment/cluster_setup.html) scripts.
+
+The mentioned legacy options have been deprecated. If they are used without
specifying the corresponding new options,
+they will be directly translated into the following new options:
+* Total Flink memory
([`jobmanager.memory.flink.size`](../config.html#jobmanager-memory-flink-size))
for [standalone](../deployment/cluster_setup.html) and
[Mesos](../deployment/mesos.html) deployments
+* Total process memory
([`jobmanager.memory.process.size`](../config.html#jobmanager-memory-process-size))
for containerized deployments ([Kubernetes](../deployment/kubernetes.html) and
[Yarn](../deployment/yarn_setup.html))
+It is also recommended to use these new options instead of the legacy ones as
they might be completely removed in the following releases.
+
+Now, if only the *total Flink memory* or *total process memory* is configured,
then the [JVM Heap](mem_setup_jm.html#configure-jvm-heap)
+is also derived as the rest of what is left after subtracting all other
components from the total memory, see also
+[how to configure total memory](mem_setup.html#configure-total-memory).
Additionally, you can now have more direct
+control over the [JVM Heap](mem_setup_jm.html#configure-jvm-heap) by adjusting
the
+[`jobmanager.memory.heap.size`](../config.html#jobmanager-memory-heap-size)
option.
## Container Cut-Off Memory
For containerized deployments, you could previously specify a cut-off memory.
This memory could accommodate for unaccounted memory allocations.
Dependencies which were not directly controlled by Flink were the main source
of those allocations, e.g. RocksDB, internals of JVM, etc.
This is no longer available and the related configuration options
(`containerized.heap-cutoff-ratio` and `containerized.heap-cutoff-min`)
-will have no effect on the task manager process anymore. The new memory model
introduced more specific memory components,
-described further, to address these concerns.
+will have no effect anymore. The new memory model introduced more specific
memory components, described further, to address these concerns.
+
+### for Task Executors
In streaming jobs which use
[RocksDBStateBackend](../state/state_backends.html#the-rocksdbstatebackend),
the RocksDB
-native memory consumption should be accounted for as a part of the [managed
memory](mem_setup.html#managed-memory) now.
+native memory consumption should be accounted for as a part of the [managed
memory](mem_setup_tm.html#managed-memory) now.
The RocksDB memory allocation is also limited by the configured size of the
[managed memory](mem_setup.html#managed-memory).
-See also [migrating managed memory](#managed-memory) and [how to configure
managed memory now](mem_setup.html#managed-memory).
+See also [migrating managed memory](#managed-memory) and [how to configure
managed memory now](mem_setup_tm.html#managed-memory).
The other direct or native off-heap memory consumers can now be addressed by
the following new configuration options:
* Task off-heap memory
([`taskmanager.memory.task.off-heap.size`](../config.html#taskmanager-memory-task-off-heap-size))
* Framework off-heap memory
([`taskmanager.memory.framework.off-heap.size`](../config.html#taskmanager-memory-framework-off-heap-size))
* JVM metaspace
([`taskmanager.memory.jvm-metaspace.size`](../config.html#taskmanager-memory-jvm-metaspace-size))
-* JVM overhead (see also [detailed new memory model](mem_detail.html))
+* [JVM overhead](mem_setup_tm.html#detailed-memory-model)
-<span class="label label-info">Note</span> The job manager still has container
cut-off memory configuration options.
-The mentioned configuration options remain valid for the job manager in the
same way as before.
+### for Job Managers
+
+The direct or native off-heap memory consumers can now be addressed by the
following new configuration options:
+* Off-heap memory
([`jobmanager.memory.off-heap.size`](../config.html#jobmanager-memory-off-heap-size))
+* JVM metaspace
([`jobmanager.memory.jvm-metaspace.size`](../config.html#jobmanager-memory-jvm-metaspace-size))
+* [JVM overhead](mem_setup_jm.html#detailed-configuration)
## Default Configuration in flink-conf.yaml
This section describes the changes of the default `flink-conf.yaml` shipped
with Flink.
-The total memory (`taskmanager.heap.size`) is replaced by
[`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
+The total memory for task executors (`taskmanager.heap.size`) is replaced by
[`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
in the default `flink-conf.yaml`. The value is also increased from 1024Mb to
1568Mb.
Review comment:
```suggestion
in the default `flink-conf.yaml`. The value is also increased from 1024Mb to
1728Mb.
```
##########
File path: docs/ops/memory/mem_migration.md
##########
@@ -144,89 +157,125 @@ It is also recommended to use these new options instead
of the legacy ones as th
See also [how to configure total memory
now](mem_setup.html#configure-total-memory).
-## JVM Heap Memory
+### JVM Heap Memory
-JVM heap memory previously consisted of the managed memory (if configured to
be on-heap) and the rest
+JVM Heap memory previously consisted of the managed memory (if configured to
be on-heap) and the rest
which included any other usages of heap memory. This rest was always
implicitly derived as the remaining part of the total memory,
see also [how to migrate managed memory](#managed-memory).
-Now, if only *total Flink memory* or *total process memory* is configured,
then the JVM heap is also derived as the rest of
+Now, if only *total Flink memory* or *total process memory* is configured,
then the JVM Heap is also derived as the rest of
what is left after subtracting all other components from the total memory, see
also [how to configure total memory](mem_setup.html#configure-total-memory).
-Additionally, you can now have more direct control over the JVM heap assigned
to the operator tasks
+Additionally, you can now have more direct control over the JVM Heap assigned
to the operator tasks
([`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size)),
-see also [Task (Operator) Heap
Memory](mem_setup.html#task-operator-heap-memory).
-The JVM heap memory is also used by the heap state backends
([MemoryStateBackend](../state/state_backends.html#the-memorystatebackend)
+see also [Task (Operator) Heap
Memory](mem_setup_tm.html#task-operator-heap-memory).
+The JVM Heap memory is also used by the heap state backends
([MemoryStateBackend](../state/state_backends.html#the-memorystatebackend)
or [FsStateBackend](../state/state_backends.html#the-fsstatebackend)) if it is
chosen for streaming jobs.
-A part of the JVM heap is now always reserved for Flink framework
+A part of the JVM Heap is now always reserved for Flink framework
([`taskmanager.memory.framework.heap.size`](../config.html#taskmanager-memory-framework-heap-size)).
-See also [Framework memory](mem_detail.html#framework-memory).
+See also [Framework memory](mem_setup_tm.html#framework-memory).
-## Managed Memory
+### Managed Memory
-See also [how to configure managed memory now](mem_setup.html#managed-memory).
+See also [how to configure managed memory
now](mem_setup_tm.html#managed-memory).
-### Explicit Size
+#### Explicit Size
The previous option to configure managed memory size
(`taskmanager.memory.size`) was renamed to
[`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size)
and deprecated.
It is recommended to use the new option because the legacy one can be removed
in future releases.
-### Fraction
+#### Fraction
If not set explicitly, the managed memory could be previously specified as a
fraction (`taskmanager.memory.fraction`)
of the total memory minus network memory and container cut-off (only for
[Yarn](../deployment/yarn_setup.html) and
[Mesos](../deployment/mesos.html) deployments). This option has been
completely removed and will have no effect if still used.
Please, use the new option
[`taskmanager.memory.managed.fraction`](../config.html#taskmanager-memory-managed-fraction)
instead.
-This new option will set the [managed memory](mem_setup.html#managed-memory)
to the specified fraction of the
+This new option will set the [managed
memory](mem_setup_tm.html#managed-memory) to the specified fraction of the
[total Flink memory](mem_setup.html#configure-total-memory) if its size is not
set explicitly by
[`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size).
-### RocksDB state
+#### RocksDB state
If the
[RocksDBStateBackend](../state/state_backends.html#the-rocksdbstatebackend) is
chosen for a streaming job,
-its native memory consumption should now be accounted for in [managed
memory](mem_setup.html#managed-memory).
-The RocksDB memory allocation is limited by the [managed
memory](mem_setup.html#managed-memory) size.
+its native memory consumption should now be accounted for in [managed
memory](mem_setup_tm.html#managed-memory).
+The RocksDB memory allocation is limited by the [managed
memory](mem_setup_tm.html#managed-memory) size.
This should prevent the killing of containers on
[Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html).
You can disable the RocksDB memory control by setting
[state.backend.rocksdb.memory.managed](../config.html#state-backend-rocksdb-memory-managed)
to `false`. See also [how to migrate container
cut-off](#container-cut-off-memory).
-### Other changes
+#### Other changes
Additionally, the following changes have been made:
-* The [managed memory](mem_setup.html#managed-memory) is always off-heap now.
The configuration option `taskmanager.memory.off-heap` is removed and will have
no effect anymore.
-* The [managed memory](mem_setup.html#managed-memory) now uses native memory
which is not direct memory. It means that the managed memory is no longer
accounted for in the JVM direct memory limit.
-* The [managed memory](mem_setup.html#managed-memory) is always lazily
allocated now. The configuration option `taskmanager.memory.preallocate` is
removed and will have no effect anymore.
+* The [managed memory](mem_setup_tm.html#managed-memory) is always off-heap
now. The configuration option `taskmanager.memory.off-heap` is removed and will
have no effect anymore.
+* The [managed memory](mem_setup_tm.html#managed-memory) now uses native
memory which is not direct memory. It means that the managed memory is no
longer accounted for in the JVM direct memory limit.
+* The [managed memory](mem_setup_tm.html#managed-memory) is always lazily
allocated now. The configuration option `taskmanager.memory.preallocate` is
removed and will have no effect anymore.
+
+## Migrate Job Manager Memory Configuration
+
+Previously, there were options responsible for the *JVM Heap* used by Flink
are:
+* `jobmanager.heap.size`
+* `jobmanager.heap.mb`
+
+Despite their naming, they represented the *JVM Heap* only for [standalone
deployments](../deployment/cluster_setup.html).
+For the containerized deployments ([Kubernetes](../deployment/kubernetes.html)
and [Yarn](../deployment/yarn_setup.html)),
+they also included other off-heap memory consumption. The size of *JVM Heap*
was additionally reduced by the container
+cut-off which has been completely removed after *1.11*.
+
+The [Mesos](../deployment/mesos.html) integration did not take into account
the mentioned legacy memory options.
+The scripts provided in Flink to start the Mesos master did not set any memory
JVM arguments. After *1.11* release,
+they are set the same way as for [standalone
deployment](../deployment/cluster_setup.html) scripts.
+
+The mentioned legacy options have been deprecated. If they are used without
specifying the corresponding new options,
+they will be directly translated into the following new options:
+* Total Flink memory
([`jobmanager.memory.flink.size`](../config.html#jobmanager-memory-flink-size))
for [standalone](../deployment/cluster_setup.html) and
[Mesos](../deployment/mesos.html) deployments
Review comment:
It's unrelated to this PR, but I'm not entirely sure whether this is a
proper behavior for Mesos. Shouldn't we keep consistent behaviors for Mesos and
K8s/Yarn?
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -30,104 +30,127 @@ Flink allows both high level and fine-grained tuning of
memory allocation within
* toc
{:toc}
-The further described memory configuration is applicable starting with the
release version *1.10*. If you upgrade Flink
-from earlier versions, check the [migration guide](mem_migration.html) because
many changes were introduced with the *1.10* release.
-
-<span class="label label-info">Note</span> This memory setup guide is relevant
<strong>only for task executors</strong>!
-Check [job manager related configuration
options](../config.html#jobmanager-heap-size) for the memory setup of job
manager.
+The further described memory configuration is applicable starting with the
release version *1.10* for task executors and
+*1.11* for job executors. If you upgrade Flink from earlier versions, check
the [migration guide](mem_migration.html)
+because many changes were introduced with the *1.10* and *1.11* releases.
## Configure Total Memory
-The *total process memory* of Flink JVM processes consists of memory consumed
by Flink application (*total Flink memory*)
-and by the JVM to run the process. The *total Flink memory* consumption
includes usage of JVM heap,
-*managed memory* (managed by Flink) and other direct (or native) memory.
+The *total process memory* of Flink JVM processes consists of memory consumed
by the Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption
includes usage of *JVM Heap* and *Off-heap*
+(*Direct* or *Native*) memory.
<center>
- <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px"
alt="Simple memory model" usemap="#simple-mem-model">
+ <img src="{{ site.baseurl }}/fig/process_mem_model.svg" width="300px"
alt="Flink's process memory model" usemap="#process-mem-model">
</center>
<br />
-If you run Flink locally (e.g. from your IDE) without creating a cluster, then
only a subset of the memory configuration
-options are relevant, see also [local
execution](mem_detail.html#local-execution) for more details.
+The simplest way to setup memory in Flink is to configure either of the two
following options:
+
+| **Component** | **Option for task
executor** | **Option
for job executor** |
+| :------------------------------------ |
:----------------------------------------------------------------------------------
|
:--------------------------------------------------------------------------------
|
+| Total Flink memory |
[`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)
|
[`jobmanager.memory.flink.size`](../config.html#jobmanager-memory-flink-size)
|
+| Total process memory |
[`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
|
[`jobmanager.memory.process.size`](../config.html#jobmanager-memory-process-size)
|
+{:.table-bordered}
+<br/>
-Otherwise, the simplest way to setup memory in Flink is to configure either of
the two following options:
-* Total Flink memory
([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size))
-* Total process memory
([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size))
+See also notes for the execution mode of [task
executor](mem_setup_tm.html#local-execution) and
+[job executor](mem_setup_jm.html#notes-for-the-execution-mode).
The rest of the memory components will be adjusted automatically, based on
default values or additionally configured options.
-[Here](mem_detail.html) are more details about the other memory components.
+See also how to set up other components for [task executor](mem_setup_tm.html)
and [job executor](mem_setup_jm.html) memory.
-Configuring *total Flink memory* is better suited for standalone deployments
where you want to declare how much memory
-is given to Flink itself. The *total Flink memory* splits up into JVM heap,
[managed memory size](#managed-memory)
-and *direct memory*.
+Configuring *total Flink memory* is better suited for [standalone
deployments](../deployment/cluster_setup.html)
+where you want to declare how much memory is given to Flink itself. The *total
Flink memory* splits up into *JVM Heap*
+and *Off-heap* memory.
+See also [how to configure memory for standalone
deployments](mem_tuning.html#configure-memory-for-standalone-deployment).
If you configure *total process memory* you declare how much memory in total
should be assigned to the Flink *JVM process*.
For the containerized deployments it corresponds to the size of the requested
container, see also
[how to configure memory for
containers](mem_tuning.html#configure-memory-for-containers)
([Kubernetes](../deployment/kubernetes.html),
[Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
-Another way to setup the memory is to set [task
heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
-([`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size)
and
[`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size)).
-This more fine-grained approach is described in more detail
[here](#configure-heap-and-managed-memory).
-
-<span class="label label-info">Note</span> One of the three mentioned ways has
to be used to configure Flink’s memory (except for local execution), or the
Flink startup will fail.
-This means that one of the following option subsets, which do not have default
values, have to be configured explicitly:
-*
[`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)
-*
[`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
-*
[`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size)
and
[`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size)
-
-<span class="label label-info">Note</span> Explicitly configuring both *total
process memory* and *total Flink memory* is not recommended.
-It may lead to deployment failures due to potential memory configuration
conflicts. Additional configuration
-of other memory components also requires caution as it can produce further
configuration conflicts.
-
-## Configure Heap and Managed Memory
-
-As mentioned before in [total memory description](#configure-total-memory),
another way to setup memory in Flink is
-to specify explicitly both [task heap](#task-operator-heap-memory) and
[managed memory](#managed-memory).
-It gives more control over the available JVM heap to Flink’s tasks and its
[managed memory](#managed-memory).
-
-The rest of the memory components will be adjusted automatically, based on
default values or additionally configured options.
-[Here](mem_detail.html) are more details about the other memory components.
-
-<span class="label label-info">Note</span> If you have configured the task
heap and managed memory explicitly, it is recommended to set neither
-*total process memory* nor *total Flink memory*. Otherwise, it may easily lead
to memory configuration conflicts.
-
-### Task (Operator) Heap Memory
-
-If you want to guarantee that a certain amount of JVM heap is available for
your user code, you can set the *task heap memory*
-explicitly
([`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size)).
-It will be added to the JVM heap size and will be dedicated to Flink’s
operators running the user code.
-
-### Managed Memory
-
-*Managed memory* is managed by Flink and is allocated as native memory
(off-heap). The following workloads use *managed memory*:
-* Streaming jobs can use it for [RocksDB state
backend](../state/state_backends.html#the-rocksdbstatebackend).
-* [Batch jobs](../../dev/batch) can use it for sorting, hash tables, caching
of intermediate results.
-
-The size of *managed memory* can be
-* either configured explicitly via
[`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size)
-* or computed as a fraction of *total Flink memory* via
[`taskmanager.memory.managed.fraction`](../config.html#taskmanager-memory-managed-fraction).
-
-*Size* will override *fraction*, if both are set.
-If neither *size* nor *fraction* is explicitly configured, the [default
fraction](../config.html#taskmanager-memory-managed-fraction) will be used.
-
-See also [how to configure memory for state
backends](mem_tuning.html#configure-memory-for-state-backends) and [batch
jobs](mem_tuning.html#configure-memory-for-batch-jobs).
-
-## Configure Off-Heap Memory (direct or native)
-
-The off-heap memory which is allocated by user code should be accounted for in
*task off-heap memory*
-([`taskmanager.memory.task.off-heap.size`](../config.html#taskmanager-memory-task-off-heap-size)).
-
-<span class="label label-info">Note</span> You can also adjust the [framework
off-heap memory](mem_detail.html#framework-memory). This option is advanced
-and only recommended to be changed if you are sure that the Flink framework
needs more memory.
-
-Flink includes the *framework off-heap memory* and *task off-heap memory* into
the *direct memory* limit of the JVM,
-see also [JVM parameters](mem_detail.html#jvm-parameters).
-
-<span class="label label-info">Note</span> Although, native non-direct memory
usage can be accounted for as a part of the
-*framework off-heap memory* or *task off-heap memory*, it will result in a
higher JVM's *direct memory* limit in this case.
-
-<span class="label label-info">Note</span> The *network memory* is also part
of JVM *direct memory* but it is managed by Flink and guaranteed
-to never exceed its configured size. Therefore, resizing the *network memory*
will not help in this situation.
-
-See also [the detailed memory model](mem_detail.html).
+Another way to set up the memory is to configure the required internal
components of the *total Flink memory* which are
+specific to the concrete Flink process. Check how to configure them for task
executor [here](mem_setup_tm.html#configure-heap-and-managed-memory)
+and for job executor [here](mem_setup_jm.html#configure-jvm-heap).
+
+<span class="label label-info">Note</span> One of the three mentioned ways has
to be used to configure Flink’s memory
+(except for local execution), or the Flink startup will fail. This means that
one of the following option subsets,
+which do not have default values, have to be configured explicitly:
+
+| **for task executor:**
| **for job manager:**
|
+|
:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|
:--------------------------------------------------------------------------------
|
+|
[`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)
|
[`jobmanager.memory.flink.size`](../config.html#jobmanager-memory-flink-size)
|
+|
[`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
|
[`jobmanager.memory.process.size`](../config.html#jobmanager-memory-process-size)
|
+|
[`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size)
<br/> and
[`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size)
| [`jobmanager.memory.heap.size`](../config.html#jobmanager-memory-heap-size)
|
+{:.table-bordered}
+<br/>
+
+<span class="label label-info">Note</span> Explicitly configuring both *total
process memory* and *total Flink memory*
+is not recommended. It may lead to deployment failures due to potential memory
configuration conflicts. Additional
+configuration of other memory components also requires caution as it can
produce further configuration conflicts.
+
+See also [here](#capped-fractionated-components) how to configure the *JVM
Overhead*.
+
+## JVM Parameters
+
+Flink explicitly adds the following memory related JVM arguments while
starting its process, based on the configured
+or derived memory component sizes:
+
+| **JVM Arguments** | **Value for task
executor** | **Value for job executor** |
+| :---------------------------------------- |
:------------------------------------------------- |
:------------------------------------------------ |
+| *-Xmx* and *-Xms* | Framework + Task Heap Memory
| JVM Heap Memory |
+| *-XX:MaxDirectMemorySize* | Framework + Task Off-heap (*) +
Network Memory | Off-heap Memory (*) |
+| *-XX:MaxMetaspaceSize* | JVM Metaspace
| JVM Metaspace |
+{:.table-bordered}
+(*) Notice, that the native non-direct usage of memory in user code can be
also accounted for as a part of the off-heap memory.
+<br/><br/>
+
+Check also the detailed memory model for [task
executor](mem_setup_tm.html#detailed-memory-model) and
+[job executor](mem_setup_jm.html#detailed-configuration) to understand how to
configure the relevant components.
+
+## Capped Fractionated Components
+
+This section describes the configuration details of the following options
which can be a fraction of a certain
+[total memory](mem_setup.html#configure-total-memory):
Review comment:
BTW, do we really need the link for total memory?
##########
File path: docs/ops/memory/mem_setup.md
##########
@@ -30,104 +30,127 @@ Flink allows both high level and fine-grained tuning of
memory allocation within
* toc
{:toc}
-The further described memory configuration is applicable starting with the
release version *1.10*. If you upgrade Flink
-from earlier versions, check the [migration guide](mem_migration.html) because
many changes were introduced with the *1.10* release.
-
-<span class="label label-info">Note</span> This memory setup guide is relevant
<strong>only for task executors</strong>!
-Check [job manager related configuration
options](../config.html#jobmanager-heap-size) for the memory setup of job
manager.
+The further described memory configuration is applicable starting with the
release version *1.10* for task executors and
+*1.11* for job executors. If you upgrade Flink from earlier versions, check
the [migration guide](mem_migration.html)
+because many changes were introduced with the *1.10* and *1.11* releases.
## Configure Total Memory
-The *total process memory* of Flink JVM processes consists of memory consumed
by Flink application (*total Flink memory*)
-and by the JVM to run the process. The *total Flink memory* consumption
includes usage of JVM heap,
-*managed memory* (managed by Flink) and other direct (or native) memory.
+The *total process memory* of Flink JVM processes consists of memory consumed
by the Flink application (*total Flink memory*)
+and by the JVM to run the process. The *total Flink memory* consumption
includes usage of *JVM Heap* and *Off-heap*
+(*Direct* or *Native*) memory.
<center>
- <img src="{{ site.baseurl }}/fig/simple_mem_model.svg" width="300px"
alt="Simple memory model" usemap="#simple-mem-model">
+ <img src="{{ site.baseurl }}/fig/process_mem_model.svg" width="300px"
alt="Flink's process memory model" usemap="#process-mem-model">
</center>
<br />
-If you run Flink locally (e.g. from your IDE) without creating a cluster, then
only a subset of the memory configuration
-options are relevant, see also [local
execution](mem_detail.html#local-execution) for more details.
+The simplest way to setup memory in Flink is to configure either of the two
following options:
+
+| **Component** | **Option for task
executor** | **Option
for job executor** |
+| :------------------------------------ |
:----------------------------------------------------------------------------------
|
:--------------------------------------------------------------------------------
|
+| Total Flink memory |
[`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)
|
[`jobmanager.memory.flink.size`](../config.html#jobmanager-memory-flink-size)
|
+| Total process memory |
[`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
|
[`jobmanager.memory.process.size`](../config.html#jobmanager-memory-process-size)
|
+{:.table-bordered}
+<br/>
-Otherwise, the simplest way to setup memory in Flink is to configure either of
the two following options:
-* Total Flink memory
([`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size))
-* Total process memory
([`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size))
+See also notes for the execution mode of [task
executor](mem_setup_tm.html#local-execution) and
+[job executor](mem_setup_jm.html#notes-for-the-execution-mode).
The rest of the memory components will be adjusted automatically, based on
default values or additionally configured options.
-[Here](mem_detail.html) are more details about the other memory components.
+See also how to set up other components for [task executor](mem_setup_tm.html)
and [job executor](mem_setup_jm.html) memory.
-Configuring *total Flink memory* is better suited for standalone deployments
where you want to declare how much memory
-is given to Flink itself. The *total Flink memory* splits up into JVM heap,
[managed memory size](#managed-memory)
-and *direct memory*.
+Configuring *total Flink memory* is better suited for [standalone
deployments](../deployment/cluster_setup.html)
+where you want to declare how much memory is given to Flink itself. The *total
Flink memory* splits up into *JVM Heap*
+and *Off-heap* memory.
+See also [how to configure memory for standalone
deployments](mem_tuning.html#configure-memory-for-standalone-deployment).
If you configure *total process memory* you declare how much memory in total
should be assigned to the Flink *JVM process*.
For the containerized deployments it corresponds to the size of the requested
container, see also
[how to configure memory for
containers](mem_tuning.html#configure-memory-for-containers)
([Kubernetes](../deployment/kubernetes.html),
[Yarn](../deployment/yarn_setup.html) or [Mesos](../deployment/mesos.html)).
-Another way to setup the memory is to set [task
heap](#task-operator-heap-memory) and [managed memory](#managed-memory)
-([`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size)
and
[`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size)).
-This more fine-grained approach is described in more detail
[here](#configure-heap-and-managed-memory).
-
-<span class="label label-info">Note</span> One of the three mentioned ways has
to be used to configure Flink’s memory (except for local execution), or the
Flink startup will fail.
-This means that one of the following option subsets, which do not have default
values, have to be configured explicitly:
-*
[`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)
-*
[`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
-*
[`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size)
and
[`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size)
-
-<span class="label label-info">Note</span> Explicitly configuring both *total
process memory* and *total Flink memory* is not recommended.
-It may lead to deployment failures due to potential memory configuration
conflicts. Additional configuration
-of other memory components also requires caution as it can produce further
configuration conflicts.
-
-## Configure Heap and Managed Memory
-
-As mentioned before in [total memory description](#configure-total-memory),
another way to setup memory in Flink is
-to specify explicitly both [task heap](#task-operator-heap-memory) and
[managed memory](#managed-memory).
-It gives more control over the available JVM heap to Flink’s tasks and its
[managed memory](#managed-memory).
-
-The rest of the memory components will be adjusted automatically, based on
default values or additionally configured options.
-[Here](mem_detail.html) are more details about the other memory components.
-
-<span class="label label-info">Note</span> If you have configured the task
heap and managed memory explicitly, it is recommended to set neither
-*total process memory* nor *total Flink memory*. Otherwise, it may easily lead
to memory configuration conflicts.
-
-### Task (Operator) Heap Memory
-
-If you want to guarantee that a certain amount of JVM heap is available for
your user code, you can set the *task heap memory*
-explicitly
([`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size)).
-It will be added to the JVM heap size and will be dedicated to Flink’s
operators running the user code.
-
-### Managed Memory
-
-*Managed memory* is managed by Flink and is allocated as native memory
(off-heap). The following workloads use *managed memory*:
-* Streaming jobs can use it for [RocksDB state
backend](../state/state_backends.html#the-rocksdbstatebackend).
-* [Batch jobs](../../dev/batch) can use it for sorting, hash tables, caching
of intermediate results.
-
-The size of *managed memory* can be
-* either configured explicitly via
[`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size)
-* or computed as a fraction of *total Flink memory* via
[`taskmanager.memory.managed.fraction`](../config.html#taskmanager-memory-managed-fraction).
-
-*Size* will override *fraction*, if both are set.
-If neither *size* nor *fraction* is explicitly configured, the [default
fraction](../config.html#taskmanager-memory-managed-fraction) will be used.
-
-See also [how to configure memory for state
backends](mem_tuning.html#configure-memory-for-state-backends) and [batch
jobs](mem_tuning.html#configure-memory-for-batch-jobs).
-
-## Configure Off-Heap Memory (direct or native)
-
-The off-heap memory which is allocated by user code should be accounted for in
*task off-heap memory*
-([`taskmanager.memory.task.off-heap.size`](../config.html#taskmanager-memory-task-off-heap-size)).
-
-<span class="label label-info">Note</span> You can also adjust the [framework
off-heap memory](mem_detail.html#framework-memory). This option is advanced
-and only recommended to be changed if you are sure that the Flink framework
needs more memory.
-
-Flink includes the *framework off-heap memory* and *task off-heap memory* into
the *direct memory* limit of the JVM,
-see also [JVM parameters](mem_detail.html#jvm-parameters).
-
-<span class="label label-info">Note</span> Although, native non-direct memory
usage can be accounted for as a part of the
-*framework off-heap memory* or *task off-heap memory*, it will result in a
higher JVM's *direct memory* limit in this case.
-
-<span class="label label-info">Note</span> The *network memory* is also part
of JVM *direct memory* but it is managed by Flink and guaranteed
-to never exceed its configured size. Therefore, resizing the *network memory*
will not help in this situation.
-
-See also [the detailed memory model](mem_detail.html).
+Another way to set up the memory is to configure the required internal
components of the *total Flink memory* which are
+specific to the concrete Flink process. Check how to configure them for task
executor [here](mem_setup_tm.html#configure-heap-and-managed-memory)
+and for job executor [here](mem_setup_jm.html#configure-jvm-heap).
+
+<span class="label label-info">Note</span> One of the three mentioned ways has
to be used to configure Flink’s memory
+(except for local execution), or the Flink startup will fail. This means that
one of the following option subsets,
+which do not have default values, have to be configured explicitly:
+
+| **for task executor:**
| **for job manager:**
|
+|
:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|
:--------------------------------------------------------------------------------
|
+|
[`taskmanager.memory.flink.size`](../config.html#taskmanager-memory-flink-size)
|
[`jobmanager.memory.flink.size`](../config.html#jobmanager-memory-flink-size)
|
+|
[`taskmanager.memory.process.size`](../config.html#taskmanager-memory-process-size)
|
[`jobmanager.memory.process.size`](../config.html#jobmanager-memory-process-size)
|
+|
[`taskmanager.memory.task.heap.size`](../config.html#taskmanager-memory-task-heap-size)
<br/> and
[`taskmanager.memory.managed.size`](../config.html#taskmanager-memory-managed-size)
| [`jobmanager.memory.heap.size`](../config.html#jobmanager-memory-heap-size)
|
+{:.table-bordered}
+<br/>
+
+<span class="label label-info">Note</span> Explicitly configuring both *total
process memory* and *total Flink memory*
+is not recommended. It may lead to deployment failures due to potential memory
configuration conflicts. Additional
+configuration of other memory components also requires caution as it can
produce further configuration conflicts.
+
+See also [here](#capped-fractionated-components) how to configure the *JVM
Overhead*.
+
+## JVM Parameters
+
+Flink explicitly adds the following memory related JVM arguments while
starting its process, based on the configured
+or derived memory component sizes:
+
+| **JVM Arguments** | **Value for task
executor** | **Value for job executor** |
+| :---------------------------------------- |
:------------------------------------------------- |
:------------------------------------------------ |
+| *-Xmx* and *-Xms* | Framework + Task Heap Memory
| JVM Heap Memory |
+| *-XX:MaxDirectMemorySize* | Framework + Task Off-heap (*) +
Network Memory | Off-heap Memory (*) |
+| *-XX:MaxMetaspaceSize* | JVM Metaspace
| JVM Metaspace |
+{:.table-bordered}
+(*) Notice, that the native non-direct usage of memory in user code can be
also accounted for as a part of the off-heap memory.
+<br/><br/>
+
+Check also the detailed memory model for [task
executor](mem_setup_tm.html#detailed-memory-model) and
+[job executor](mem_setup_jm.html#detailed-configuration) to understand how to
configure the relevant components.
+
+## Capped Fractionated Components
+
+This section describes the configuration details of the following options
which can be a fraction of a certain
+[total memory](mem_setup.html#configure-total-memory):
Review comment:
```suggestion
This section describes the configuration details of the following options
which try to make a fraction of a certain
[total memory](mem_setup.html#configure-total-memory) while constrained by a
min-max range:
```
Managed memory can also be derived from a fraction, but it's not a capped
fractionated component because it doesn't have the min-max range.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
