tillrohrmann commented on a change in pull request #14079:
URL: https://github.com/apache/flink/pull/14079#discussion_r524279099
##########
File path: docs/ops/memory/mem_setup_tm.md
##########
@@ -87,6 +88,26 @@ If neither *size* nor *fraction* is explicitly configured,
the [default fraction
See also [how to configure memory for state
backends](mem_tuning.html#configure-memory-for-state-backends) and [batch
jobs](mem_tuning.html#configure-memory-for-batch-jobs).
+#### Consumer Weights
+
+If your job contains multiple types of managed memory consumers, you can also
control how managed memory should be shared across these types.
+The configuration option [`taskmanager.memory.managed.consumer-weights`]({%
link ops/config.md %}#taskmanager-memory-managed-consumer-weights) allows you
to set a weight for each type, to which Flink will reserve managed memory
proportionally.
+Valid consumer types are:
+* `DATAPROC`: for RocksDB state backend in streaming and built-in algorithms
in batch.
+* `PYTHON`: for python processes.
Review comment:
```suggestion
* `PYTHON`: for Python processes.
```
##########
File path: docs/_includes/generated/task_manager_memory_configuration.html
##########
@@ -50,6 +50,12 @@
<td>MemorySize</td>
<td>Min JVM Overhead size for the TaskExecutors. This is off-heap
memory reserved for JVM overhead, such as thread stack space, compile cache,
etc. This includes native memory but not direct memory, and will not be counted
when Flink calculates JVM max direct memory size parameter. The size of JVM
Overhead is derived to make up the configured fraction of the Total Process
Memory. If the derived size is less/greater than the configured min/max size,
the min/max size will be used. The exact size of JVM Overhead can be explicitly
specified by setting the min/max size to the same value.</td>
</tr>
+ <tr>
+ <td><h5>taskmanager.memory.managed.consumer-weights</h5></td>
+ <td style="word-wrap: break-word;">DATAPROC:70,PYTHON:30</td>
+ <td>Map</td>
+ <td>Managed memory weights for different kinds of consumers. A
slot’s managed memory is shared by all kinds of consumers it contains,
proportionally to the kinds’ weights and regardless of the number of consumers
from each kind. Currently supported kinds of consumers are DATAPROC (for
RocksDB state backend in streaming and built-in algorithms in batch) and PYTHON
(for python processes).</td>
Review comment:
```suggestion
<td>Managed memory weights for different kinds of consumers. A
slot’s managed memory is shared by all kinds of consumers it contains,
proportionally to the kinds’ weights and regardless of the number of consumers
from each kind. Currently supported kinds of consumers are DATAPROC (for
RocksDB state backend in streaming and built-in algorithms in batch) and PYTHON
(for Python processes).</td>
```
##########
File path: docs/ops/memory/mem_setup_tm.md
##########
@@ -87,6 +88,26 @@ If neither *size* nor *fraction* is explicitly configured,
the [default fraction
See also [how to configure memory for state
backends](mem_tuning.html#configure-memory-for-state-backends) and [batch
jobs](mem_tuning.html#configure-memory-for-batch-jobs).
+#### Consumer Weights
+
+If your job contains multiple types of managed memory consumers, you can also
control how managed memory should be shared across these types.
+The configuration option [`taskmanager.memory.managed.consumer-weights`]({%
link ops/config.md %}#taskmanager-memory-managed-consumer-weights) allows you
to set a weight for each type, to which Flink will reserve managed memory
proportionally.
+Valid consumer types are:
+* `DATAPROC`: for RocksDB state backend in streaming and built-in algorithms
in batch.
+* `PYTHON`: for python processes.
+
+E.g. if a streaming job uses both RocksDB state backend and Python UDFs, and
the consumer weights are configured as `DATAPROC:70,PYTHON:30`, Flink will
reserve `70%` of the total managed memory for RocksDB state backend and `30%`
for Python processes.
+
+<span class="label label-info">Note</span>
+For each type, Flink reserves managed memory only if the job contains managed
memory consumer of that type.
+E.g, if a streaming job uses heap state backend and Python UDFs, and the
consumer weights are configured as `DATAPROC:70,PYTHON:30`, Flink will use all
of its managed memory for Python processes, because heap state backend does not
use managed memory.
Review comment:
```suggestion
E.g, if a streaming job uses the heap state backend and Python UDFs, and the
consumer weights are configured as `DATAPROC:70,PYTHON:30`, Flink will use all
of its managed memory for Python processes, because the heap state backend does
not use managed memory.
```
##########
File path: docs/ops/memory/mem_setup_tm.md
##########
@@ -87,6 +88,26 @@ If neither *size* nor *fraction* is explicitly configured,
the [default fraction
See also [how to configure memory for state
backends](mem_tuning.html#configure-memory-for-state-backends) and [batch
jobs](mem_tuning.html#configure-memory-for-batch-jobs).
+#### Consumer Weights
+
+If your job contains multiple types of managed memory consumers, you can also
control how managed memory should be shared across these types.
+The configuration option [`taskmanager.memory.managed.consumer-weights`]({%
link ops/config.md %}#taskmanager-memory-managed-consumer-weights) allows you
to set a weight for each type, to which Flink will reserve managed memory
proportionally.
+Valid consumer types are:
+* `DATAPROC`: for RocksDB state backend in streaming and built-in algorithms
in batch.
+* `PYTHON`: for python processes.
+
+E.g. if a streaming job uses both RocksDB state backend and Python UDFs, and
the consumer weights are configured as `DATAPROC:70,PYTHON:30`, Flink will
reserve `70%` of the total managed memory for RocksDB state backend and `30%`
for Python processes.
+
+<span class="label label-info">Note</span>
+For each type, Flink reserves managed memory only if the job contains managed
memory consumer of that type.
Review comment:
```suggestion
For each type, Flink reserves managed memory only if the job contains
managed memory consumers of that type.
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
