dawidwys closed pull request #6444: [FLINK-9933] Simplify taskmanager memory
default values
URL: https://github.com/apache/flink/pull/6444
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/docs/_includes/generated/task_manager_configuration.html
b/docs/_includes/generated/task_manager_configuration.html
index c18c5d4c36d..16c5d98f8be 100644
--- a/docs/_includes/generated/task_manager_configuration.html
+++ b/docs/_includes/generated/task_manager_configuration.html
@@ -84,7 +84,7 @@
</tr>
<tr>
<td><h5>taskmanager.memory.segment-size</h5></td>
- <td style="word-wrap: break-word;">"32768"</td>
+ <td style="word-wrap: break-word;">"32kb"</td>
<td>Size of memory buffers used by the network stack and the
memory manager.</td>
</tr>
<tr>
@@ -114,12 +114,12 @@
</tr>
<tr>
<td><h5>taskmanager.network.memory.max</h5></td>
- <td style="word-wrap: break-word;">"1073741824"</td>
+ <td style="word-wrap: break-word;">"1gb"</td>
<td>Maximum memory size for network buffers.</td>
</tr>
<tr>
<td><h5>taskmanager.network.memory.min</h5></td>
- <td style="word-wrap: break-word;">"67108864"</td>
+ <td style="word-wrap: break-word;">"64mb"</td>
<td>Minimum memory size for network buffers.</td>
</tr>
<tr>
diff --git a/docs/ops/config.md b/docs/ops/config.md
index fd0df0c2370..d370e56d695 100644
--- a/docs/ops/config.md
+++ b/docs/ops/config.md
@@ -211,10 +211,10 @@ Previously, the number of network buffers was set
manually which became a quite
network buffers with the following configuration parameters:
- `taskmanager.network.memory.fraction`: Fraction of JVM memory to use for
network buffers (DEFAULT: 0.1),
-- `taskmanager.network.memory.min`: Minimum memory size for network buffers in
bytes (DEFAULT: 64 MB),
-- `taskmanager.network.memory.max`: Maximum memory size for network buffers in
bytes (DEFAULT: 1 GB), and
+- `taskmanager.network.memory.min`: Minimum memory size for network buffers
(DEFAULT: 64MB),
+- `taskmanager.network.memory.max`: Maximum memory size for network buffers
(DEFAULT: 1GB), and
- `taskmanager.memory.segment-size`: Size of memory buffers used by the memory
manager and the
-network stack in bytes (DEFAULT: 32768 (= 32 KiBytes)).
+network stack in bytes (DEFAULT: 32KB).
#### Setting the Number of Network Buffers directly
diff --git
a/flink-core/src/main/java/org/apache/flink/configuration/TaskManagerOptions.java
b/flink-core/src/main/java/org/apache/flink/configuration/TaskManagerOptions.java
index be76cf8c4db..98fa47f6697 100644
---
a/flink-core/src/main/java/org/apache/flink/configuration/TaskManagerOptions.java
+++
b/flink-core/src/main/java/org/apache/flink/configuration/TaskManagerOptions.java
@@ -190,11 +190,11 @@
//
------------------------------------------------------------------------
/**
- * Size of memory buffers used by the network stack and the memory
manager (in bytes).
+ * Size of memory buffers used by the network stack and the memory
manager.
*/
public static final ConfigOption<String> MEMORY_SEGMENT_SIZE =
key("taskmanager.memory.segment-size")
- .defaultValue("32768")
+ .defaultValue("32kb")
.withDescription("Size of memory buffers used by the
network stack and the memory manager.");
/**
@@ -268,19 +268,19 @@
"` and \"taskmanager.network.memory.max\" may
override this fraction.");
/**
- * Minimum memory size for network buffers (in bytes).
+ * Minimum memory size for network buffers.
*/
public static final ConfigOption<String> NETWORK_BUFFERS_MEMORY_MIN =
key("taskmanager.network.memory.min")
- .defaultValue(String.valueOf(64L << 20)) // 64 MB
+ .defaultValue("64mb")
.withDescription("Minimum memory size for network
buffers.");
/**
- * Maximum memory size for network buffers (in bytes).
+ * Maximum memory size for network buffers.
*/
public static final ConfigOption<String> NETWORK_BUFFERS_MEMORY_MAX =
key("taskmanager.network.memory.max")
- .defaultValue(String.valueOf(1024L << 20)) // 1 GB
+ .defaultValue("1gb")
.withDescription("Maximum memory size for network
buffers.");
/**
diff --git a/flink-dist/src/main/resources/flink-conf.yaml
b/flink-dist/src/main/resources/flink-conf.yaml
index 62ed13dd81a..f4ec6a7e424 100644
--- a/flink-dist/src/main/resources/flink-conf.yaml
+++ b/flink-dist/src/main/resources/flink-conf.yaml
@@ -184,8 +184,8 @@ rest.port: 8081
# of network buffers" error. The default min is 64MB, teh default max is 1GB.
#
# taskmanager.network.memory.fraction: 0.1
-# taskmanager.network.memory.min: 67108864
-# taskmanager.network.memory.max: 1073741824
+# taskmanager.network.memory.min: 64mb
+# taskmanager.network.memory.max: 1gb
#==============================================================================
# Flink Cluster Security Configuration
diff --git
a/flink-end-to-end-tests/test-scripts/test_high_parallelism_iterations.sh
b/flink-end-to-end-tests/test-scripts/test_high_parallelism_iterations.sh
index 5a06266da64..3b684b664ba 100755
--- a/flink-end-to-end-tests/test-scripts/test_high_parallelism_iterations.sh
+++ b/flink-end-to-end-tests/test-scripts/test_high_parallelism_iterations.sh
@@ -30,9 +30,9 @@ backup_config
set_conf "taskmanager.heap.mb" "52" # 52Mb x 100 TMs = 5Gb total heap
set_conf "taskmanager.memory.size" "8" # 8Mb
-set_conf "taskmanager.network.memory.min" "8388608" # 8Mb
-set_conf "taskmanager.network.memory.max" "8388608" # 8Mb
-set_conf "taskmanager.memory.segment-size" "8192" # 8Kb
+set_conf "taskmanager.network.memory.min" "8mb"
+set_conf "taskmanager.network.memory.max" "8mb"
+set_conf "taskmanager.memory.segment-size" "8kb"
set_conf "taskmanager.network.netty.server.numThreads" "1"
set_conf "taskmanager.network.netty.client.numThreads" "1"
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
