[GitHub] flink pull request #6294: [FLINK-9013][docs] Document yarn.containers.vcores...
Github user zentol commented on a diff in the pull request:
https://github.com/apache/flink/pull/6294#discussion_r202968456
--- Diff:
flink-core/src/main/java/org/apache/flink/configuration/description/TextElement.java
---
@@ -53,6 +55,16 @@ public static TextElement text(String text) {
return new TextElement(text, Collections.emptyList());
}
+ /**
+* Tries to format the text as code.
+*
+* @return text element with applied formatting
+*/
+ public TextElement formatAsCode() {
--- End diff --
alternatively we could add an explicit `Code` `InlineElement`.
---
[GitHub] flink pull request #6294: [FLINK-9013][docs] Document yarn.containers.vcores...
Github user NicoK commented on a diff in the pull request:
https://github.com/apache/flink/pull/6294#discussion_r202720265
--- Diff:
flink-yarn/src/main/java/org/apache/flink/yarn/configuration/YarnConfigOptions.java
---
@@ -65,7 +65,11 @@
key("yarn.containers.vcores")
.defaultValue(-1)
.withDescription("The number of virtual cores (vcores) per YARN
container. By default, the number of vcores" +
- " is set to the number of slots per TaskManager, if
set, or to 1, otherwise.");
+ " is set to the number of slots per TaskManager, if
set, or to 1, otherwise. In order for this parameter " +
+ "to be used your cluster must have CPU scheduling
enabled. You can do this e.g. by setting the " +
+
"org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler or
enabling " +
+
"org.apache.hadoop.yarn.util.resource.DominantResourceCalculator for " +
+
"org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler");
--- End diff --
Also, please put the configuration parameters into some code environment,
e.g.
`org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler`,
for the config.html page.
---
[GitHub] flink pull request #6294: [FLINK-9013][docs] Document yarn.containers.vcores...
Github user NicoK commented on a diff in the pull request: https://github.com/apache/flink/pull/6294#discussion_r202720821 --- Diff: docs/ops/deployment/yarn_setup.md --- @@ -132,7 +132,7 @@ Stop the YARN session by stopping the unix process (using CTRL+C) or by entering Flink on YARN will only start all requested containers if enough resources are available on the cluster. Most YARN schedulers account for the requested memory of the containers, some account also for the number of vcores. By default, the number of vcores is equal to the processing slots (`-s`) argument. The `yarn.containers.vcores` allows overwriting the --- End diff -- I agree with @zentol - a link here (in addition) would be better to simply click your way through the docs ---
[GitHub] flink pull request #6294: [FLINK-9013][docs] Document yarn.containers.vcores...
Github user NicoK commented on a diff in the pull request:
https://github.com/apache/flink/pull/6294#discussion_r202719580
--- Diff:
flink-yarn/src/main/java/org/apache/flink/yarn/configuration/YarnConfigOptions.java
---
@@ -65,7 +65,11 @@
key("yarn.containers.vcores")
.defaultValue(-1)
.withDescription("The number of virtual cores (vcores) per YARN
container. By default, the number of vcores" +
- " is set to the number of slots per TaskManager, if
set, or to 1, otherwise.");
+ " is set to the number of slots per TaskManager, if
set, or to 1, otherwise. In order for this parameter " +
+ "to be used your cluster must have CPU scheduling
enabled. You can do this e.g. by setting the " +
+
"org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler or
enabling " +
+
"org.apache.hadoop.yarn.util.resource.DominantResourceCalculator for " +
+
"org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler");
--- End diff --
Have you actually tried - back when I created this issue, simply changing
to `DominantResourceCalculator` did not lead to the desired effect.
---
[GitHub] flink pull request #6294: [FLINK-9013][docs] Document yarn.containers.vcores...
Github user NicoK commented on a diff in the pull request:
https://github.com/apache/flink/pull/6294#discussion_r202722162
--- Diff: docs/ops/deployment/yarn_setup.md ---
@@ -132,7 +132,7 @@ Stop the YARN session by stopping the unix process
(using CTRL+C) or by entering
Flink on YARN will only start all requested containers if enough resources
are available on the cluster. Most YARN schedulers account for the requested
memory of the containers,
some account also for the number of vcores. By default, the number of
vcores is equal to the processing slots (`-s`) argument. The
`yarn.containers.vcores` allows overwriting the
-number of vcores with a custom value.
+number of vcores with a custom value. In order for this parameter to work
you should enable CPU scheduling in your cluster, see more [here]({{
site.baseurl }}/ops/config.html#yarn-containers-vcores)
--- End diff --
You should, however, never link on "here" - maybe put the link on "enable
CPU scheduling" instead?
---
[GitHub] flink pull request #6294: [FLINK-9013][docs] Document yarn.containers.vcores...
Github user dawidwys commented on a diff in the pull request: https://github.com/apache/flink/pull/6294#discussion_r201399926 --- Diff: docs/ops/deployment/yarn_setup.md --- @@ -132,7 +132,7 @@ Stop the YARN session by stopping the unix process (using CTRL+C) or by entering Flink on YARN will only start all requested containers if enough resources are available on the cluster. Most YARN schedulers account for the requested memory of the containers, some account also for the number of vcores. By default, the number of vcores is equal to the processing slots (`-s`) argument. The `yarn.containers.vcores` allows overwriting the --- End diff -- The way it is right now it implies there is more information on the CPU scheduling in the description of the parameter. ---
[GitHub] flink pull request #6294: [FLINK-9013][docs] Document yarn.containers.vcores...
Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/6294#discussion_r201398701 --- Diff: docs/ops/deployment/yarn_setup.md --- @@ -132,7 +132,7 @@ Stop the YARN session by stopping the unix process (using CTRL+C) or by entering Flink on YARN will only start all requested containers if enough resources are available on the cluster. Most YARN schedulers account for the requested memory of the containers, some account also for the number of vcores. By default, the number of vcores is equal to the processing slots (`-s`) argument. The `yarn.containers.vcores` allows overwriting the --- End diff -- why not replace "`yarn.containers.vcores`" with the link? ---
[GitHub] flink pull request #6294: [FLINK-9013][docs] Document yarn.containers.vcores...
Github user dawidwys commented on a diff in the pull request: https://github.com/apache/flink/pull/6294#discussion_r201394304 --- Diff: docs/ops/deployment/yarn_setup.md --- @@ -132,7 +132,10 @@ Stop the YARN session by stopping the unix process (using CTRL+C) or by entering Flink on YARN will only start all requested containers if enough resources are available on the cluster. Most YARN schedulers account for the requested memory of the containers, some account also for the number of vcores. By default, the number of vcores is equal to the processing slots (`-s`) argument. The `yarn.containers.vcores` allows overwriting the -number of vcores with a custom value. +number of vcores with a custom value. In order for this parameter to be used your cluster must have CPU scheduling enabled. You can do this e.g. by + + * setting the `org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler` or + * enabling `org.apache.hadoop.yarn.util.resource.DominantResourceCalculator` for `org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler` --- End diff -- Changed it as you suggested. ---
[GitHub] flink pull request #6294: [FLINK-9013][docs] Document yarn.containers.vcores...
Github user zentol commented on a diff in the pull request: https://github.com/apache/flink/pull/6294#discussion_r201329612 --- Diff: docs/ops/deployment/yarn_setup.md --- @@ -132,7 +132,10 @@ Stop the YARN session by stopping the unix process (using CTRL+C) or by entering Flink on YARN will only start all requested containers if enough resources are available on the cluster. Most YARN schedulers account for the requested memory of the containers, some account also for the number of vcores. By default, the number of vcores is equal to the processing slots (`-s`) argument. The `yarn.containers.vcores` allows overwriting the -number of vcores with a custom value. +number of vcores with a custom value. In order for this parameter to be used your cluster must have CPU scheduling enabled. You can do this e.g. by + + * setting the `org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler` or + * enabling `org.apache.hadoop.yarn.util.resource.DominantResourceCalculator` for `org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler` --- End diff -- I would suggest adding this to the options documentation, and then simply link to the configuration section. (you can link to individual options!) ---
[GitHub] flink pull request #6294: [FLINK-9013][docs] Document yarn.containers.vcores...
GitHub user dawidwys opened a pull request: https://github.com/apache/flink/pull/6294 [FLINK-9013][docs] Document yarn.containers.vcores only being effective whe⦠Added required configuration documentation on the cluster side for the parameter `yarn.containers.vcores` to take effect. You can merge this pull request into a Git repository by running: $ git pull https://github.com/dawidwys/flink FLINK-9013 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/6294.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #6294 commit 2ccd82efa2ab109758ed96301c381c68883d1025 Author: Dawid Wysakowicz Date: 2018-07-10T09:55:39Z [FLINK-9013] Document yarn.containers.vcores only being effective when adapting YARN config ---
