xintongsong commented on a change in pull request #9106: [FLINK-13184][yarn]
Support launching task executors with multi-thread on YARN.
URL: https://github.com/apache/flink/pull/9106#discussion_r313785156
##########
File path:
flink-yarn/src/main/java/org/apache/flink/yarn/configuration/YarnConfigOptions.java
##########
@@ -188,6 +188,14 @@
.defaultValue("")
.withDescription("A comma-separated list of tags to apply to
the Flink YARN application.");
+ /**
+ * The max number of threads for starting yarn containers in yarn
resource manager.
+ */
+ public static final ConfigOption<Integer> CONTAINER_LAUNCHER_NUM_MAX =
Review comment:
I think the benefit of having many threads starting containers mainly comes
from that Flink RM can concurrently start containers from different Yarn NMs.
Imagine a job with 1000s of containers on a Yarn cluster of 100s of NMs,
ideally Flink RM would be able to concurrently start containers from 100
different NMs, and averagely each NM only have about 10 containers to start.
The burden for each NM is affordable even with multiple Flink RMs, because it
is not a common case that all the Flink RMs start massive of containers at the
same time.
I can try provide some benchmarks. This is not a rare case in our production.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
