tillrohrmann commented on a change in pull request #11353: [FLINK-16438][yarn]
Make YarnResourceManager starts workers using WorkerResourceSpec requested by
SlotManager
URL: https://github.com/apache/flink/pull/11353#discussion_r404128962
##########
File path:
flink-yarn/src/main/java/org/apache/flink/yarn/configuration/YarnConfigOptionsInternal.java
##########
@@ -34,4 +37,24 @@
.stringType()
.noDefaultValue()
.withDescription("**DO NOT USE** The
location of the log config file, e.g. the path to your log4j.properties for
log4j.");
+
+ /**
+ * **DO NO USE** Whether {@link YarnResourceManager} should match the
vcores of allocated containers with those requested.
+ *
+ * <p>By default, Yarn ignores vcores in the container requests, and
always allocate 1 vcore for each container.
+ * Iff 'yarn.scheduler.capacity.resource-calculator' is set to
'DominantResourceCalculator' for Yarn, will it
+ * allocate container vcores as requested. Unfortunately, this
configuration option is dedicated for Yarn Scheduler,
+ * and is only accessible to applications in Hadoop 2.6+.
+ *
+ * <p>ATM, it should be fine to not match vcores, because with the
current {@link SlotManagerImpl} all the TM
+ * containers should have the same resources.
+ *
+ * <p>If later we add another {@link SlotManager} implementation that
may have TMs with different resources, we can
+ * switch this option on only for the new SM, and the new SM can also
be available on Hadoop 2.6+ only.
+ */
+ public static final ConfigOption<Boolean> MATCH_CONTAINER_VCORES =
+
key("$internal.yarn.resourcemanager.enable-vcore-matching")
+ .booleanType()
+ .defaultValue(false)
+ .withDescription("**DO NOT USE**
Whether YarnResourceManager should match the container vcores.");
Review comment:
Ok, thanks for the clarification. Automatically configuring this flag with
Hadoop >= 2.6 sounds like a good idea.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
