[
https://issues.apache.org/jira/browse/FLINK-5791?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15880753#comment-15880753
]
ASF GitHub Bot commented on FLINK-5791:
---------------------------------------
Github user tillrohrmann commented on a diff in the pull request:
https://github.com/apache/flink/pull/3304#discussion_r102748761
--- Diff:
flink-yarn/src/main/java/org/apache/flink/yarn/YarnResourceManager.java ---
@@ -342,6 +346,14 @@ private ContainerLaunchContext
createTaskExecutorLaunchContext(Resource resource
FiniteDuration teRegistrationTimeout = new
FiniteDuration(timeout, TimeUnit.SECONDS);
final Configuration taskManagerConfig =
BootstrapTools.generateTaskManagerConfiguration(
flinkConfig, "", 0, 1, teRegistrationTimeout);
+ // Add resource profile of slots to task executor config.
+ // For yarn, all slots in a task executor have same resource
profile
+ ByteArrayOutputStream output = new ByteArrayOutputStream();
+ ObjectOutputStream rpOutput = new ObjectOutputStream(output);
+
rpOutput.writeObject(getResourceProfile(priority.getPriority()));
+ rpOutput.close();
+
taskManagerConfig.setString(ConfigConstants.TASK_MANAGER_RESOURCE_PROFILE_KEY,
+ new
String(Base64.encodeBase64(output.toByteArray())));
--- End diff --
I think we shouldn't base64 encode the resource profile into the
`taskManagerConfig`. Instead use `InstantiationUtil.writeObjectToConfig` to
write serialized data to the configuration.
But I'm a little bit torn apart here because so far we used to transfer
this kind of information via the environment variables. Maybe @rmetzger can
chime in to say what the most idiomatic way to transfer TM data would be.
> Resource should be strictly matched when allocating for yarn
> ------------------------------------------------------------
>
> Key: FLINK-5791
> URL: https://issues.apache.org/jira/browse/FLINK-5791
> Project: Flink
> Issue Type: Improvement
> Components: YARN
> Reporter: shuai.xu
> Assignee: shuai.xu
> Labels: flip-6
>
> In flip6, for yarn mode, resource should be assigned as requested to avoid
> resource wasting and OOM.
> 1. YarnResourceManager will request container according to ResourceProfile
> in slot request form JM.
> 2. RM will pass the ResourceProfile to TM for initializing its slots.
> 3. RM should match the slots offered by TM with SlotRequest from JM strictly.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
