[jira] [Comment Edited] (FLINK-13694) Refactor Flink on YARN configuration with relavent overlay classes
[ https://issues.apache.org/jira/browse/FLINK-13694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907089#comment-16907089 ] vinoyang edited comment on FLINK-13694 at 8/14/19 9:42 AM: --- Our platform needs to provide a centralized submitter. We need to provide a customized env variable {{HADOOP_USER_NAME}}, it is hardcoded in {{ApplicationYarnClusterDescriptor}} : {code:java} appMasterEnv.put(YarnConfigKeys.ENV_HADOOP_USER_NAME, UserGroupInformation.getCurrentUser().getUserName()); {code} I prepared to give a config option. However, I found there is a better choice: we can use {{HadoopUserOverlay}}. I found {{ContainerOverlay}} provides a good abstract, but it only been used in Flink on Mesos, while it was defined in the flink-runtime module. IMO, we either use a unified abstract or move {{ContainerOverlay}} into Flink on Mesos module. was (Author: yanghua): Our platform needs to provide a centralized submitter. We need to provide a customized env variable {{HADOOP_USER_NAME}}, it is hardcoded in {{ApplicationYarnClusterDescriptor}} : {code:java} appMasterEnv.put(YarnConfigKeys.ENV_HADOOP_USER_NAME, UserGroupInformation.getCurrentUser().getUserName()); {code} I prepared to give a config option. However, I found there is a better choice: we can use {{HadoopUserOverlay}}. I found {{ContainerOverlay}} provides a good abstract, but it only been used in Flink on Mesos, while it was defined in the flink-runtime module. IMO, we either use a unified abstract or move it into Flink on Mesos module. > Refactor Flink on YARN configuration with relavent overlay classes > -- > > Key: FLINK-13694 > URL: https://issues.apache.org/jira/browse/FLINK-13694 > Project: Flink > Issue Type: Improvement > Components: Deployment / YARN >Reporter: vinoyang >Priority: Major > > There is an abstract layer beyond special deployment named > {{ContainerOverlay}}. > However, currently, only Flink on Mesos used this mechanism. Flink on YARN > still used some trivial configuration items in > {{AbstractYarnClusterDescriptor}}. IMO we need to use container overlay to > refactor Flink on YARN related implementation. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Comment Edited] (FLINK-13694) Refactor Flink on YARN configuration with relavent overlay classes
[ https://issues.apache.org/jira/browse/FLINK-13694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907089#comment-16907089 ] vinoyang edited comment on FLINK-13694 at 8/14/19 9:41 AM: --- Our platform needs to provide a centralized submitter. We need to provide a customized env variable {{HADOOP_USER_NAME}}, it is hardcoded in {{ApplicationYarnClusterDescriptor}} : {code:java} appMasterEnv.put(YarnConfigKeys.ENV_HADOOP_USER_NAME, UserGroupInformation.getCurrentUser().getUserName()); {code} I prepared to give a config option. However, I found there is a better choice: we can use {{HadoopUserOverlay}}. I found {{ContainerOverlay}} provides a good abstract, but it only been used in Flink on Mesos, while it was defined in the flink-runtime module. IMO, we either use a unified abstract or move it into Flink on Mesos module. was (Author: yanghua): We need to provide a customized env variable {{HADOOP_USER_NAME}}, it is hardcoded in {{ApplicationYarnClusterDescriptor}} : {code:java} appMasterEnv.put(YarnConfigKeys.ENV_HADOOP_USER_NAME, UserGroupInformation.getCurrentUser().getUserName()); {code} I prepared to give a config option. However, I found there is a better choice: we can use {{HadoopUserOverlay}}. I found {{ContainerOverlay}} provides a good abstract, but it only been used in Flink on Mesos, while it was defined in the flink-runtime module. > Refactor Flink on YARN configuration with relavent overlay classes > -- > > Key: FLINK-13694 > URL: https://issues.apache.org/jira/browse/FLINK-13694 > Project: Flink > Issue Type: Improvement > Components: Deployment / YARN >Reporter: vinoyang >Priority: Major > > There is an abstract layer beyond special deployment named > {{ContainerOverlay}}. > However, currently, only Flink on Mesos used this mechanism. Flink on YARN > still used some trivial configuration items in > {{AbstractYarnClusterDescriptor}}. IMO we need to use container overlay to > refactor Flink on YARN related implementation. -- This message was sent by Atlassian JIRA (v7.6.14#76016)