[
https://issues.apache.org/jira/browse/FLINK-20935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17268530#comment-17268530
]
Yang Wang commented on FLINK-20935:
-----------------------------------
I think you are right.
{{path.getFileSystem(yarnConfiguration).makeQualified(path)}} is not what we
want. At the very beginning, I mean to qualify the path(aka adding file://
protocol) in the {{YarnApplicationFileUploader#registerSingleLocalResource}} if
necessary, instead of ensuring all the caller to have a qualified path.
But after a little more consideration, I think let the caller do the decision
is more appropriate.
> can't write flink configuration to tmp file and add it to local resource in
> yarn session mode
> ---------------------------------------------------------------------------------------------
>
> Key: FLINK-20935
> URL: https://issues.apache.org/jira/browse/FLINK-20935
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
> Affects Versions: 1.12.0, 1.13.0
> Reporter: yuemeng
> Assignee: yuemeng
> Priority: Major
> Labels: pull-request-available
>
> In flink 1.12.0 or lastest version,when we execute command such as
> bin/yarn-session.sh -n 20 -jm 9096 -nm 4096 -st,the depoy will be failed with
> follow errors:
> {code}
> org.apache.flink.client.deployment.ClusterDeploymentException: Couldn't
> deploy Yarn session cluster
> at
> org.apache.flink.yarn.YarnClusterDescriptor.deploySessionCluster(YarnClusterDescriptor.java:411)
> at
> org.apache.flink.yarn.cli.FlinkYarnSessionCli.run(FlinkYarnSessionCli.java:498)
> at
> org.apache.flink.yarn.cli.FlinkYarnSessionCli.lambda$main$4(FlinkYarnSessionCli.java:730)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754)
> at
> org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
> at
> org.apache.flink.yarn.cli.FlinkYarnSessionCli.main(FlinkYarnSessionCli.java:730)
> Caused by: java.io.FileNotFoundException: File does not exist:
> /tmp/application_1573723355201_0036-flink-conf.yaml688141408443326132.tmp
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$22.doCall(DistributedFileSystem.java:1309)
> {code}
> when we called startAppMaster method in YarnClusterDescriptor,it will be try
> to write flink configuration to tmp file and add it to local resource. but
> the follow code will make the tmp file system as a distribute file system
> {code}
> // Upload the flink configuration
> // write out configuration file
> File tmpConfigurationFile = null;
> try {
> tmpConfigurationFile = File.createTempFile(appId +
> "-flink-conf.yaml", null);
> BootstrapTools.writeConfiguration(configuration,
> tmpConfigurationFile);
> String flinkConfigKey = "flink-conf.yaml";
> fileUploader.registerSingleLocalResource(
> flinkConfigKey,
> new
> Path(tmpConfigurationFile.getAbsolutePath()),
> "",
> LocalResourceType.FILE,
> true,
> true);
>
> classPathBuilder.append("flink-conf.yaml").append(File.pathSeparator);
> } finally {
> if (tmpConfigurationFile != null &&
> !tmpConfigurationFile.delete()) {
> LOG.warn("Fail to delete temporary file {}.",
> tmpConfigurationFile.toPath());
> }
> }
> {code}
> {code} tmpConfigurationFile.getAbsolutePath() {code} method will be return a
> path without file schema and the file system will be considered as a
> distribute file system
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
