[jira] [Commented] (FLINK-33424) Resolved an issue in YarnClusterDescriptor where temporary files created locally by flink-conf.yaml are treated as remote files

[slankka (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-33424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841454#comment-17841454
 ] 

slankka commented on FLINK-33424:
-

[[FLINK-29797][flink-yarn] Fix fs.default-scheme will accidentally cau… by 
slankka · Pull Request #24733 · apache/flink 
(github.com)|https://github.com/apache/flink/pull/24733] And please assign to 
me.

> Resolved an issue in YarnClusterDescriptor where temporary files created 
> locally by flink-conf.yaml are treated as remote files
> ---
>
> Key: FLINK-33424
> URL: https://issues.apache.org/jira/browse/FLINK-33424
> Project: Flink
>  Issue Type: Bug
>  Components: Client / Job Submission
>Affects Versions: 1.17.1
>Reporter: zhengzhili
>Priority: Major
> Fix For: 1.19.0
>
>
> After creating the flink-conf.yaml temporary file in 
> YarnClusterDescriptor#startAppMaster, Through YarnApplicationFileUploader# 
> registerSingleLocalResource registered local files,
> {quote}fileUploader.registerSingleLocalResource(
> flinkConfigKey,
> new Path(tmpConfigurationFile.getAbsolutePath()),
> "",
> LocalResourceType.FILE,
> true,
> true);
> {quote}
>  
> In this case, if (Utils.isRemotePath(resourcePath.toString())) is used to 
> determine whether the file is remote based on the path.
> {quote}if (Utils.isRemotePath(resourcePath.toString())) {
> final FileStatus fileStatus = fileSystem.getFileStatus(resourcePath);
> LOG.debug("Using remote file {} to register local resource", 
> fileStatus.getPath());
> final YarnLocalResourceDescriptor descriptor =
> YarnLocalResourceDescriptor.fromFileStatus(
> key, fileStatus, LocalResourceVisibility.APPLICATION, resourceType);
> addToEnvShipResourceList(whetherToAddToEnvShipResourceList, descriptor);
> localResources.put(key, descriptor.toLocalResource());
> return descriptor;
> }
> {quote}
>  
> The incoming tmpConfigurationFile#getAbsolutePath makes Utils. IsRemotePath 
> considered remote file
> When FileSystem is S3AFileSystem, the getFileStatus method displays an error 
> indicating that the file does not exist
>  
>  
> Incoming parameters should be new Path (tmpConfigurationFile.toURI ()) to the 
> new Path (tmpConfigurationFile. getAbsolutePath ())



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (FLINK-29797) can't run a job on yarn, if set fs.default-scheme

[slankka (Jira)]

[ 
https://issues.apache.org/jira/browse/FLINK-29797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17841455#comment-17841455
 ] 

slankka commented on FLINK-29797:
-

Please assign to me... 

> can't run a job on yarn, if set fs.default-scheme
> -
>
> Key: FLINK-29797
> URL: https://issues.apache.org/jira/browse/FLINK-29797
> Project: Flink
>  Issue Type: Bug
>  Components: Deployment / YARN
>Affects Versions: 1.16.0
>Reporter: wangmenglong
>Priority: Major
>  Labels: pull-request-available
> Attachments: issue.png
>
>
> If I set the value of *fs.default-scheme* to other scheme and run a job on 
> yarn.
> I got this exception:
> {code:java}
> Caused by: java.io.FileNotFoundException: File does not exist: 
> /tmp/application_1667097340114_0001-flink-conf.yaml9110057305807570477.tmp
>         at 
> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1128)
>         at 
> org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1120)
>         at 
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>         at 
> org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1120)
>         at 
> org.apache.flink.yarn.YarnApplicationFileUploader.registerSingleLocalResource(YarnApplicationFileUploader.java:168)
>         at 
> org.apache.flink.yarn.YarnClusterDescriptor.startAppMaster(YarnClusterDescriptor.java:1047)
>         at 
> org.apache.flink.yarn.YarnClusterDescriptor.deployInternal(YarnClusterDescriptor.java:623)
>         at 
> org.apache.flink.yarn.YarnClusterDescriptor.deployJobCluster(YarnClusterDescriptor.java:490)
>         ... 24 more {code}
> I think the cause of this problem may be 
> *tmpConfigurationFile.getAbsolutePath() !issue.png!*
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)