[
https://issues.apache.org/jira/browse/FLINK-33424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17873770#comment-17873770
]
slankka edited comment on FLINK-33424 at 8/15/24 2:22 AM:
----------------------------------------------------------
Please be aware of that the method `registerSingleLocalResource` in file
[flink/flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java
at master · apache/flink
(github.com)|https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java#L1106]
has two similar code:
*commited by FLINK-18362, it is originally 'tmpConfigurationFile.toUri()'*
{code:java}
//Line: 1104
fileUploader.registerSingleLocalResource(
flinkConfigFileName,
new Path(tmpConfigurationFile.getAbsolutePath()),
"",
LocalResourceType.FILE,
true,
true); {code}
the accident merge commit is:
[FLINK-18362[yarn] Fix mistakenly merged commit 0e10fd5b8ee0 ·
apache/flink@a0227e2
(github.com)|https://github.com/apache/flink/commit/a0227e20430ee9eaff59464023de2385378f71ea#diff-02416e2d6ca99e1456f9c3949f3d7c2ac523d3fe25378620c09632e4aac34e4eL889]
[https://github.com/apache/flink/commit/a0227e20430ee9eaff59464023de2385378f71ea]
And if you check Line 1071 around, `tmpJobGraphFile.toURI()` is correct.
{code:java}
//Line: 1071
fileUploader.registerSingleLocalResource(
jobGraphFilename,
new Path(tmpJobGraphFile.toURI()),
"",
LocalResourceType.FILE,
true,
false); {code}
The problem is still exists and still not being MERGED.
was (Author: adrian z):
Please be aware of that `registerSingleLocalResource` in file
[flink/flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java
at master · apache/flink
(github.com)|https://github.com/apache/flink/blob/master/flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java#L1106]
has two similar code:
{code:java}
//Line: 1104
fileUploader.registerSingleLocalResource(
flinkConfigFileName,
new Path(tmpConfigurationFile.getAbsolutePath()), //Which
is commit by FLINK-18362, it is originally 'tmpConfigurationFile.toUri()'
"",
LocalResourceType.FILE,
true,
true); {code}
[[FLINK-18362][yarn] Fix mistakenly merged commit 0e10fd5b8ee0 ·
apache/flink@a0227e2
(github.com)|https://github.com/apache/flink/commit/a0227e20430ee9eaff59464023de2385378f71ea#diff-02416e2d6ca99e1456f9c3949f3d7c2ac523d3fe25378620c09632e4aac34e4eL889]
[https://github.com/apache/flink/commit/a0227e20430ee9eaff59464023de2385378f71ea]
{code:java}
//Line: 1071
fileUploader.registerSingleLocalResource(
jobGraphFilename,
new Path(tmpJobGraphFile.toURI()),
"",
LocalResourceType.FILE,
true,
false); {code}
This is still not correct and still not being MERGED.
> Resolved an issue in YarnClusterDescriptor where temporary files created
> locally by flink-conf.yaml are treated as remote files
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-33424
> URL: https://issues.apache.org/jira/browse/FLINK-33424
> Project: Flink
> Issue Type: Bug
> Components: Client / Job Submission
> Affects Versions: 1.17.1
> Reporter: zhengzhili
> Priority: Major
> Fix For: 1.19.0
>
>
> After creating the flink-conf.yaml temporary file in
> YarnClusterDescriptor#startAppMaster, Through YarnApplicationFileUploader#
> registerSingleLocalResource registered local files,
> {quote}fileUploader.registerSingleLocalResource(
> flinkConfigKey,
> new Path(tmpConfigurationFile.getAbsolutePath()),
> "",
> LocalResourceType.FILE,
> true,
> true);
> {quote}
>
> In this case, if (Utils.isRemotePath(resourcePath.toString())) is used to
> determine whether the file is remote based on the path.
> {quote}if (Utils.isRemotePath(resourcePath.toString())) {
> final FileStatus fileStatus = fileSystem.getFileStatus(resourcePath);
> LOG.debug("Using remote file {} to register local resource",
> fileStatus.getPath());
> final YarnLocalResourceDescriptor descriptor =
> YarnLocalResourceDescriptor.fromFileStatus(
> key, fileStatus, LocalResourceVisibility.APPLICATION, resourceType);
> addToEnvShipResourceList(whetherToAddToEnvShipResourceList, descriptor);
> localResources.put(key, descriptor.toLocalResource());
> return descriptor;
> }
> {quote}
>
> The incoming tmpConfigurationFile#getAbsolutePath makes Utils. IsRemotePath
> considered remote file
> When FileSystem is S3AFileSystem, the getFileStatus method displays an error
> indicating that the file does not exist
>
>
> Incoming parameters should be new Path (tmpConfigurationFile.toURI ()) to the
> new Path (tmpConfigurationFile. getAbsolutePath ())
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
