[
https://issues.apache.org/jira/browse/FLINK-33424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhengzhili updated FLINK-33424:
-------------------------------
Description:
After creating the flink-conf.yaml temporary file in
YarnClusterDescriptor#startAppMaster, Through YarnApplicationFileUploader#
registerSingleLocalResource registered local files,
{quote}fileUploader.registerSingleLocalResource(
flinkConfigKey,
new Path(tmpConfigurationFile.getAbsolutePath()),
"",
LocalResourceType.FILE,
true,
true);
{quote}
In this case, if (Utils.isRemotePath(resourcePath.toString())) is used to
determine whether the file is remote based on the path.
{quote}if (Utils.isRemotePath(resourcePath.toString())) {
final FileStatus fileStatus = fileSystem.getFileStatus(resourcePath);
LOG.debug("Using remote file {} to register local resource",
fileStatus.getPath());
final YarnLocalResourceDescriptor descriptor =
YarnLocalResourceDescriptor.fromFileStatus(
key, fileStatus, LocalResourceVisibility.APPLICATION, resourceType);
addToEnvShipResourceList(whetherToAddToEnvShipResourceList, descriptor);
localResources.put(key, descriptor.toLocalResource());
return descriptor;
}
{quote}
The incoming tmpConfigurationFile#getAbsolutePath makes Utils. IsRemotePath
considered remote file
When FileSystem is S3AFileSystem, the getFileStatus method displays an error
indicating that the file does not exist
Incoming parameters should be new Path (tmpConfigurationFile.toURI ()) to the
new Path (tmpConfigurationFile. getAbsolutePath ())
was:
After creating the flink-conf.yaml temporary file in
YarnClusterDescriptor#startAppMaster, Through YarnApplicationFileUploader#
registerSingleLocalResource registered local files,
{quote}
fileUploader.registerSingleLocalResource(
flinkConfigKey,
new Path(tmpConfigurationFile.getAbsolutePath()),
"",
LocalResourceType.FILE,
true,
true);{quote}
In this case, if (Utils.isRemotePath(resourcePath.toString())) is used to
determine whether the file is remote based on the path.
{quote}if (Utils.isRemotePath(resourcePath.toString())) {
final FileStatus fileStatus = fileSystem.getFileStatus(resourcePath);
LOG.debug("Using remote file {} to register local resource",
fileStatus.getPath());
final YarnLocalResourceDescriptor descriptor =
YarnLocalResourceDescriptor.fromFileStatus(
key, fileStatus, LocalResourceVisibility.APPLICATION, resourceType);
addToEnvShipResourceList(whetherToAddToEnvShipResourceList, descriptor);
localResources.put(key, descriptor.toLocalResource());
return descriptor;
}{quote}
The incoming tmpConfigurationFile#getAbsolutePath makes Utils. IsRemotePath
considered remote file
When FileSystem is S3AFileSystem, the getFileStatus method displays an error
indicating that the file does not exist
Incoming parameters should be new Path (tmpConfigurationFile toURI ()) to the
new Path (tmpConfigurationFile. GetAbsolutePath ())
> Resolved an issue in YarnClusterDescriptor where temporary files created
> locally by flink-conf.yaml are treated as remote files
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: FLINK-33424
> URL: https://issues.apache.org/jira/browse/FLINK-33424
> Project: Flink
> Issue Type: Bug
> Components: Client / Job Submission
> Affects Versions: 1.17.1
> Reporter: zhengzhili
> Priority: Major
> Fix For: 1.19.0
>
>
> After creating the flink-conf.yaml temporary file in
> YarnClusterDescriptor#startAppMaster, Through YarnApplicationFileUploader#
> registerSingleLocalResource registered local files,
> {quote}fileUploader.registerSingleLocalResource(
> flinkConfigKey,
> new Path(tmpConfigurationFile.getAbsolutePath()),
> "",
> LocalResourceType.FILE,
> true,
> true);
> {quote}
>
> In this case, if (Utils.isRemotePath(resourcePath.toString())) is used to
> determine whether the file is remote based on the path.
> {quote}if (Utils.isRemotePath(resourcePath.toString())) {
> final FileStatus fileStatus = fileSystem.getFileStatus(resourcePath);
> LOG.debug("Using remote file {} to register local resource",
> fileStatus.getPath());
> final YarnLocalResourceDescriptor descriptor =
> YarnLocalResourceDescriptor.fromFileStatus(
> key, fileStatus, LocalResourceVisibility.APPLICATION, resourceType);
> addToEnvShipResourceList(whetherToAddToEnvShipResourceList, descriptor);
> localResources.put(key, descriptor.toLocalResource());
> return descriptor;
> }
> {quote}
>
> The incoming tmpConfigurationFile#getAbsolutePath makes Utils. IsRemotePath
> considered remote file
> When FileSystem is S3AFileSystem, the getFileStatus method displays an error
> indicating that the file does not exist
>
>
> Incoming parameters should be new Path (tmpConfigurationFile.toURI ()) to the
> new Path (tmpConfigurationFile. getAbsolutePath ())
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
