yuruguo commented on a change in pull request #14538:
URL: https://github.com/apache/flink/pull/14538#discussion_r552454921
##########
File path:
flink-yarn/src/main/java/org/apache/flink/yarn/YarnApplicationFileUploader.java
##########
@@ -163,8 +164,15 @@ YarnLocalResourceDescriptor registerSingleLocalResource(
addToRemotePaths(whetherToAddToRemotePaths, resourcePath);
if (Utils.isRemotePath(resourcePath.toString())) {
- final FileStatus fileStatus =
fileSystem.getFileStatus(resourcePath);
- LOG.debug("Using remote file {} to register local resource",
fileStatus.getPath());
+ final FileSystem srcFs =
resourcePath.getFileSystem(fileSystem.getConf());
+ final FileStatus resourceFileStatus =
srcFs.getFileStatus(resourcePath);
+ LOG.debug("Using remote file {} to register local resource",
resourceFileStatus.getPath());
+ final Path applicationDir = getApplicationDirPath(homeDir,
applicationId);
+ final String suffix = (relativeDstPath.isEmpty() ? "" :
relativeDstPath + "/") + resourcePath.getName();
+ final Path dst = new Path(applicationDir, suffix);
+ LOG.debug("Copying from {} to {} with replication factor {}",
resourcePath, dst, fileReplication);
+ FileUtil.copy(srcFs, resourcePath, fileSystem, dst, false,
fileSystem.getConf());
+ final FileStatus fileStatus = fileSystem.getFileStatus(dst);
Review comment:
For tests, existing case
`testRegisterMultipleLocalResourcesWithRemoteFiles` can cover it in class
`org.apache.flink.yarn.YarnFileStageTest`, and I can add some.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
