yuruguo commented on a change in pull request #14538:
URL: https://github.com/apache/flink/pull/14538#discussion_r552450129
##########
File path:
flink-yarn/src/main/java/org/apache/flink/yarn/YarnApplicationFileUploader.java
##########
@@ -163,8 +164,15 @@ YarnLocalResourceDescriptor registerSingleLocalResource(
addToRemotePaths(whetherToAddToRemotePaths, resourcePath);
if (Utils.isRemotePath(resourcePath.toString())) {
- final FileStatus fileStatus =
fileSystem.getFileStatus(resourcePath);
- LOG.debug("Using remote file {} to register local resource",
fileStatus.getPath());
+ final FileSystem srcFs =
resourcePath.getFileSystem(fileSystem.getConf());
+ final FileStatus resourceFileStatus =
srcFs.getFileStatus(resourcePath);
+ LOG.debug("Using remote file {} to register local resource",
resourceFileStatus.getPath());
+ final Path applicationDir = getApplicationDirPath(homeDir,
applicationId);
+ final String suffix = (relativeDstPath.isEmpty() ? "" :
relativeDstPath + "/") + resourcePath.getName();
+ final Path dst = new Path(applicationDir, suffix);
+ LOG.debug("Copying from {} to {} with replication factor {}",
resourcePath, dst, fileReplication);
+ FileUtil.copy(srcFs, resourcePath, fileSystem, dst, false,
fileSystem.getConf());
Review comment:
It is not necessary if the `srcPath` and `dstPath` have the sameschema,
for example: form hdfs to hdfs. Otherwise, It needs to copy files between
different FS. as follows:
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
