yuruguo commented on a change in pull request #14538:
URL: https://github.com/apache/flink/pull/14538#discussion_r552446014
##########
File path:
flink-yarn/src/main/java/org/apache/flink/yarn/YarnApplicationFileUploader.java
##########
@@ -163,8 +164,15 @@ YarnLocalResourceDescriptor registerSingleLocalResource(
addToRemotePaths(whetherToAddToRemotePaths, resourcePath);
if (Utils.isRemotePath(resourcePath.toString())) {
- final FileStatus fileStatus =
fileSystem.getFileStatus(resourcePath);
- LOG.debug("Using remote file {} to register local resource",
fileStatus.getPath());
+ final FileSystem srcFs =
resourcePath.getFileSystem(fileSystem.getConf());
+ final FileStatus resourceFileStatus =
srcFs.getFileStatus(resourcePath);
+ LOG.debug("Using remote file {} to register local resource",
resourceFileStatus.getPath());
+ final Path applicationDir = getApplicationDirPath(homeDir,
applicationId);
+ final String suffix = (relativeDstPath.isEmpty() ? "" :
relativeDstPath + "/") + resourcePath.getName();
+ final Path dst = new Path(applicationDir, suffix);
+ LOG.debug("Copying from {} to {} with replication factor {}",
resourcePath, dst, fileReplication);
+ FileUtil.copy(srcFs, resourcePath, fileSystem, dst, false,
fileSystem.getConf());
+ final FileStatus fileStatus = fileSystem.getFileStatus(dst);
Review comment:
Ok, I will put this processing logic to `uploadLocalFileToRemote`. Then
is it necessary to add a flag to distinguish between local and remote files in
`uploadLocalFileToRemote`? because the processing of the two is bit different.
For example,`uploadLocalFileToRemote(final Path localSrcPath, final String
relativeDstPath, final boolean isLocal)`.
And copy local/remote files to Hadoop FS in `copyToRemoteApplicationDir`, as
follows:
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
