[
https://issues.apache.org/jira/browse/FLINK-18087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Wang updated FLINK-18087:
------------------------------
Description:
In FLINK-17632, we add the support remote user jar. However, uploading user
artifact for Yarn job cluster is broken exceptionally. In the following code,
we should only upload local files. Now it has the contrary behavior.
{code:java}
// only upload local files
if (Utils.isRemotePath(entry.getValue().filePath)) {
Path localPath = new Path(entry.getValue().filePath);
Tuple2<Path, Long> remoteFileInfo =
fileUploader.uploadLocalFileToRemote(localPath, entry.getKey());
jobGraph.setUserArtifactRemotePath(entry.getKey(),
remoteFileInfo.f0.toString());
}
{code}
Another problem is the related tests {{testPerJobModeWithDistributedCache}}
does not fail because we do not fetch the artifact from remote filesystem(i.e.
HDFS). We directly get it from local file. It also needs to be enhanced.
was:
In FLINK-17632, we add the support remote user jar. However, uploading user
artifact for Yarn job cluster is broken exceptionally. In the following code,
we should only upload local files. Now it has the contrary behavior.
{code:java}
// only upload local files
if (Utils.isRemotePath(entry.getValue().filePath)) {
Path localPath = new Path(entry.getValue().filePath);
Tuple2<Path, Long> remoteFileInfo =
fileUploader.uploadLocalFileToRemote(localPath, entry.getKey());
jobGraph.setUserArtifactRemotePath(entry.getKey(),
remoteFileInfo.f0.toString());
}
{code}
Another problem is the related tests {{testPerJobModeWithDistributedCache}}
does not fail because we do not fetch the artifact from blob server. We
directly get it from local file. It also needs to be enhanced.
> Uploading user artifact for Yarn job cluster could not work
> -----------------------------------------------------------
>
> Key: FLINK-18087
> URL: https://issues.apache.org/jira/browse/FLINK-18087
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
> Affects Versions: 1.11.0, 1.12.0
> Reporter: Yang Wang
> Assignee: Yang Wang
> Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.11.0, 1.12.0
>
>
> In FLINK-17632, we add the support remote user jar. However, uploading user
> artifact for Yarn job cluster is broken exceptionally. In the following code,
> we should only upload local files. Now it has the contrary behavior.
> {code:java}
> // only upload local files
> if (Utils.isRemotePath(entry.getValue().filePath)) {
> Path localPath = new Path(entry.getValue().filePath);
> Tuple2<Path, Long> remoteFileInfo =
> fileUploader.uploadLocalFileToRemote(localPath, entry.getKey());
> jobGraph.setUserArtifactRemotePath(entry.getKey(),
> remoteFileInfo.f0.toString());
> }
> {code}
>
> Another problem is the related tests {{testPerJobModeWithDistributedCache}}
> does not fail because we do not fetch the artifact from remote
> filesystem(i.e. HDFS). We directly get it from local file. It also needs to
> be enhanced.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
