[
https://issues.apache.org/jira/browse/FLINK-17632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Wang updated FLINK-17632:
------------------------------
Description:
After FLINK-13938, we could support to register remote shared libs as Yarn
local resources and prevent unnecessary uploading and downloading for system
Flink jars.
However, we still need to specify a local user jar to run a Flink job on Yarn.
This ticket aims to add the remote path support. It have at least two purposes.
* Accelerate the submission
* Better integration with application mode. Since the user main code is
executed in the cluster(jobmanager), we do not need the user jar exists locally.
A very typical use case is like following.
{code:java}
./bin/flink run-application -p 10 -t yarn-application \
-yD yarn.provided.lib.dirs="hdfs://myhdfs/flink/lib" \
hdfs://myhdfs/jars/WindowJoin.jar{code}
was:
After FLINK-13938, we could support to register remote shared libs as Yarn
local resources and prevent unnecessary uploading and downloading for system
Flink jars.
However, we still need to specify a local user jar to run a Flink job on Yarn.
This ticket aims to add the remote path support. It have at least two purposes.
* Accelerate the submission
* Better integration with application mode. Since the user main code is
executed in the cluster(jobmanager), we do not need the user jar exists locally.
A very typical use case is like following.
{code:java}
./bin/flink run-application -p 10 -t yarn-application \
-yD yarn.provided.lib.dirs="hdfs://myhdfs/flink/lib" \
hdfs://myhdfs/jars/WindowJoin.jar{code}
> Support to specify a remote path for job jar
> --------------------------------------------
>
> Key: FLINK-17632
> URL: https://issues.apache.org/jira/browse/FLINK-17632
> Project: Flink
> Issue Type: Sub-task
> Components: Client / Job Submission, Deployment / YARN
> Affects Versions: 1.11.0
> Reporter: Yang Wang
> Assignee: Yang Wang
> Priority: Major
> Labels: pull-request-available
>
> After FLINK-13938, we could support to register remote shared libs as Yarn
> local resources and prevent unnecessary uploading and downloading for system
> Flink jars.
> However, we still need to specify a local user jar to run a Flink job on
> Yarn. This ticket aims to add the remote path support. It have at least two
> purposes.
> * Accelerate the submission
> * Better integration with application mode. Since the user main code is
> executed in the cluster(jobmanager), we do not need the user jar exists
> locally.
> A very typical use case is like following.
> {code:java}
> ./bin/flink run-application -p 10 -t yarn-application \
> -yD yarn.provided.lib.dirs="hdfs://myhdfs/flink/lib" \
> hdfs://myhdfs/jars/WindowJoin.jar{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
