[
https://issues.apache.org/jira/browse/FLINK-14964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17053737#comment-17053737
]
Andreas Hailu commented on FLINK-14964:
---------------------------------------
Hi - I was linked to this ticket through the user distribution list as it's a
feature our team is looking for. Subject: _Flink Conf "yarn.flink-dist-jar"
Question_.
As we submit thousands of Flink apps a day, we run out of Hadoop user disk
space quota, causing our apps to fail. We noticed that a lot of disk space was
consumed by
/user/<ourUser>/.flink/application_<appId>/flink-dist_2.11-1.9.1.jar. Spark
solves this solution by passing in the HDFS location of the spark assembly
package as a option during submission (see "spark.yarn.jar" option).
It would be great if we could have a similar command line option behavior, or a
flink-conf option.
> Support configure remote flink jar
> ----------------------------------
>
> Key: FLINK-14964
> URL: https://issues.apache.org/jira/browse/FLINK-14964
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / YARN
> Reporter: Zili Chen
> Assignee: Zili Chen
> Priority: Major
>
> Corresponding discussion happens on
> [https://lists.apache.org/x/thread.html/a2fef19f6925d43dd2cf3f9a46a2263b20562bd61d4da970f609568d@%3Cdev.flink.apache.org%3E]
> This ticket focuses on support configure remote flink jar. That is, when user
> configure flink jar as "hdfs://..." we also respect the pattern and instead
> of uploading from local, directly register as YARN's {{LocalResource}}.
> This improvement will speed up the deployment on YARN.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
