[jira] [Commented] (FLINK-14964) Support configure remote flink jar
[
https://issues.apache.org/jira/browse/FLINK-14964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17187540#comment-17187540
]
Till Rohrmann commented on FLINK-14964:
---
It looks as if FLINK-13938 already solves the problem. Hence, I believe that we
can keep this ticket closed if you agree [~tison].
> Support configure remote flink jar
> --
>
> Key: FLINK-14964
> URL: https://issues.apache.org/jira/browse/FLINK-14964
> Project: Flink
> Issue Type: Improvement
> Components: Client / Job Submission, Deployment / YARN
>Reporter: Zili Chen
>Assignee: Zili Chen
>Priority: Major
> Attachments:
> [XXX]_Enable_configure_shared_libraries_[YYY]_Enable_configure_remote_flink_jar.patch
>
>
> Corresponding discussion happens on
> [https://lists.apache.org/x/thread.html/a2fef19f6925d43dd2cf3f9a46a2263b20562bd61d4da970f609568d@%3Cdev.flink.apache.org%3E]
> This ticket focuses on support configure remote flink jar. That is, when user
> configure flink jar as "hdfs://..." we also respect the pattern and instead
> of uploading from local, directly register as YARN's {{LocalResource}}.
> This improvement will speed up the deployment on YARN.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14964) Support configure remote flink jar
[
https://issues.apache.org/jira/browse/FLINK-14964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17186562#comment-17186562
]
Andreas Hailu commented on FLINK-14964:
---
@[~trohrmann] What's the difference between this feature and
[FLINK-13938|https://issues.apache.org/jira/browse/FLINK-13938] ?
> Support configure remote flink jar
> --
>
> Key: FLINK-14964
> URL: https://issues.apache.org/jira/browse/FLINK-14964
> Project: Flink
> Issue Type: Improvement
> Components: Client / Job Submission, Deployment / YARN
>Reporter: Zili Chen
>Priority: Major
> Attachments:
> [XXX]_Enable_configure_shared_libraries_[YYY]_Enable_configure_remote_flink_jar.patch
>
>
> Corresponding discussion happens on
> [https://lists.apache.org/x/thread.html/a2fef19f6925d43dd2cf3f9a46a2263b20562bd61d4da970f609568d@%3Cdev.flink.apache.org%3E]
> This ticket focuses on support configure remote flink jar. That is, when user
> configure flink jar as "hdfs://..." we also respect the pattern and instead
> of uploading from local, directly register as YARN's {{LocalResource}}.
> This improvement will speed up the deployment on YARN.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14964) Support configure remote flink jar
[
https://issues.apache.org/jira/browse/FLINK-14964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17186555#comment-17186555
]
Zili Chen commented on FLINK-14964:
---
Let me recall the context and hopefully reply in tomorrow :P
> Support configure remote flink jar
> --
>
> Key: FLINK-14964
> URL: https://issues.apache.org/jira/browse/FLINK-14964
> Project: Flink
> Issue Type: Improvement
> Components: Client / Job Submission, Deployment / YARN
>Reporter: Zili Chen
>Priority: Major
> Attachments:
> [XXX]_Enable_configure_shared_libraries_[YYY]_Enable_configure_remote_flink_jar.patch
>
>
> Corresponding discussion happens on
> [https://lists.apache.org/x/thread.html/a2fef19f6925d43dd2cf3f9a46a2263b20562bd61d4da970f609568d@%3Cdev.flink.apache.org%3E]
> This ticket focuses on support configure remote flink jar. That is, when user
> configure flink jar as "hdfs://..." we also respect the pattern and instead
> of uploading from local, directly register as YARN's {{LocalResource}}.
> This improvement will speed up the deployment on YARN.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14964) Support configure remote flink jar
[
https://issues.apache.org/jira/browse/FLINK-14964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17186424#comment-17186424
]
Till Rohrmann commented on FLINK-14964:
---
[~tison] any chance that we can include your patch for this feature?
> Support configure remote flink jar
> --
>
> Key: FLINK-14964
> URL: https://issues.apache.org/jira/browse/FLINK-14964
> Project: Flink
> Issue Type: Improvement
> Components: Client / Job Submission, Deployment / YARN
>Reporter: Zili Chen
>Priority: Major
> Attachments:
> [XXX]_Enable_configure_shared_libraries_[YYY]_Enable_configure_remote_flink_jar.patch
>
>
> Corresponding discussion happens on
> [https://lists.apache.org/x/thread.html/a2fef19f6925d43dd2cf3f9a46a2263b20562bd61d4da970f609568d@%3Cdev.flink.apache.org%3E]
> This ticket focuses on support configure remote flink jar. That is, when user
> configure flink jar as "hdfs://..." we also respect the pattern and instead
> of uploading from local, directly register as YARN's {{LocalResource}}.
> This improvement will speed up the deployment on YARN.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14964) Support configure remote flink jar
[
https://issues.apache.org/jira/browse/FLINK-14964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17084476#comment-17084476
]
Zili Chen commented on FLINK-14964:
---
I've submit a patch based on 1.9.
Generally speaking we don't do auto-detection but have another method to
register remote ship files(or called shared libs). The original registering
ship files method remains and still serves for local ship files. The difference
between them is we don't have to upload files before registering as local
resources.
> Support configure remote flink jar
> --
>
> Key: FLINK-14964
> URL: https://issues.apache.org/jira/browse/FLINK-14964
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / YARN
>Reporter: Zili Chen
>Priority: Major
> Attachments:
> [XXX]_Enable_configure_shared_libraries_[YYY]_Enable_configure_remote_flink_jar.patch
>
>
> Corresponding discussion happens on
> [https://lists.apache.org/x/thread.html/a2fef19f6925d43dd2cf3f9a46a2263b20562bd61d4da970f609568d@%3Cdev.flink.apache.org%3E]
> This ticket focuses on support configure remote flink jar. That is, when user
> configure flink jar as "hdfs://..." we also respect the pattern and instead
> of uploading from local, directly register as YARN's {{LocalResource}}.
> This improvement will speed up the deployment on YARN.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14964) Support configure remote flink jar
[
https://issues.apache.org/jira/browse/FLINK-14964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17084162#comment-17084162
]
Zili Chen commented on FLINK-14964:
---
I have a patch for the feature but it seems just patching the messy
{{YARNClusterDescriptor}}. Will add a patch tomorrow and anyone who wants to
work on this issue can make use of it. I myself would like to include it with
FLINK-16733 but given it is a valuable improvement I'm glad to see if someone
takes over it.
> Support configure remote flink jar
> --
>
> Key: FLINK-14964
> URL: https://issues.apache.org/jira/browse/FLINK-14964
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / YARN
>Reporter: Zili Chen
>Priority: Major
>
> Corresponding discussion happens on
> [https://lists.apache.org/x/thread.html/a2fef19f6925d43dd2cf3f9a46a2263b20562bd61d4da970f609568d@%3Cdev.flink.apache.org%3E]
> This ticket focuses on support configure remote flink jar. That is, when user
> configure flink jar as "hdfs://..." we also respect the pattern and instead
> of uploading from local, directly register as YARN's {{LocalResource}}.
> This improvement will speed up the deployment on YARN.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14964) Support configure remote flink jar
[
https://issues.apache.org/jira/browse/FLINK-14964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17084151#comment-17084151
]
Canbin Zheng commented on FLINK-14964:
--
It's a good improvement in native Kubernetes as well. BTW, do we have a plan on
supporting remote shipLibs?
> Support configure remote flink jar
> --
>
> Key: FLINK-14964
> URL: https://issues.apache.org/jira/browse/FLINK-14964
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / YARN
>Reporter: Zili Chen
>Priority: Major
>
> Corresponding discussion happens on
> [https://lists.apache.org/x/thread.html/a2fef19f6925d43dd2cf3f9a46a2263b20562bd61d4da970f609568d@%3Cdev.flink.apache.org%3E]
> This ticket focuses on support configure remote flink jar. That is, when user
> configure flink jar as "hdfs://..." we also respect the pattern and instead
> of uploading from local, directly register as YARN's {{LocalResource}}.
> This improvement will speed up the deployment on YARN.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14964) Support configure remote flink jar
[
https://issues.apache.org/jira/browse/FLINK-14964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17084143#comment-17084143
]
Till Rohrmann commented on FLINK-14964:
---
I agree this could be a valuable usability improvement for Flink's Yarn
integration.
> Support configure remote flink jar
> --
>
> Key: FLINK-14964
> URL: https://issues.apache.org/jira/browse/FLINK-14964
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / YARN
>Reporter: Zili Chen
>Priority: Major
>
> Corresponding discussion happens on
> [https://lists.apache.org/x/thread.html/a2fef19f6925d43dd2cf3f9a46a2263b20562bd61d4da970f609568d@%3Cdev.flink.apache.org%3E]
> This ticket focuses on support configure remote flink jar. That is, when user
> configure flink jar as "hdfs://..." we also respect the pattern and instead
> of uploading from local, directly register as YARN's {{LocalResource}}.
> This improvement will speed up the deployment on YARN.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14964) Support configure remote flink jar
[
https://issues.apache.org/jira/browse/FLINK-14964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17053737#comment-17053737
]
Andreas Hailu commented on FLINK-14964:
---
Hi - I was linked to this ticket through the user distribution list as it's a
feature our team is looking for. Subject: _Flink Conf "yarn.flink-dist-jar"
Question_.
As we submit thousands of Flink apps a day, we run out of Hadoop user disk
space quota, causing our apps to fail. We noticed that a lot of disk space was
consumed by
/user//.flink/application_/flink-dist_2.11-1.9.1.jar. Spark
solves this solution by passing in the HDFS location of the spark assembly
package as a option during submission (see "spark.yarn.jar" option).
It would be great if we could have a similar command line option behavior, or a
flink-conf option.
> Support configure remote flink jar
> --
>
> Key: FLINK-14964
> URL: https://issues.apache.org/jira/browse/FLINK-14964
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / YARN
>Reporter: Zili Chen
>Assignee: Zili Chen
>Priority: Major
>
> Corresponding discussion happens on
> [https://lists.apache.org/x/thread.html/a2fef19f6925d43dd2cf3f9a46a2263b20562bd61d4da970f609568d@%3Cdev.flink.apache.org%3E]
> This ticket focuses on support configure remote flink jar. That is, when user
> configure flink jar as "hdfs://..." we also respect the pattern and instead
> of uploading from local, directly register as YARN's {{LocalResource}}.
> This improvement will speed up the deployment on YARN.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14964) Support configure remote flink jar
[
https://issues.apache.org/jira/browse/FLINK-14964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16987631#comment-16987631
]
Zili Chen commented on FLINK-14964:
---
[~chesnay] it should be the same as current, i.e., {{-yj ...}}. The difference
is that we previously only allow "file://" scheme but in this ticket it plans
to extend support for "hdfs://" scheme. I don't have enough time to get it
implemented before 1.10 feature freeze. Let's revisit this one in the next
develop cycle.
> Support configure remote flink jar
> --
>
> Key: FLINK-14964
> URL: https://issues.apache.org/jira/browse/FLINK-14964
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / YARN
>Reporter: Zili Chen
>Assignee: Zili Chen
>Priority: Major
>
> Corresponding discussion happens on
> [https://lists.apache.org/x/thread.html/a2fef19f6925d43dd2cf3f9a46a2263b20562bd61d4da970f609568d@%3Cdev.flink.apache.org%3E]
> This ticket focuses on support configure remote flink jar. That is, when user
> configure flink jar as "hdfs://..." we also respect the pattern and instead
> of uploading from local, directly register as YARN's {{LocalResource}}.
> This improvement will speed up the deployment on YARN.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14964) Support configure remote flink jar
[
https://issues.apache.org/jira/browse/FLINK-14964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16983599#comment-16983599
]
Chesnay Schepler commented on FLINK-14964:
--
what's the planned behavior outside of yarn?
> Support configure remote flink jar
> --
>
> Key: FLINK-14964
> URL: https://issues.apache.org/jira/browse/FLINK-14964
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / YARN
>Reporter: Zili Chen
>Assignee: Zili Chen
>Priority: Major
> Fix For: 1.10.0
>
>
> Corresponding discussion happens on
> [https://lists.apache.org/x/thread.html/a2fef19f6925d43dd2cf3f9a46a2263b20562bd61d4da970f609568d@%3Cdev.flink.apache.org%3E]
> This ticket focuses on support configure remote flink jar. That is, when user
> configure flink jar as "hdfs://..." we also respect the pattern and instead
> of uploading from local, directly register as YARN's {{LocalResource}}.
> This improvement will speed up the deployment on YARN.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
