[
https://issues.apache.org/jira/browse/FLINK-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16969172#comment-16969172
]
Zili Chen commented on FLINK-13938:
-----------------------------------
Thanks for proposing this [~fly_in_gis]! We meet a similar requirement here
that instead uploading share libs every time, let AM/TM localize the resource
directly from HDFS. However, as you said your change would "eliminate the
unnecessary jars downloading and make launching container faster.", IIUC there
is always overhead localize resources from HDFS to container?
BTW, what's the progress on this issue now?
> Use yarn public distributed cache to speed up containers launch
> ---------------------------------------------------------------
>
> Key: FLINK-13938
> URL: https://issues.apache.org/jira/browse/FLINK-13938
> Project: Flink
> Issue Type: New Feature
> Components: Deployment / YARN
> Reporter: Yang Wang
> Assignee: Yang Wang
> Priority: Major
>
> By default, the LocalResourceVisibility is APPLICATION, so they will be
> downloaded only once and shared for all taskmanager containers of a same
> application in the same node. However, different applications will have to
> download all jars every time, including the flink-dist.jar. I think we could
> use the yarn public cache to eliminate the unnecessary jars downloading and
> make launching container faster.
>
> How to use the shared lib feature?
> # Upload a copy of flink release binary to hdfs.
> # Use the -ysl argument to specify the shared lib
> {code:java}
> ./bin/flink run -d -m yarn-cluster -p 20 -ysl
> hdfs:///flink/release/flink-1.9.0/lib examples/streaming/WindowJoin.jar{code}
>
> -ysl, --yarnsharedLib <path> Upload a copy of flink lib beforehand
> and specify the
> path to use public
> visibility feature
> of YARN NodeManager
> localizing
> resources.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
