[
https://issues.apache.org/jira/browse/FLINK-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16974709#comment-16974709
]
Zili Chen commented on FLINK-13938:
-----------------------------------
Sure. I will give it a look now.
> Use yarn public distributed cache to speed up containers launch
> ---------------------------------------------------------------
>
> Key: FLINK-13938
> URL: https://issues.apache.org/jira/browse/FLINK-13938
> Project: Flink
> Issue Type: New Feature
> Components: Deployment / YARN
> Reporter: Yang Wang
> Assignee: Yang Wang
> Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Currently, every time we start a flink cluster, flink lib jars need to be
> uploaded to hdfs and then register Yarn local resource so that it could be
> downloaded to jobmanager and all taskmanager container. I think we could have
> two optimizations.
> # Use pre-uploaded flink binary to avoid uploading of flink system jars
> # Use the yarn public cache to eliminate the unnecessary jars downloading
> and make launching container faster. The public cache could be shared by
> different applications.
>
> By default, the LocalResourceVisibility is APPLICATION, so they will be
> downloaded only once and shared for all taskmanager containers of a same
> application in the same node. However, different applications will have to
> download all jars every time, including the flink-dist.jar. We could use the
> yarn public cache to eliminate the unnecessary jars downloading and make
> launching container faster.
>
> How to use the shared lib feature?
> # Upload a copy of flink release binary to hdfs.
> # Use the -ypl argument to specify the shared lib
> {code:java}
> ./bin/flink run -d -m yarn-cluster -p 20 -ypl
> hdfs://hdpdev/flink/release/flink-1.9.0
> examples/streaming/WindowJoin.jar{code}
>
> -ypl, --yarnpreUpload <path> Upload a copy of flink binary beforehand
> and specify the
> path to use public
> visibility feature
> of YARN NodeManager
> localizing
> resources.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
