[
https://issues.apache.org/jira/browse/FLINK-13938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Wang updated FLINK-13938:
------------------------------
Description:
By default, the LocalResourceVisibility is APPLICATION, so they will be
downloaded only once and shared for all taskmanager containers of a same
application in the same node. However, different applications will have to
download all jars every time, including the flink-dist.jar. I think we could
use the yarn public cache to eliminate the unnecessary jars downloading and
make launching container faster.
How to use the shared lib feature?
# Upload a copy of flink release binary to hdfs.
# Use the -ysl argument to specify the shared lib
{code:java}
./bin/flink run -d -m yarn-cluster -p 20 -ysl
hdfs:///flink/release/flink-1.9.0/lib examples/streaming/WindowJoin.jar{code}
-ysl, --yarnsharedLib <path> Upload a copy of flink lib beforehand
and specify the path
to use public
visibility feature of
YARN NodeManager
localizing resources.
was:
By default, the LocalResourceVisibility is APPLICATION, so they will be
downloaded only once and shared for all taskmanager containers of a same
application in the same node. However, different applications will have to
download all jars every time, including the flink-dist.jar. I think we could
use the yarn public cache to eliminate the unnecessary jars downloading and
make launching container faster.
How to use the shared lib feature?
# Upload a copy of flink release binary to hdfs.
# Use the -ysl argument to specify the shared lib
{code:java}
./bin/flink run -d -m yarn-cluster -p 20 -ysl
hdfs:///flink/release/flink-1.9.0/lib examples/streaming/WindowJoin.jar{code}
-ysl,--yarnsharedLib <path> Upload a copy of flink lib beforehand
and specify the path
to use public
visibility feature of
YARN NodeManager
localizing resources.
> Use yarn public distributed cache to speed up containers launch
> ---------------------------------------------------------------
>
> Key: FLINK-13938
> URL: https://issues.apache.org/jira/browse/FLINK-13938
> Project: Flink
> Issue Type: New Feature
> Reporter: Yang Wang
> Priority: Major
>
> By default, the LocalResourceVisibility is APPLICATION, so they will be
> downloaded only once and shared for all taskmanager containers of a same
> application in the same node. However, different applications will have to
> download all jars every time, including the flink-dist.jar. I think we could
> use the yarn public cache to eliminate the unnecessary jars downloading and
> make launching container faster.
>
> How to use the shared lib feature?
> # Upload a copy of flink release binary to hdfs.
> # Use the -ysl argument to specify the shared lib
> {code:java}
> ./bin/flink run -d -m yarn-cluster -p 20 -ysl
> hdfs:///flink/release/flink-1.9.0/lib examples/streaming/WindowJoin.jar{code}
>
> -ysl, --yarnsharedLib <path> Upload a copy of flink lib beforehand
> and specify the
> path to use public
> visibility feature
> of YARN NodeManager
> localizing
> resources.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
