Yang Wang created FLINK-13938:
---------------------------------
Summary: Use yarn public distributed cache to speed up containers
launch
Key: FLINK-13938
URL: https://issues.apache.org/jira/browse/FLINK-13938
Project: Flink
Issue Type: New Feature
Reporter: Yang Wang
By default, the LocalResourceVisibility is APPLICATION, so they will be
downloaded only once and shared for all taskmanager containers of a same
application in the same node. However, different applications will have to
download all jars every time, including the flink-dist.jar. I think we could
use the yarn public cache to eliminate the unnecessary jars downloading and
make launching container faster.
How to use the shared lib feature?
# Upload a copy of flink release binary to hdfs.
# Use the -ysl argument to specify the shared lib
{code:java}
./bin/flink run -d -m yarn-cluster -p 20 -ysl
hdfs:///flink/release/flink-1.9.0/lib examples/streaming/WindowJoin.jar{code}
-ysl,--yarnsharedLib <path> Upload a copy of flink lib beforehand
and specify the path
to use public
visibility feature of
YARN NodeManager
localizing resources.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
