Zhiyuan Yang created YARN-5396:
----------------------------------
Summary: YARN large file broadcast service
Key: YARN-5396
URL: https://issues.apache.org/jira/browse/YARN-5396
Project: Hadoop YARN
Issue Type: New Feature
Reporter: Zhiyuan Yang
In Hadoop and related softwares, there are demands of broadcasting large files.
For example, YARN application may localize large jar files on each node; Hive
may distribute large tables in fragment-replicate joins; docker integration may
broadcast large container image. The current local resource based solution is
to put the files on HDFS and let each node download from HDFS, which is
inefficient and not scalable. So we want to build a better file transfer
service in YARN so that all applications can use it broadcast large file
efficiently.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
