[ https://issues.apache.org/jira/browse/YARN-5396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Zhiyuan Yang updated YARN-5396: ------------------------------- Attachment: slides-prototype.pdf Attach the slides which explains the big picture. > YARN large file broadcast service > --------------------------------- > > Key: YARN-5396 > URL: https://issues.apache.org/jira/browse/YARN-5396 > Project: Hadoop YARN > Issue Type: New Feature > Reporter: Zhiyuan Yang > Assignee: Zhiyuan Yang > Attachments: YARN-broadcast-prototype.patch, slides-prototype.pdf > > > In Hadoop and related softwares, there are demands of broadcasting large > files. For example, YARN application may localize large jar files on each > node; Hive may distribute large tables in fragment-replicate joins; docker > integration may broadcast large container image. The current local resource > based solution is to put the files on HDFS and let each node download from > HDFS, which is inefficient and not scalable. So we want to build a better > file transfer service in YARN so that all applications can use it broadcast > large file efficiently. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org