[
https://issues.apache.org/jira/browse/MESOS-3596?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14945543#comment-14945543
]
Ian Downes commented on MESOS-3596:
-----------------------------------
For O(10000) applications there will be structure in the network would could be
used to further optimize the distribution. For example, it's better for slaves
in the same rack to exchange data through the rack switch rather than between
slaves across the DC going through the network core. I'm imagining something
where hdfs might be the initial seed, seeding out the racks, and then
distributing within racks, i.e., adding some hierarchical structure to
bittorrent or similar p2p protocol.
> Supporting fetching using P2P protocol.
> ---------------------------------------
>
> Key: MESOS-3596
> URL: https://issues.apache.org/jira/browse/MESOS-3596
> Project: Mesos
> Issue Type: Wish
> Reporter: Jie Yu
>
> Package distribution in a large scale cluster is a non-trivial problem. This
> problem will get worse once we start to support container images
> (MESOS-2840). Imaging O(10000) agents simultaneously fetching an O(GB)
> container image from HDFS which only has 3 replicas for the given image file.
> It'll be great if Mesos can support fetching packages using a P2P protocol
> (e.g., BitTorrent). We can also configure the P2P clients so that they are
> locality aware (e.g., fetching from peers in the same rack).
> Content distribution using P2P protocols is not a new thing. It has been
> discussed and used in production, and proven to be successful:
> http://www.ebaytechblog.com/2012/01/31/bittorrent-for-package-distribution-in-the-enterprise/
> https://blog.twitter.com/2010/murder-fast-datacenter-code-deploys-using-bittorrent
> https://torrentfreak.com/facebook-uses-bittorrent-and-they-love-it-100625/
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
