+1. Depending on one of unified container issue MESOS-2970 <https://issues.apache.org/jira/browse/MESOS-2970>, We need to support cache for container image as our next step. It would be perfect if we can reuse some parts of fetcher cache. It may also be helpful to many other issues related to caching.
Thank you for writing up the proposal above! Gilbert On Tue, Nov 10, 2015 at 3:45 PM, Jie Yu <yujie....@gmail.com> wrote: > Hi, > > Fetcher was originally designed to fetch CommandInfo::URIs (e.g., executor > binary) for executors/tasks. A recent refactor (MESOS-336 > <https://issues.apache.org/jira/browse/MESOS-336>) added caching support > to > the fetcher. The recent work on filesystem isolation/unified containerizer > ( > MESOS-2840 <https://issues.apache.org/jira/browse/MESOS-2840>) requires > Mesos to fetch filesystem images (e.g., APPC/DOCKER images) as well. The > natural question is: can we leverage the fetcher to fetch those filesystem > images (and cache them accordingly)? Unfortunately, the existing fetcher > interface is tightly coupled with CommandInfo::URIs for executors/tasks, > making it very hard to be re-used to fetch/cache filesystem images. > > Another motivation for the refactor is that we want to extend the fetcher > to support more types of schemes. For instance, we want to support magnet > URI to enable p2p fetching. This is in fact quite important for operating a > large cluster (MESOS-3596 < > https://issues.apache.org/jira/browse/MESOS-3596>). > The goal here is to allow fetcher to be extended (e.g., using modules) so > that operators can add custom fetching support. > > I proposed a solution in this doc > < > https://docs.google.com/document/d/1p8tmQrGtxG6keZVo19gvHPr9WHxeny6PHTVofcLWqco/edit?usp=sharing > >. > The main idea is to decouple artifacts fetching from artifacts cache > management. We can make artifacts fetching extensible (e.g. to support p2p > fetching), and solve the cache management part later. > > Let me know your thoughts! Thanks! > > - Jie >
