This looks like a great change, btw! I have a quick question, how does this change affect things like the executable/extract bits that are available in the existing fetcher? Would that logic move outside of the fetcher itself, or would it live on the URI?
I’m not sure if I’ve missed something in the design doc about this, but it came to mind… Tom. > On 10 Nov 2015, at 23:45, Jie Yu <yujie....@gmail.com> wrote: > > Hi, > > Fetcher was originally designed to fetch CommandInfo::URIs (e.g., executor > binary) for executors/tasks. A recent refactor (MESOS-336 > <https://issues.apache.org/jira/browse/MESOS-336>) added caching support to > the fetcher. The recent work on filesystem isolation/unified containerizer ( > MESOS-2840 <https://issues.apache.org/jira/browse/MESOS-2840>) requires > Mesos to fetch filesystem images (e.g., APPC/DOCKER images) as well. The > natural question is: can we leverage the fetcher to fetch those filesystem > images (and cache them accordingly)? Unfortunately, the existing fetcher > interface is tightly coupled with CommandInfo::URIs for executors/tasks, > making it very hard to be re-used to fetch/cache filesystem images. > > Another motivation for the refactor is that we want to extend the fetcher > to support more types of schemes. For instance, we want to support magnet > URI to enable p2p fetching. This is in fact quite important for operating a > large cluster (MESOS-3596 <https://issues.apache.org/jira/browse/MESOS-3596>). > The goal here is to allow fetcher to be extended (e.g., using modules) so > that operators can add custom fetching support. > > I proposed a solution in this doc > <https://docs.google.com/document/d/1p8tmQrGtxG6keZVo19gvHPr9WHxeny6PHTVofcLWqco/edit?usp=sharing>. > The main idea is to decouple artifacts fetching from artifacts cache > management. We can make artifacts fetching extensible (e.g. to support p2p > fetching), and solve the cache management part later. > > Let me know your thoughts! Thanks! > > - Jie
