+1 On Wed, Nov 11, 2015 at 1:57 AM, Jie Yu <yujie....@gmail.com> wrote:
> Tom, I don't have immediate plan to replace CommandInfo::URI (with > executable/extract bits) with URI (in the doc), at least for now. The > existing Fetcher will still perform extraction, chown, etc. for now > (eventually, though I think those logics should be moved to > slave/containerizer). The existing Fetcher can download the artifacts by > leveraging the new uri::Fetcher interface (need a little refactor). > > On Tue, Nov 10, 2015 at 4:44 PM, Tom Arnfeld <t...@duedil.com> wrote: > > > This looks like a great change, btw! > > > > I have a quick question, how does this change affect things like the > > executable/extract bits that are available in the existing fetcher? Would > > that logic move outside of the fetcher itself, or would it live on the > URI? > > > > I’m not sure if I’ve missed something in the design doc about this, but > it > > came to mind… > > > > Tom. > > > > > On 10 Nov 2015, at 23:45, Jie Yu <yujie....@gmail.com> wrote: > > > > > > Hi, > > > > > > Fetcher was originally designed to fetch CommandInfo::URIs (e.g., > > executor > > > binary) for executors/tasks. A recent refactor (MESOS-336 > > > <https://issues.apache.org/jira/browse/MESOS-336>) added caching > > support to > > > the fetcher. The recent work on filesystem isolation/unified > > containerizer ( > > > MESOS-2840 <https://issues.apache.org/jira/browse/MESOS-2840>) > requires > > > Mesos to fetch filesystem images (e.g., APPC/DOCKER images) as well. > The > > > natural question is: can we leverage the fetcher to fetch those > > filesystem > > > images (and cache them accordingly)? Unfortunately, the existing > fetcher > > > interface is tightly coupled with CommandInfo::URIs for > executors/tasks, > > > making it very hard to be re-used to fetch/cache filesystem images. > > > > > > Another motivation for the refactor is that we want to extend the > fetcher > > > to support more types of schemes. For instance, we want to support > magnet > > > URI to enable p2p fetching. This is in fact quite important for > > operating a > > > large cluster (MESOS-3596 < > > https://issues.apache.org/jira/browse/MESOS-3596>). > > > The goal here is to allow fetcher to be extended (e.g., using modules) > so > > > that operators can add custom fetching support. > > > > > > I proposed a solution in this doc > > > < > > > https://docs.google.com/document/d/1p8tmQrGtxG6keZVo19gvHPr9WHxeny6PHTVofcLWqco/edit?usp=sharing > > >. > > > The main idea is to decouple artifacts fetching from artifacts cache > > > management. We can make artifacts fetching extensible (e.g. to support > > p2p > > > fetching), and solve the cache management part later. > > > > > > Let me know your thoughts! Thanks! > > > > > > - Jie > > > > >
