> On May 29, 2014, 3:05 p.m., Tom Arnfeld wrote:
> > include/mesos/mesos.proto, line 198
> > <https://reviews.apache.org/r/21316/diff/6/?file=594885#file594885line198>
> >
> > What's the reason you're only caching on a per-framework basis? Surely
> > if the URI is identical the content would be... If I have multiple Hadoop
> > or Spark clients running but using the same executor, it's a shame they
> > wouldn't share a cache entry.
>
> Bernd Mathiske wrote:
> Security concerns. I don't know to what extent what users trust the code
> behind a URI to remain the same. I can change it to per slave if there is
> consensus. Or we could have a flag that decides between the two options.
After some thought, I have come to the conclusion that I should extend the enum
type that indicates how to cache NOW, not later. Here is what I have in mind in
mesos.proto:
enum FetcherCache {
CACHE_NONE = 0;
CACHE_EXTERNAL = 1;
CACHE_PER_SLAVE = 2;
CACHE_PER_FRAMEWORK = 3;
}
I'll implement this before the next patch update.
- Bernd
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/21316/#review44301
-----------------------------------------------------------
On May 31, 2014, 8:04 a.m., Bernd Mathiske wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/21316/
> -----------------------------------------------------------
>
> (Updated May 31, 2014, 8:04 a.m.)
>
>
> Review request for mesos and Benjamin Hindman.
>
>
> Bugs: MESOS-336
> https://issues.apache.org/jira/browse/MESOS-336
>
>
> Repository: mesos-git
>
>
> Description
> -------
>
> The first cut at fetcher caching. See MESOS-336 JIRA for explanation for this
> approach: keep the cache info in the MesosContainerizerProcess in the save,
> leverage actor single-threadedness to deal with concurrency issues without
> head ache.
>
> Features so far:
> - If URI flag "fetched_externally" (default: false) is set, the fetcher does
> what it did in Mesos 0.18 and before.
> - If URI flag "cached" (default: false) is not set, the fetcher also fetches
> every time as in Mesos 0.18 and before.
> - If URI flag "cached" is set, the UIR is only fetched once and all
> subsequent fetch attempts copy from the cache file.
> - URIs are cached separately per framework (ID).
> - Recovery is implemented by simply wiping the entire cache.
> - GC for cache files. Global flag sets lifetime after last use. Default is 1
> hour.
>
> Potential future features:
> - symlinks instead of copying
> - extraction directly from URI, without cache file
> - combine that with symlinks
> - Refreshing, explicit cache invalidation
> - ...
>
>
> Diffs
> -----
>
> include/mesos/mesos.proto ce780ca
> src/Makefile.am ae576c5
> src/launcher/fetcher.cpp c4425eb
> src/local/local.cpp 5d26aff
> src/slave/constants.hpp ace4590
> src/slave/constants.cpp 51f65bb
> src/slave/containerizer/mesos_containerizer.hpp 1f5908a
> src/slave/containerizer/mesos_containerizer.cpp d01d443
> src/slave/containerizer/mesos_fetcher.hpp PRE-CREATION
> src/slave/containerizer/mesos_fetcher.cpp PRE-CREATION
> src/slave/flags.hpp 15e5b64
> src/slave/slave.hpp 769bd00
> src/slave/slave.cpp a4b9570
> src/tests/containerizer_tests.cpp 2f4888d
> src/tests/fetcher_tests.cpp PRE-CREATION
> src/tests/slave_tests.cpp 85ca5c4
>
> Diff: https://reviews.apache.org/r/21316/diff/
>
>
> Testing
> -------
>
> Tests have been written, have been run successfully, and are included in the
> patch.
>
>
> Thanks,
>
> Bernd Mathiske
>
>
