-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30626/
-----------------------------------------------------------
(Updated Feb. 8, 2015, 2:14 a.m.)
Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and Timothy
Chen.
Changes
-------
Rebased. Still needs a little work. Posted for review before final touchups.
Bugs: MESOS-2072
https://issues.apache.org/jira/browse/MESOS-2072
Repository: mesos
Description
-------
Fetcher cache eviction happens when the cache does not have enough space to
accomodate upcoming downloads to the cache. Necessary provisions included here:
- mesos-fetcher does not run until eviction was successful
- Cache space is reserved while (async) waiting for eviction to succeed. If it
fails, the reservation gets undone.
- Reservations can be partly from available space, partly from evictions. All
math included :-)
- To find out how much space is needed, downloading has a prelude in which we
query the download size from the URI. This works for all URI types that
mesos-fetcher currently supports, including http and hdfs.
- Size-determination requests are not synchronized and can be repeated. That is
deemed OK, since they are small. But downloading still is synchronized (by the
fetcher actor) since MESOS-2057. This avoids repeated downloads and potential
bandwidth choking.
- There is cleanup code for all kinds of error situations. Lists of URIs or
cache files are reached down as shared pointers to continuations, which can add
to these lists. At the very end of the fetch attempt, each list is processed
for undoing things like space reservations and eviction disabling.
- Eviction gets disabled for URIs that are currently in use, i.e. the related
cache files are. We use reference counting for this, since there may be
concurrent fetch attempts using the same cache files.
This patch depends on a series of smaller preparatory ones.
Diffs (updated)
-----
src/slave/containerizer/fetcher.hpp bfd98dbe16e2bd5df3e2c8e9b10e303654f33446
src/slave/containerizer/fetcher.cpp 6e6bce08d76bb8a5813c905e3ffeff9b2411fd6d
Diff: https://reviews.apache.org/r/30626/diff/
Testing
-------
This still has a couple of minor bugs in the fetcher cache tests (not
necessarily in the code the tests run, just in the tests themselves). I posted
it already because the necessary changes should be minimal and reviews on this
code can start earlier this way. I will update and test shortly.
Thanks,
Bernd Mathiske
