> On Jan. 23, 2015, 5:01 p.m., Adam B wrote: > > include/mesos/fetcher/fetcher.proto, lines 44-45 > > <https://reviews.apache.org/r/30037/diff/6/?file=829684#file829684line44> > > > > How does this fail for an item not present in the cache? Why wouldn't > > you just want to check for presence in the cache, retrieve if present and > > download&cache if absent? > > Bernd Mathiske wrote: > Because I already know what's in the cache and what isn't. This > bookkeeping is NOT in mesos-fetcher, to which this is a parameter. It is in > FetcherProcess, which is in the slave.
Is there a writeup somewhere about what responsibilties fall in the realm of the FetcherProcess vs. mesos-fetcher? It would be interesting to know (especially as an implementer of an external/module fetcher) what state/logic belongs where, so that the API between the two is clear. > On Jan. 23, 2015, 5:01 p.m., Adam B wrote: > > include/mesos/fetcher/fetcher.proto, line 62 > > <https://reviews.apache.org/r/30037/diff/6/?file=829684#file829684line62> > > > > Where did this go? > > Bernd Mathiske wrote: > Turns out the env var HADOOP_HOME from which this was a carry-over is not > used anywhere in mesos-fetcher. It has been passed as an argument for ages > and not used at all. I checked back to April 2014. Nothing. So I removed it. Have you tested your fetcher with HDFS fetching? Mesos assumes that the hdfs binary is either on the path or locatable from slave --hadoop_home or the HADOOP_HOME environment variable. See src/hdfs/hdfs.hpp for the HDFS() constructors. I don't think you need to make it a part of the MESOS_FETCHER_INFO protobuf/environment variable, but if the slave has a --hadoop_home set, that should be passed on to the fetcher in the HADOOP_HOME environment variable. This smells broken. - Adam ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30037/#review69514 ----------------------------------------------------------- On Jan. 25, 2015, 9:44 a.m., Bernd Mathiske wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/30037/ > ----------------------------------------------------------- > > (Updated Jan. 25, 2015, 9:44 a.m.) > > > Review request for mesos, Adam B, Benjamin Hindman, Till Toenshoff, and > Timothy Chen. > > > Bugs: MESOS-2069 > https://issues.apache.org/jira/browse/MESOS-2069 > > > Repository: mesos-git > > > Description > ------- > > Extends the fetcher info protobuf with "actions" (fetch directly, through > cache, retrieve from cache). Switches the basis for dealing with uris to > "items", which contain the uri, the action, and potentially and cache file > name. Refactors fetch() and run(), so there is only one of each. Introduces > about half of the actual cache logic, including a hashmap of cache file > objects for bookkeeping and basic operations on it. > > > Diffs > ----- > > include/mesos/fetcher/fetcher.proto > facb87b92bf3194516f636dcc348e136af537721 > src/launcher/fetcher.cpp fed0105946da579a38357a30e7ae56e646e05b89 > src/slave/containerizer/fetcher.hpp > 1db0eaf002c8d0eaf4e0391858e61e0912b35829 > src/slave/containerizer/fetcher.cpp > d290f95251def3952c5ee34f600e1d71467f6293 > > Diff: https://reviews.apache.org/r/30037/diff/ > > > Testing > ------- > > make check > > > Thanks, > > Bernd Mathiske > >
