I believe there is already a JIRA ticket for this, if you search for fetcher in Mesos JIRA I think you can find it.
Tim On Fri, Oct 31, 2014 at 3:27 PM, Ankur Chauhan <[email protected]> wrote: > Hi, > > I have been looking at some of the stuff around the fetcher and saw > something interesting. The code for fetcher::fetch method is dependent on a > hard coded list of url schemes. No doubt that this works but is very > restrictive. > Hadoop/HDFS in general is pretty flexible when it comes to being able to > fetch stuff from urls and has the ability to fetch a large number of types > of urls and can be extended by adding configuration into the > conf/hdfs-site.xml and core-site.xml > > What I am proposing is that we refactor the fetcher.cpp to prefer to use > the hdfs (using hdfs/hdfs.hpp) to do all the fetching if HADOOP_HOME is set > and $HADOOP_HOME/bin/hadoop is available. This logic already exists and we > can just use it. The fallback logic for using net::download or local file > copy is may be left in place for installations that do not have hadoop > configured. This means that if hadoop is present we can directly fetch urls > such as tachyon://... snackfs:// ... cfs:// .... ftp://... s3://... > http:// ... file:// with no extra effort. This makes up for a much better > experience when it comes to debugging and extensibility. > > What do others think about this? > > - Ankur
