I think it's too late to be included, since it's going to take some rounds of review, and Ian is cutting the release today.
We'll have to tag this for the next release. Tim On Mon, Nov 3, 2014 at 10:22 AM, Ankur Chauhan <an...@malloc64.com> wrote: > Hi Tim/others, > > Is this to be included in the 0.21.0 release? If so, I don't know how to tag > it etc. I would really (shamelessly) love it to be included as it would > really simplify my intended usecase of using snackfs (cassandra backed > filesystem). > > -- Ankur > >> On 3 Nov 2014, at 09:28, Ankur Chauhan <an...@malloc64.com> wrote: >> >> Yea, I saw those today morning. I'll hold off a little mesos-336 changes a >> lot of stuff. >> >> Sent from my iPhone >> >> On Nov 3, 2014, at 9:18 AM, Adam Bordelon <a...@mesosphere.io >> <mailto:a...@mesosphere.io>> wrote: >> >>> + Bernd, who has done some fetcher work, including additional testing, for >>> MESOS-1316, MESOS-1945, and MESOS-336 >>> >>> On Mon, Nov 3, 2014 at 9:04 AM, Dominic Hamon <dha...@twopensource.com >>> <mailto:dha...@twopensource.com>> wrote: >>> Hi Ankur >>> >>> I think this is a great approach. It makes the code much simpler, >>> extensible, and more testable. Anyone that's heard me rant knows I am a big >>> fan of unit tests over integration tests, so this shouldn't surprise anyone >>> :) >>> >>> If you haven't already, please read the documentation on contributing to >>> Mesos and the style guide to ensure all the naming is as expected, then you >>> can push the patch to reviewboard to get it reviewed and committed. >>> >>> On Mon, Nov 3, 2014 at 12:49 AM, Ankur Chauhan <an...@malloc64.com >>> <mailto:an...@malloc64.com>> wrote: >>> Hi, >>> >>> I did some learning today! This is pretty much a very rough draft of the >>> tests/refactor of mesos-fetcher that I have come up with. Again, If there >>> are some obvious mistakes, please let me know. (this is my first pass after >>> all). >>> https://github.com/ankurcha/mesos/compare/prefer_2 >>> <https://github.com/ankurcha/mesos/compare/prefer_2> >>> >>> My main intention is to break the logic of the fetcher info some very >>> discrete components that i can write tests against. I am still re-learning >>> cpp/mesos code styles etc so I may be a little slow to catch up but I would >>> really appreciate any comments and/or suggestions. >>> >>> -- Ankur >>> @ankurcha >>> >>>> On 2 Nov 2014, at 18:17, Ankur Chauhan <an...@malloc64.com >>>> <mailto:an...@malloc64.com>> wrote: >>>> >>>> Hi, >>>> >>>> I noticed that the current set of tests in `src/tests/fetcher_tests.cpp` >>>> is pretty coarse grained and are more on the lines of a functional test. I >>>> was going to add some tests but it seems like if I am to do that I would >>>> need to add a test dependency on hadoop. >>>> >>>> As an alternative, I propose adding a good set of unit tests around the >>>> methods used by `src/launcher/fetcher.cpp` and `src/hdfs/hdfs.cpp`. This >>>> should be able to catch a good portion of cases at the same time keeping >>>> the dependencies and runtime of tests low. What do you guys thing about >>>> this? >>>> >>>> PS: I am pretty green in terms of gtest and the overall c++ testing >>>> methodology. Can someone give me pointers to good examples of tests in the >>>> codebase. >>>> >>>> -- Ankur >>>> >>>>> On 1 Nov 2014, at 22:54, Adam Bordelon <a...@mesosphere.io >>>>> <mailto:a...@mesosphere.io>> wrote: >>>>> >>>>> Thank you Ankur. At first glance, it looks great. We'll do a more >>>>> thorough review of it very soon. >>>>> I know Tim St. Clair had some ideas for fixing MESOS-1711 >>>>> <https://issues.apache.org/jira/browse/MESOS-1711>; he may want to review >>>>> too. >>>>> >>>>> On Sat, Nov 1, 2014 at 8:49 PM, Ankur Chauhan <an...@malloc64.com >>>>> <mailto:an...@malloc64.com>> wrote: >>>>> Hi Tim, >>>>> >>>>> I just created a review https://reviews.apache.org/r/27483/ >>>>> <https://reviews.apache.org/r/27483/> It's my first stab at it and I will >>>>> try to add more tests as I figure out how to do the hadoop mocking and >>>>> stuff. Have a look and let me know what you think about it so far. >>>>> >>>>> -- Ankur >>>>> >>>>>> On 1 Nov 2014, at 20:05, Ankur Chauhan <an...@malloc64.com >>>>>> <mailto:an...@malloc64.com>> wrote: >>>>>> >>>>>> Yea, i saw that the minute i pressed send. I'll start the review board >>>>>> so that people can have a look at the change. >>>>>> >>>>>> -- Ankur >>>>>> >>>>>>> On 1 Nov 2014, at 20:01, Tim Chen <t...@mesosphere.io >>>>>>> <mailto:t...@mesosphere.io>> wrote: >>>>>>> >>>>>>> Hi Ankur, >>>>>>> >>>>>>> There is a fetcher_tests.cpp in src/tests. >>>>>>> >>>>>>> Tim >>>>>>> >>>>>>> On Sat, Nov 1, 2014 at 7:27 PM, Ankur Chauhan <an...@malloc64.com >>>>>>> <mailto:an...@malloc64.com>> wrote: >>>>>>> Hi Tim, >>>>>>> >>>>>>> I am trying to find/write some test cases. I couldn't find a >>>>>>> fetcher_tests.{cpp|hpp} so once I have something, I'll post on review >>>>>>> board. I am new to gmock/gtest so bear with me while i get up to speed. >>>>>>> >>>>>>> -- Ankur >>>>>>> >>>>>>>> On 1 Nov 2014, at 19:23, Timothy Chen <t...@mesosphere.io >>>>>>>> <mailto:t...@mesosphere.io>> wrote: >>>>>>>> >>>>>>>> Hi Ankur, >>>>>>>> >>>>>>>> Can you post on reviewboard? We can discuss more about the code there. >>>>>>>> >>>>>>>> Tim >>>>>>>> >>>>>>>> Sent from my iPhone >>>>>>>> >>>>>>>> On Nov 1, 2014, at 6:29 PM, Ankur Chauhan <an...@malloc64.com >>>>>>>> <mailto:an...@malloc64.com>> wrote: >>>>>>>> >>>>>>>>> Hi Tim, >>>>>>>>> >>>>>>>>> I don't think there is an issue which is directly in line with what i >>>>>>>>> wanted but the closest one that I could find in JIRA is >>>>>>>>> https://issues.apache.org/jira/browse/MESOS-1711 >>>>>>>>> <https://issues.apache.org/jira/browse/MESOS-1711> >>>>>>>>> >>>>>>>>> I have a branch ( >>>>>>>>> https://github.com/ankurcha/mesos/compare/prefer_hadoop_fetcher >>>>>>>>> <https://github.com/ankurcha/mesos/compare/prefer_hadoop_fetcher> ) >>>>>>>>> that has a change that would enable users to specify whatever hdfs >>>>>>>>> compatible uris to the mesos-fetcher but maybe you can weight in on >>>>>>>>> it. Do you think this is the right track? if so, i would like to pick >>>>>>>>> this issue and submit a patch for review. >>>>>>>>> >>>>>>>>> -- Ankur >>>>>>>>> >>>>>>>>> >>>>>>>>>> On 1 Nov 2014, at 04:32, Tom Arnfeld <t...@duedil.com >>>>>>>>>> <mailto:t...@duedil.com>> wrote: >>>>>>>>>> >>>>>>>>>> Completely +1 to this. There are now quite a lot of hadoop >>>>>>>>>> compatible filesystem wrappers out in the wild and this would >>>>>>>>>> certainly be very useful. >>>>>>>>>> >>>>>>>>>> I'm happy to contribute a patch. Here's a few related issues that >>>>>>>>>> might be of interest; >>>>>>>>>> >>>>>>>>>> - https://issues.apache.org/jira/browse/MESOS-1887 >>>>>>>>>> <https://issues.apache.org/jira/browse/MESOS-1887> >>>>>>>>>> - https://issues.apache.org/jira/browse/MESOS-1316 >>>>>>>>>> <https://issues.apache.org/jira/browse/MESOS-1316> >>>>>>>>>> - https://issues.apache.org/jira/browse/MESOS-336 >>>>>>>>>> <https://issues.apache.org/jira/browse/MESOS-336> >>>>>>>>>> - https://issues.apache.org/jira/browse/MESOS-1248 >>>>>>>>>> <https://issues.apache.org/jira/browse/MESOS-1248> >>>>>>>>>> >>>>>>>>>> On 31 October 2014 22:39, Tim Chen <t...@mesosphere.io >>>>>>>>>> <mailto:t...@mesosphere.io>> wrote: >>>>>>>>>> I believe there is already a JIRA ticket for this, if you search for >>>>>>>>>> fetcher in Mesos JIRA I think you can find it. >>>>>>>>>> >>>>>>>>>> Tim >>>>>>>>>> >>>>>>>>>> On Fri, Oct 31, 2014 at 3:27 PM, Ankur Chauhan <an...@malloc64.com >>>>>>>>>> <mailto:an...@malloc64.com>> wrote: >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> I have been looking at some of the stuff around the fetcher and saw >>>>>>>>>> something interesting. The code for fetcher::fetch method is >>>>>>>>>> dependent on a hard coded list of url schemes. No doubt that this >>>>>>>>>> works but is very restrictive. >>>>>>>>>> Hadoop/HDFS in general is pretty flexible when it comes to being >>>>>>>>>> able to fetch stuff from urls and has the ability to fetch a large >>>>>>>>>> number of types of urls and can be extended by adding configuration >>>>>>>>>> into the conf/hdfs-site.xml and core-site.xml >>>>>>>>>> >>>>>>>>>> What I am proposing is that we refactor the fetcher.cpp to prefer to >>>>>>>>>> use the hdfs (using hdfs/hdfs.hpp) to do all the fetching if >>>>>>>>>> HADOOP_HOME is set and $HADOOP_HOME/bin/hadoop is available. This >>>>>>>>>> logic already exists and we can just use it. The fallback logic for >>>>>>>>>> using net::download or local file copy is may be left in place for >>>>>>>>>> installations that do not have hadoop configured. This means that if >>>>>>>>>> hadoop is present we can directly fetch urls such as tachyon://... >>>>>>>>>> snackfs:// ... cfs:// .... ftp://... s3://... http:// ... file:// >>>>>>>>>> with no extra effort. This makes up for a much better experience >>>>>>>>>> when it comes to debugging and extensibility. >>>>>>>>>> >>>>>>>>>> What do others think about this? >>>>>>>>>> >>>>>>>>>> - Ankur >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>>> >>>> >>> >>> >>> >>> >>> -- >>> Dominic Hamon | @mrdo | Twitter >>> There are no bad ideas; only good ideas that go horribly wrong. >>> >
