On 10/11/12 8:01 AM, "Steve Loughran" <ste...@hortonworks.com> wrote:
>On 11 October 2012 00:34, Thilee Subramaniam <thi...@quantcast.com> wrote: > >> >> >> My initial goal was to make Hadoop use QFS the same way it used KFS. >>Since >> Hadoop branch-1 had lib/kfs.xx.jar, I was expecting to include a >> qfs.x.x.jar in the Hadoop release; my first patch was to use such jar. >>But >> now I see that Hadoop trunk links to external maven repos. >> >> It may be reasonable to link qfs.jar from an external source (I haven't >> yet figured out how to serve the maven repo from github for qfs.jar for >> Hadoop - any help on this will be appreciated). > > > >The issue is not so much where qfs.jar comes from as where the >implementation of Hadoop FileSystem goes -the big question being: should >it >live in your (OSS) codebase rather than Hadoop's? That would keep it 100% >in sync with your back end, give you the release schedule to suit you, and >ensure that bugreps end up in your bug reporting tools. > > >> This way your nightly >> builds will work and the tests can catch any qfs related regressions. >> > >That regression testing is going to have be on QFS -which means your >infrastructure, real or virtual. We have made the changes recommended here, and made available a 'Hadoop QFS jar' with QFS. This plugin and the QFS libraries will be maintained & released by the QFS open-source project. Please see the download and usage instructions at https://github.com/quantcast/qfs/wiki/Migration-Guide The QFS tarball contains a hadoop-qfs jar each for Hadoop 0.23.4, 1.0.2, 1.0.4, 1.1.0, and 2.0.2-alpha. Since the interfaces seem similar, I am not sure if this is an overkill: one each for trunk and branch1 may suffice. Could you comment on this? Also, is there documentation on Apache Hadoop website that describe available alternatives to HDFS (or how to add an alternative file system to Hadoop)? Please let us know. The JIRA issue HADOOP-8885 (https://issues.apache.org/jira/browse/HADOOP-8885) can be closed now. Please advise whether I should close it or one of the admins will close it. Thanks, -Thilee
