Gokul, What we expect from a filesystem is defined in (a) the HDFS code , (b) the filesystem spec as derived from (a), and (c) contract tests derived from (a) and (b)
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/filesystem/index.html There's a wiki page to go with this : https://wiki.apache.org/hadoop/HCFS Current policy on 3rd party FS support is pretty much: 1. Nothing in the hadoop codebase that isn't easy for anyone to test 2. Nothing in the hadoop codebase that isn't straightforward for anyone to maintain 3. No FS support bundled in the Hadoop distribution that doesn't match (1) or (2). Note that you are free to use Bigtop to build your own stack with your own FS, which is what Redhat do with GFS, intel did with LustreFS, etc. for a definition of "anyone" as "anyone who knows roughly what they are doing and is familiar with the hadoop codebase" If your proposal had been for some 3rd party FS, the answer would be "keep it in your own source tree". Except this isn't quite an FS, is it? It's an NFS client, which should be able to talk to any NFSv3 server, including the standard Linux and OSX ones, as well as the NFS support that you get with Hadoop. Is that right? That with this code I could run tests on my Linux box which would verify the client works with NFS? On 15 January 2015 at 02:15, Gokul Soundararajan <gokulsoun...@gmail.com> wrote: > Hi Colin, > > Yeah, I should add the reasons to the README. We tried LocalFileSystem when > we started out but we think we can do tighter Hadoop integration if we > write a connector. > Some examples include: > 1. Limit over-prefetching of data - MapReduce splits the jobs into 128MB > splits and standard NFS driver tends to over-prefetch from a file. We limit > the prefetching to the split size. > Its more the MR asks the FS for the block size, and partitions are based on that value. You'd want to return the prefetch-optimised value > 2. Lazy write commits - For writes, we can relax the guarantees for writes > (and making it faster) and commit just before when the task ends. > We haven't actually specified (yet) what output streams do. A key requirement is that flush() persists, at least as far as the Java code is aware of..there's the risk that the underlying OS can be lazy, and in a VM, the virtual hardware can be even lazier. Note that the s3n, s3a and swift break this rule utterly -this isn't considered satisfactory. > 3. Provide for location awareness - Later, we can hook some NFS smarts into > getFileBlockLocations() (Have some ideas but not implemented them yet). > > That could be good. SwiftFS does this with some extensions to OpenStack swift that went in in sync with our code. Like I said, a new FS client for an external FS wouldn't get in to the hadoop codebase. We care about HDFS, somewhat about file://, though primarily for testing, and for integration with cloud storage. An NFS client though, one that can be tested and is general purpose to work with implementations of NFSv3, and ideally integrates with the Hadoop kerberos auth mechanism, that could be nice. > Hope this helps. > > Gokul > > > > On Wed, Jan 14, 2015 at 10:47 AM, Colin McCabe <cmcc...@alumni.cmu.edu> > wrote: > > > Why not just use LocalFileSystem with an NFS mount (or several)? I read > > through the README but I didn't see that question answered anywhere. > > > > best, > > Colin > > > > On Tue, Jan 13, 2015 at 1:35 PM, Gokul Soundararajan < > > gokulsoun...@gmail.com > > > wrote: > > > > > Hi, > > > > > > We (Jingxin Feng, Xing Lin, and I) have been working on providing a > > > FileSystem implementation that allows Hadoop to utilize a NFSv3 storage > > > server as a filesystem. It leverages code from hadoop-nfs project for > all > > > the request/response handling. We would like your help to add it as > part > > of > > > hadoop tools (similar to the way hadoop-aws and hadoop-azure). > > > > > > In more detail, the Hadoop NFS Connector allows Apache Hadoop (2.2+) > and > > > Apache Spark (1.2+) to use a NFSv3 storage server as a storage > endpoint. > > > The NFS Connector can be run in two modes: (1) secondary filesystem - > > where > > > Hadoop/Spark runs using HDFS as its primary storage and can use NFS as > a > > > second storage endpoint, and (2) primary filesystem - where > Hadoop/Spark > > > runs entirely on a NFSv3 storage server. > > > > > > The code is written in a way such that existing applications do not > have > > to > > > change. All one has to do is to copy the connector jar into the lib/ > > > directory of Hadoop/Spark. Then, modify core-site.xml to provide the > > > necessary details. > > > > > > The current version can be seen at: > > > https://github.com/NetApp/NetApp-Hadoop-NFS-Connector > > > > > > It is my first time contributing to the Hadoop codebase. It would be > > great > > > if someone on the Hadoop team can guide us through this process. I'm > > > willing to make the necessary changes to integrate the code. What are > the > > > next steps? Should I create a JIRA entry? > > > > > > Thanks, > > > > > > Gokul > > > > > > -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
