Grant Ingersoll wrote:
On May 6, 2008, at 8:04 AM, phonechen wrote:
sorry , I make a mistake,
what I means is that ,shall we put the doc to be classified to HDFS and
leave the Model files on the HDFS and
make the whole classify process run on the HDFS,
so what to change is :
=====================
Configuration conf = new JobConf();
FileSystem raw = new RawLocalFileSystem();
raw.setConf(conf);
FileSystem fs = new LocalFileSystem(raw);
==================
to
========================
Configuration conf = new JobConf();
FileSystem fs = new DistributedFileSystem();
=======================
Speaking as a Hadoop developer ... You should do neither, i.e. you
should not instantiate explicitly any FileSystem implementations. There
are many reasons for this (object pooling, cleanup, caching, etc).
The canonical idiom for this is the following:
FileSystem fs = FileSystem.get(conf);
This way you get either a local FS, or HDFS, or Amazon S3, or KFS, or
whatever else is configured as the default filesystem. The benefit is
obvious - you don't have to change the code if your configuration
changes, i.e. you can transparently move your application from local FS
to DFS or S3. Some FS implementations may use pooling, which happens
behind the scenes if you use the above.
If you really, really need a local fs, you should use the following idiom:
LocalFileSystem localFS = FileSystem.getLocal(conf);
Depending on the configuration (and Hadoop version) you could get
different subclasses of a local FS.
Now, what to do if you use something (e.g. HDFS) by default, but you
want to make sure that you retrieve some resource that resides on
specific other FS? You should use a fully qualified URI when
constructing a Path, i.e. a URI that also contains a schema.
Example:
Path localPath = new Path("file:///etc/hosts");
Path hdfsPath = new Path("hdfs://namenode:9000/user/data/file");
localPath will use a LocalFileSystem, no matter what FS is the default,
and hdfsPath will use DistributedFileSystem that can be reached at the
host "namenode" and port 9000, no matter what is the current FS
configuration.
And finally - to learn what is the current FileSystem that a Path refers
to, do the following:
Path unqualified = new Path("/etc/hosts");
FileSystem fs = unqualified.getFileSystem(conf);
You can also make a fully qualified path from a path that is missing
explicit schema, and may be relative to your current working directory:
Path unqualified = new Path("test");
Path qualified = unqualified.makeQualified(fs);
If your cwd=/home/nutch and your file system is local, then
qualified.toString() would give "file:///home/nutch/test".
Hope this helps ...
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
