HDFS is designed with Hadoop in mind, so there are certain advantages (e.g. performance, reliability, and ease of use) to using HDFS for Hadoop. However, it's not required. For example, when you run Hadoop in standalone mode, it just uses the file system on your local machine. When you run it on Amazon AWS, it can use S3 as a file system.
On Tue, Apr 28, 2009 at 6:15 AM, Adam Retter <adam.ret...@landmark.co.uk>wrote: > > > Each document processing is independent and can be processed > > parallelly, so that part could be done in a map reduce job. > > Now whether it suits this use case depends on rate at which new > > URI's are discovered for processing and acceptable delay in processing > > of a document. The way I see it you can batch the URI's > > and input that to mapreduce job. Each mapper can work on sublist of > URIs. > > You can choose to make DB inserts from mapper itself. In that case > > you can set no of reducers to 0. Otherwise if batching of the queries > > is an option then you can consider making batch inserts in reducer. It > > will help in reducing load on DB. > > So I don't have to use HDFS at all when using Hadoop? > > > > Registered Office: 7 Abbey Court, Eagle Way, Sowton, Exeter, Devon, EX2 7HY > Registered Number 2892803 Registered in England and Wales > > This email has been scanned by the MessageLabs Email Security System. > For more information please visit http://www.messagelabs.com/email > > The information contained in this e-mail is confidential and may be subject > to > legal privilege. If you are not the intended recipient, you must not use, > copy, > distribute or disclose the e-mail or any part of its contents or take any > action in reliance on it. If you have received this e-mail in error, please > e-mail the sender by replying to this message. All reasonable precautions > have > been taken to ensure no viruses are present in this e-mail. Landmark > Information > Group Limited cannot accept responsibility for loss or damage arising from > the > use of this e-mail or attachments and recommend that you subject these to > your virus checking procedures prior to use. > >
