I would concur with the above. Correct me if I'm wrong, but the paradigm of Hadoop/Nutch is such that it needs local, commodity machines on local racks. You should bring the computation to the data, not the other way around. This generally precludes SANs, even with fiber channel connectivity :)
On Tue, Mar 11, 2008 at 1:16 PM, <[EMAIL PROTECTED]> wrote: > I haven't followed this thread, but my little piece of advice is to > reconsider using NFS (and consider using your SAN over Fiber). > Reading/writing ver NFS is slooow both compared to writing/reading to a local > disk and to a remote SAN over FC. > > Otis > -- > Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch > > > > ----- Original Message ---- > From: Developer Developer <[EMAIL PROTECTED]> > To: [email protected] > Sent: Monday, March 10, 2008 11:29:59 AM > Subject: Re: Setting nutch/hadopp multi node environment on a SAN device. > > Hi folks, > > any more comments from hadoop experts ? > > On Sun, Mar 9, 2008 at 7:56 AM, Developer Developer <[EMAIL PROTECTED]> > wrote: > > > it is nfs mount > > > > > > On Sat, Mar 8, 2008 at 9:07 PM, Dennis Kubes <[EMAIL PROTECTED]> wrote: > > > > > How is the san accessed, as a network drive, a special protocol? > > > > > > Dennis > > > > > > Developer Developer wrote: > > > > Any comments? > > > > > > > > On Sat, Mar 8, 2008 at 1:36 PM, Developer Developer < > > > [EMAIL PROTECTED]> > > > > wrote: > > > > > > > >> Hello Friends, > > > >> > > > >> > > > >> The tutorial @ http://wiki.apache.org/nutch/NutchHadoopTutorial says > > > >> don't use DFS on an NFS mount (this would be pretty stupid anyway).. > > > >> > > > >> I am setting up multi node nutch/hadoop environment with lots of > > > storage > > > >> available on a SAN device. I tried running nutch/hadoop with DFS file > > > system > > > >> pointed to a directory on the SAN drive which was mounted as NFS . It > > > failed > > > >> ( expected ?). > > > >> > > > >> How do I make it work ? In other words how do I make nutch/hadoop > > > node > > > >> stores its content on a SAN drive directly ? Is it possible ? If not > > > what > > > >> is the work around ? Do I have to make lots of local hard drive > > > available > > > >> for crawling and then copy it to the NFS drive when the crawl > > > completes ? > > > >> That would be very stupid solution ! > > > >> > > > >> Thanks ! > > > >> > > > >> > > > >> > > > > > > > > > > > > > > >
