Stefan, Thanks for the links. Good reading, i'll see if i can't come up with any code.
The only concern i would have is a distributed file system would almost inherently limit us to a single platform. (linux i would assume) I'll see if i can't come across any more good resources on this topic. I am interested in the Distributed WebDB as it is now to spread the load, however i think something that can scale from managing distributed indexes to distributed webdb would be the best approach :) -byron --- Stefan Groschupf <[EMAIL PROTECTED]> wrote: > Well I have set of interesting resources related to > distributed fs: > > http://www.coda.cs.cmu.edu/doc/html/index.html > http://pl.atyp.us/pm/comments.php?id=552_0_1_0_C16 > > WebStore: Efficient Storage and Access of the Web > Daniel Golovin, Sandeep Pandey Computer Science > Department Carnegie > Mellon University Pittsburgh, PA 15213 {dgolovin, > [EMAIL PROTECTED] > > The Google File System Sanjay Ghemawat, Howard > Gobioff, and Shun-Tak > Leung > > Doug already post it: > http://www.lustre.org/whatislustre.html > > > Sorrento: A Self-Organizing Storage Cluster for > Parallel Data-Intensive > Applications Hong Tang, Aziz Gulbeden, Jingyu Zhou, > Lingkun Chu, and > Tao Yang Department of Computer Science University > of California, Santa > Barbara, CA 93106 {htang, gulbeden, jzhou, lkchu, > [EMAIL PROTECTED] > > I found some more but that wasn't that interesting > from a first look > over. > > HTH > Stefan > > > Am 15.06.2004 um 00:14 schrieb Abdul Chaudhry: > > > > > From what I have noticed , implementing your own > > buffer manager, disk cache or using NIO or > whatever > > you want to call it, does improve performance in > > "some" situations. > > It avoids using the Operating System Disk Cache > which > > has some overheads. > > In the database world it can sometimes yield > higher > > than 30% performance gains - assuming your > index'es > > are much larger than available ram. > > This would make sense for a smaller implementation > of > > nutch , say, for intranet/enterprise search rather > > than internet search. > > > > Anyway, someone posted that a distributed > filesystem > > would be better and I guess that's the reason > google > > uses its own distributed filesystem. > > Here is a paper that I'm sure someone else posted > here > > at some point. > > > http://www.cs.rochester.edu/sosp2003/papers/p125-ghemawat.pdf > > > > -- Ab's > > > > --- Byron Miller <[EMAIL PROTECTED]> wrote: > >> Has anyone done any testing of converting > >> read/writes > >> & io to NIO or do you think that wouldn't offer > much > >> gain? > >> > >> > >> > > > ------------------------------------------------------- > >> This SF.Net email is sponsored by The 2004 > >> JavaOne(SM) Conference > >> Learn from the experts at JavaOne(SM), Sun's > >> Worldwide Java Developer > >> Conference, June 28 - July 1 at the Moscone > Center > >> in San Francisco, CA > >> REGISTER AND SAVE! http://java.sun.com/javaone/sf > >> Priority Code NWMGYKND > >> _______________________________________________ > >> Nutch-developers mailing list > >> [EMAIL PROTECTED] > >> > > > https://lists.sourceforge.net/lists/listinfo/nutch-developers > >> > > > > > > > > > ------------------------------------------------------- > > This SF.Net email is sponsored by The 2004 > JavaOne(SM) Conference > > Learn from the experts at JavaOne(SM), Sun's > Worldwide Java Developer > > Conference, June 28 - July 1 at the Moscone Center > in San Francisco, CA > > REGISTER AND SAVE! http://java.sun.com/javaone/sf > Priority Code > > NWMGYKND > > _______________________________________________ > > Nutch-developers mailing list > > [EMAIL PROTECTED] > > > https://lists.sourceforge.net/lists/listinfo/nutch-developers > > > > > --------------------------------------------------------------- > enterprise information technology consulting > open technology: http://www.media-style.com > open source: http://www.weta-group.net > open discussion: http://www.text-mining.org > > > > ------------------------------------------------------- > This SF.Net email is sponsored by The 2004 > JavaOne(SM) Conference > Learn from the experts at JavaOne(SM), Sun's > Worldwide Java Developer > Conference, June 28 - July 1 at the Moscone Center > in San Francisco, CA > REGISTER AND SAVE! http://java.sun.com/javaone/sf > Priority Code NWMGYKND > _______________________________________________ > Nutch-developers mailing list > [EMAIL PROTECTED] > https://lists.sourceforge.net/lists/listinfo/nutch-developers > ------------------------------------------------------- This SF.Net email is sponsored by The 2004 JavaOne(SM) Conference Learn from the experts at JavaOne(SM), Sun's Worldwide Java Developer Conference, June 28 - July 1 at the Moscone Center in San Francisco, CA REGISTER AND SAVE! http://java.sun.com/javaone/sf Priority Code NWMGYKND _______________________________________________ Nutch-developers mailing list [EMAIL PROTECTED] https://lists.sourceforge.net/lists/listinfo/nutch-developers
