the google file system support multiple clients writing to one file ( or may chunk).
In case we porting nutch functionality to map and reduce this would be very useful as well.
For example a set of crawlers writing to one 'segment file'.
Does the actually implementation of ndfs support this functionality as well?
NDFS does not currently implement multiple writers to the same file, nor are there any plans to implement that at present.
I don't think it would make crawling much simpler. Long-term, most database operations will be map/reduce-based, including fetchlist generation and db update. Map/reduce naturally accepts sets of files as input and produces sets of files as output. So having each fetcher agent read and write separate segment files works well with map/reduce.
In the GFS paper I think the multiple writers feature is used to implement queues. Perhaps someday we will discover a critical need for queues of this sort, but, right now we want to spare the complexity it would add to the file system.
Doug
------------------------------------------------------- The SF.Net email is sponsored by: Beat the post-holiday blues Get a FREE limited edition SourceForge.net t-shirt from ThinkGeek. It's fun and FREE -- well, almost....http://www.thinkgeek.com/sfshirt _______________________________________________ Nutch-developers mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/nutch-developers
