Hi all, I would like to implement a distributed crawl which would be something like this :
- A main machine that would store all nutch database (1) - n machines that would only be used for fetching (because I use specific computations in the fetcher process which are time consuming). (2) After finishing fetching, each of this n machines (2) would directly update the database on the main machine (1). Here my questions : 1 - After reading documentation on NDFS, I'm not sure it can be used in that way. If so, is there a better way to do it? 2 - If it's possible, what happen if two machines (2) try to updatedb on the main machine (1) at the same time? Would it be a synchronized operation? Thanks for your help, -- greg
