On Jul 11, 2010, at 1:54 AM, Ryan Chan wrote: > Hello, > > I have a customized crawler project which consists of 4 subsystems. > > a. Links discovery (5+ EC2 instance) > b. Web crawlers (20+ EC2 instance) > c. Links DB (MongoDB x 1) > d. Contents DB (MongoDB x 1) > > > 1. Web crawlers (b) fetch links in batch from Links DB (c) and do the > crawling, and save the result into the Contents DB (d). > 2. Links discovery fetch content in batch from Contents DB (d) and do > the analyze and save in the interested links in Link DB (c) > > > Current system work fine, but I want to explore a more scalable way to > design the system such that (c) and (d) would become the bottom-neck > in the future. > > Seems both (c) and (d) can be replaced by 0mq?
I'm not sure I understand your use-case. 0mq is a transport mechanism. Mongodb is a database. Why would you think that 0mq could replace Mongodb? cr _______________________________________________ zeromq-dev mailing list [email protected] http://lists.zeromq.org/mailman/listinfo/zeromq-dev
