Mahadev Konar:
> Hi Thomas,
>   There are a couple of projects inside Yahoo! that use ZooKeeper as an
> event manager for feed processing.
> I am little bit unclear on your example below. As I understand it-
> 1. There are 1 million feeds that will be stored in Hbase.
> 2. A map reduce job will be run on these feeds to find out which feeds need
> to be fetched.
> 3. This will create queues in ZooKeeper to fetch the feeds
> 4.  Workers will pull items from this queue and process feeds
> Did I understand it correctly? Also, if above is the case, how many queue
> items would you anticipate be accumulated every hour?
Yes. That's exactly what I'm thinking about. Currently one node processes like 
20000 Feeds an hour and we have 5 feed-fetch-nodes. This would mean ~100000 
queue items/hour. Each queue item should carry some meta informations, most 
important the feed items, that are already known to the system so that only 
new items get processed.

Thomas Koch,

Reply via email to