Re: znode cversion decreasing?

2010-04-12 Thread Patrick Hunt
Hi Kevin, The server increments a znode's cversion by one each time a change to it's child list is made. Every znode has it's own cversion. It should never decrease. If you delete a znode and create it a new then the cversion is reset for that znode. The cversion also happens to be used for

feed queue fetcher with hadoop/zookeeper/gearman?

2010-04-12 Thread Thomas Koch
Hi, I'd like to implement a feed loader with Hadoop and most likely HBase. I've got around 1 million feeds, that should be loaded and checked for new entries. However the feeds have different priorities based on their average update frequency in the past and their relevance. The feeds (url,

Re: feed queue fetcher with hadoop/zookeeper/gearman?

2010-04-12 Thread Mahadev Konar
Hi Thomas, There are a couple of projects inside Yahoo! that use ZooKeeper as an event manager for feed processing. I am little bit unclear on your example below. As I understand it- 1. There are 1 million feeds that will be stored in Hbase. 2. A map reduce job will be run on these feeds to

Re: feed queue fetcher with hadoop/zookeeper/gearman?

2010-04-12 Thread Thomas Koch
Mahadev Konar: Hi Thomas, There are a couple of projects inside Yahoo! that use ZooKeeper as an event manager for feed processing. I am little bit unclear on your example below. As I understand it- 1. There are 1 million feeds that will be stored in Hbase. 2. A map reduce job will be

Re: feed queue fetcher with hadoop/zookeeper/gearman?

2010-04-12 Thread Patrick Hunt
See this environment http://bit.ly/4ekN8G. Subsequently I used the 3 server setup, each configured with 8gig of heap in the jvm and 4 CPUs/jvm (I think I used 10second session timeouts for this) for some additional testing that I've not written up yet. I was able to run ~500 clients (same test

Re: znode cversion decreasing?

2010-04-12 Thread Kevin Webb
On Mon, 12 Apr 2010 09:27:46 -0700 Mahadev Konar maha...@yahoo-inc.com wrote: HI Kevin, The cversion should be monotonically increasing for the the znode. It would be a bug if its not. Can you please elaborate in which cases you are seeing the cversion decreasing? If you can reproduce with

Re: znode cversion decreasing?

2010-04-12 Thread Kevin Webb
On Mon, 12 Apr 2010 14:33:44 -0700 Mahadev Konar maha...@yahoo-inc.com wrote: Hi Kevin, Thanks for the info. Could you cut and paste the code you are using that prints the view info? That would help. We can then create a jira and follow up on that. Also, a zookeeper client can never go

Re: znode cversion decreasing?

2010-04-12 Thread Patrick Hunt
We did have a case where the user setup 3 servers, each was standalone. :-) Doesn't look like that's the problem here though given you only specify 1 server in the connect string (although as mahadev mentioned you don't need to worry about that aspect). After it goes 7-11-9, does it ever go

Re: znode cversion decreasing?

2010-04-12 Thread Patrick Hunt
On 04/12/2010 03:58 PM, Kevin Webb wrote: On Mon, 12 Apr 2010 15:09:20 -0700 Patrick Huntph...@apache.org wrote: We did have a case where the user setup 3 servers, each was standalone. :-) Doesn't look like that's the problem here though given you only specify 1 server in the connect string

Re: znode cversion decreasing?

2010-04-12 Thread Patrick Hunt
Probably reaching for straws but could you print path, just to confirm it's what you know it is? Patrick On 04/12/2010 02:53 PM, Kevin Webb wrote: On Mon, 12 Apr 2010 14:33:44 -0700 Mahadev Konarmaha...@yahoo-inc.com wrote: Hi Kevin, Thanks for the info. Could you cut and paste the code