Re: zookeeper for gearman?
Thomas, I've looked at integrating the two, so far as to download the gearman source and examine it a bit. I didn't see a huge near-term win implementing a plugin as gearman already has support for drizzle/memcached/sqlite4/pq. While ZK could be used to provide highly reliable/available persistence it wouldn't really add anything over these other options (assuming these other options are configured to be highly reliable). Obviously if someone is already using ZK then there is the benefit of not having to add an additional persistence component, so that is one plus. Longer term it did seem like ZK could benefit gearman by providing support for job server failover. If I understand the way gearman job servers work, even though the jobs are stored persistently the failed gearman server must be restarted (or another to take it's place) and re-read the persisted jobs before those jobs can be made available to workers again. ZK could facilitate this, perhaps even being used to re-distribute the load btw the available job servers (those still active). This was one concrete idea I had, I'm sure ZK could be applied in other areas as well. Patrick Thomas Koch wrote: CC to zookeeper-user Hi, I've not keeped myself up to date on gearman development in the last weeks (months) since I've been occupied with other duties, mostly the introduction of hadoop[1] in our company. One subproject of hadoop is zookeeper[2], "a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.". One of the documented use cases of zookeeper is a distributed queue[3]. (However this document doesn't seem to be that well written.) I woundered if anyone from the gearman project has already heard of zookeeper and eventually considered a gearman implementation on top of it? It shouldn't be that hard and it would get you replication for free. Maybe I'll try ones my current project is done. :-) [1] http://hadoop.apache.org/ [2] http://hadoop.apache.org/zookeeper/ [3] http://hadoop.apache.org/zookeeper/docs/current/zookeeperTutorial.html Best regards, Thomas Koch, http://www.koch.ro
Re: Ruby client binding for ZooKeeper now available.
I'd be happy to contribute it. This is a fork of someone else's client, which I forward-ported and added much functionality to. It is slightly incomplete, I will be finishing it soon. The current client is MIT licensed. Is the ASF license compatible? If not I can ask the original author to relicense his code, or rewrite the client. Eric On Feb 19, 2010, at 4:27 PM, Jeff Hammerbacher wrote: > Are there plans to integrate this work with the Apache project? > https://issues.apache.org/jira/browse/ZOOKEEPER-661. > > On Fri, Feb 19, 2010 at 10:23 AM, Patrick Hunt wrote: > >> Sorry if this is a dup for those of you following me on twitter ( >> http://twitter.com/phunt) but I wanted to let you know that twitter (the >> company) has contributed a Ruby client binding for ZooKeeper. You can learn >> more about the gem here: http://bit.ly/b9VB6k >> >> Regards, >> >> Patrick >>
Re: Ruby client binding for ZooKeeper now available.
Are there plans to integrate this work with the Apache project? https://issues.apache.org/jira/browse/ZOOKEEPER-661. On Fri, Feb 19, 2010 at 10:23 AM, Patrick Hunt wrote: > Sorry if this is a dup for those of you following me on twitter ( > http://twitter.com/phunt) but I wanted to let you know that twitter (the > company) has contributed a Ruby client binding for ZooKeeper. You can learn > more about the gem here: http://bit.ly/b9VB6k > > Regards, > > Patrick >
Re: Zab documentation ?
Ah, good idea, I put it up on the Zab page, thanks. http://wiki.apache.org/hadoop/ZooKeeper/Zab Patrick Sudipto Das wrote: I found this paper a bit more detailed, and helpful. *A simple totally ordered broadcast protocol http://portal.acm.org/citation.cfm?id=1529978 * Best Regards Sudipto Das PhD Candidate CS @ UCSB Santa Barbara, CA 93106, USA http://www.cs.ucsb.edu/~sudipto On Fri, Feb 19, 2010 at 9:08 AM, Patrick Hunt wrote: Mahadev can you put a link to that up on the wiki? Patrick Mahadev Konar wrote: Hi Kay, There is a paper "Brief Announcement Zab A Practical Totally Ordered Broadcast Protocol." in DISC 2009 By ben and flavio which has the details of zab in it. Thanks mahadev On 2/18/10 6:07 PM, "Kay Kay" wrote: I am trying to figure out more information about Zab and came across the wiki here - http://wiki.apache.org/hadoop/ZooKeeper/Zab . Can you help add more information about the same / documentation available as appropriate .
Re: Zab documentation ?
I found this paper a bit more detailed, and helpful. *A simple totally ordered broadcast protocol http://portal.acm.org/citation.cfm?id=1529978 * Best Regards Sudipto Das PhD Candidate CS @ UCSB Santa Barbara, CA 93106, USA http://www.cs.ucsb.edu/~sudipto On Fri, Feb 19, 2010 at 9:08 AM, Patrick Hunt wrote: > Mahadev can you put a link to that up on the wiki? > > Patrick > > > Mahadev Konar wrote: > >> Hi Kay, >> There is a paper "Brief Announcement Zab A Practical Totally Ordered >> Broadcast Protocol." in >> DISC 2009 >> >> By ben and flavio which has the details of zab in it. >> >> Thanks >> mahadev >> >> >> On 2/18/10 6:07 PM, "Kay Kay" wrote: >> >> I am trying to figure out more information about Zab and came across the >>> wiki here - >>> >>> http://wiki.apache.org/hadoop/ZooKeeper/Zab . >>> >>> Can you help add more information about the same / documentation >>> available as appropriate . >>> >>> >>
Ruby client binding for ZooKeeper now available.
Sorry if this is a dup for those of you following me on twitter (http://twitter.com/phunt) but I wanted to let you know that twitter (the company) has contributed a Ruby client binding for ZooKeeper. You can learn more about the gem here: http://bit.ly/b9VB6k Regards, Patrick
Re: Zab documentation ?
Mahadev can you put a link to that up on the wiki? Patrick Mahadev Konar wrote: Hi Kay, There is a paper "Brief Announcement Zab A Practical Totally Ordered Broadcast Protocol." in DISC 2009 By ben and flavio which has the details of zab in it. Thanks mahadev On 2/18/10 6:07 PM, "Kay Kay" wrote: I am trying to figure out more information about Zab and came across the wiki here - http://wiki.apache.org/hadoop/ZooKeeper/Zab . Can you help add more information about the same / documentation available as appropriate .
zookeeper for gearman?
CC to zookeeper-user Hi, I've not keeped myself up to date on gearman development in the last weeks (months) since I've been occupied with other duties, mostly the introduction of hadoop[1] in our company. One subproject of hadoop is zookeeper[2], "a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.". One of the documented use cases of zookeeper is a distributed queue[3]. (However this document doesn't seem to be that well written.) I woundered if anyone from the gearman project has already heard of zookeeper and eventually considered a gearman implementation on top of it? It shouldn't be that hard and it would get you replication for free. Maybe I'll try ones my current project is done. :-) [1] http://hadoop.apache.org/ [2] http://hadoop.apache.org/zookeeper/ [3] http://hadoop.apache.org/zookeeper/docs/current/zookeeperTutorial.html Best regards, Thomas Koch, http://www.koch.ro
