Re: ZKClient
Thanks Travis, I've slated this for 3.4.0, I think it would be useful to add more examples so feel free to add more if you have any ideas for useful ones. For future reference, we ask that contributions come in the form of a patch: http://wiki.apache.org/hadoop/ZooKeeper/HowToContribute It's fine this time around, but in future it would be helpful. (also click on submit patch link when you are ready for review - that pushes it through the process, incl automated testing/verification, that's why we ask for a patch off the root btw) Thanks! Patrick On 05/04/2010 04:00 PM, Travis Crawford wrote: On Tue, May 4, 2010 at 3:45 PM, Ted Dunningted.dunn...@gmail.com wrote: Travis, Attachments are stripped from this mailing list. Can you file a JIRA and put your attachment on that instead? Here is a link to get you started: https://issues.apache.org/jira/browse/ZOOKEEPER Whoops. Filed: https://issues.apache.org/jira/browse/ZOOKEEPER-765 --travis On Tue, May 4, 2010 at 3:43 PM, Travis Crawfordtraviscrawf...@gmail.comwrote: Attached is a skeleton application I extracted from a script I use -- perhaps we could add this as a recipe? If there are issues I'm more than happy to fix them, or add more comments, whatever. It took a while to figure this out and I'd love to save others that time in the future. --travis On Tue, May 4, 2010 at 3:16 PM, Mahadev Konarmaha...@yahoo-inc.com wrote: Hi Adam, I don't think zk is very very hard to get right. There are exmaples in src/recipes which implements locks/queues/others. There is ZOOKEEPER-22 to make it even more easier for application to use. Regarding re registration of watches, you can deifnitely write code and submit is as a part of well documented contrib module which lays out the assumptions/design of it. It could very well be useful for others. Its just that folks havent had much time to focus on these areas as yet. Thanks mahadev On 5/4/10 2:58 PM, Adam Rosiena...@rosien.net wrote: I use zkclient in my work at kaChing and I have mixed feelings about it. On one hand it makes easy things easy which is great, but on the other hand I very few ideas what assumptions it makes under the hood. I also dislike some of the design choices such as unchecked exceptions, but that's neither here nor there. It would take some extensive documentation work by the authors to really enumerate the model and assumptions, but the project doesn't seem to be active (either from it being adequate for its current users or just inactive). I'm not sure I could derive the assumptions myself. I'm a bit frustrated that zk is very, very hard to really get right. At a project level, can't we create structures to avoid most of these errors? Can there be a standard model with detailed assumptions and implementations of all the recipes? How can we start this? Is there something that makes this too hard? I feel like a recipe page is a big fail; wouldn't an example app that uses locks and barriers be that much more compelling? For the common FAQ items like you need to re-register the watch, can't we just create code that implements this pattern? My goal is to live up to the motto: a good API is impossible to use incorrectly. .. Adam On Tue, May 4, 2010 at 2:21 PM, Ted Dunningted.dunn...@gmail.com wrote: In general, writing this sort of layer on top of ZK is very, very hard to get really right for general use. In a simple use-case, you can probably nail it but distributed systems are a Zoo, to coin a phrase. The problem is that you are fundamentally changing the metaphors in use so assumptions can come unglued or be introduced pretty easily. One example of this is the fact that ZK watches *don't* fire for every change but when you write listener oriented code, you kind of expect that they will. That makes it really, really easy to introduce that assumption in the heads of the programmer using the event listener library on top of ZK. Another example is how the atomic get content/set watch call works in ZK is easy to violate in an event driven architecture because the thread that watches ZK probably resets the watch. If you assume that the listener will read the data, then you have introduced a timing mismatch between the read of the data and the resetting of the watch. That might be OK or it might not be. The point is that these changes are subtle and tricky to get exactly right. On Tue, May 4, 2010 at 1:48 PM, Jonathan Holloway jonathan.hollo...@gmail.com wrote: Is there any reason why this isn't part of the Zookeeper trunk already?
Re: ZKClient
While I agree DS is hard, I don't think we should lose the useful feedback given by Jonathan/Adam - that getting started with ZK is challenging and can be frustrating. We need to learn from this feedback and create some action items to address. One of the main things I've heard so far that we can act on today is that we should add examples/docs to round things out. I agree with this. Also the recipes page should be updated to point to the recipe implementations we recently added to the release. One suggestion, it's much easier for new contributors/users to contribute to the examples than it is to jump into ZK core development. New users feel the pain most directly (recently), I'd encourage you to contribute back by creating an example or two. I'm sure the existing contributors would be happy to work with you to get them committed and released. Regards, Patrick On 05/04/2010 03:43 PM, Ted Dunning wrote: Creating recipes is a great thing, but that doesn't change the fact that distributed systems are inherently a bit tricky, especially if you start with the assumption (as many people do) that Peter Deutsch was wrong. One of the great contributions of MapReduce style parallelism or the java concurrent package is that it provides safe trails in a pretty scary forest. Good Zookeeper recipes could provide similar guidance with similar positive effects. On Tue, May 4, 2010 at 3:24 PM, Adam Rosiena...@rosien.net wrote: I'll check it out, but it is repeated in this list and on the web site that it's not as easy as it seems. I just want to enumerate the failure points and create abstractions to avoid them. .. Adam On Tue, May 4, 2010 at 3:16 PM, Mahadev Konarmaha...@yahoo-inc.com wrote: Hi Adam, I don't think zk is very very hard to get right. There are exmaples in src/recipes which implements locks/queues/others. There is ZOOKEEPER-22 to make it even more easier for application to use. Regarding re registration of watches, you can deifnitely write code and submit is as a part of well documented contrib module which lays out the assumptions/design of it. It could very well be useful for others. Its just that folks havent had much time to focus on these areas as yet. Thanks mahadev On 5/4/10 2:58 PM, Adam Rosiena...@rosien.net wrote: I use zkclient in my work at kaChing and I have mixed feelings about it. On one hand it makes easy things easy which is great, but on the other hand I very few ideas what assumptions it makes under the hood. I also dislike some of the design choices such as unchecked exceptions, but that's neither here nor there. It would take some extensive documentation work by the authors to really enumerate the model and assumptions, but the project doesn't seem to be active (either from it being adequate for its current users or just inactive). I'm not sure I could derive the assumptions myself. I'm a bit frustrated that zk is very, very hard to really get right. At a project level, can't we create structures to avoid most of these errors? Can there be a standard model with detailed assumptions and implementations of all the recipes? How can we start this? Is there something that makes this too hard? I feel like a recipe page is a big fail; wouldn't an example app that uses locks and barriers be that much more compelling? For the common FAQ items like you need to re-register the watch, can't we just create code that implements this pattern? My goal is to live up to the motto: a good API is impossible to use incorrectly. .. Adam On Tue, May 4, 2010 at 2:21 PM, Ted Dunningted.dunn...@gmail.com wrote: In general, writing this sort of layer on top of ZK is very, very hard to get really right for general use. In a simple use-case, you can probably nail it but distributed systems are a Zoo, to coin a phrase. The problem is that you are fundamentally changing the metaphors in use so assumptions can come unglued or be introduced pretty easily. One example of this is the fact that ZK watches *don't* fire for every change but when you write listener oriented code, you kind of expect that they will. That makes it really, really easy to introduce that assumption in the heads of the programmer using the event listener library on top of ZK. Another example is how the atomic get content/set watch call works in ZK is easy to violate in an event driven architecture because the thread that watches ZK probably resets the watch. If you assume that the listener will read the data, then you have introduced a timing mismatch between the read of the data and the resetting of the watch. That might be OK or it might not be. The point is that these changes are subtle and tricky to get exactly right. On Tue, May 4, 2010 at 1:48 PM, Jonathan Holloway jonathan.hollo...@gmail.com wrote: Is there any reason why this isn't part of the Zookeeper trunk already?
Re: GSoC 2010: ZooKeeper Failure Detector Module
Hi Abmar, cool project! I wrote an implementation of the Phi Accrual Detector which I would be happy to donate. It has the nice property that its parameters are based on expectation odds rather than something more arbitrary like # retries. You can steal it at http://github.com/arosien/failure. It works well but let me know if you have any suggestions. .. Adam On Wed, May 5, 2010 at 1:28 PM, Abmar Barros abma...@gmail.com wrote: Hello all, As Henry Robinson first introduced me, my name is Abmar Barros and I am one of the GSoC accepted students for ZooKeeper. In the next 4 months, I will be being mentored by Flavio Junqueira in order to improve ZooKeeper's failure detector module. In this project I will abstract the failure detection code to a separate module, in order to make the code cleaner and to ease the implementation of new methods. Also, I will implement several failure detectors and I will compare their appropriateness for ZooKeeper. I have created a page ( http://wiki.apache.org/hadoop/ZooKeeper/GSoCFailureDetector) for this project in the ZooKeeper wiki and also there is a related JIRA issue ( https://issues.apache.org/jira/browse/ZOOKEEPER-702). Suggestions and specific requirements are always welcome :) Best regards -- Abmar Barros MSc candidate on Computer Science at Federal University of Campina Grande - www.ufcg.edu.br OurGrid Team Member - www.ourgrid.org Paraíba - Brazil
