Please review ZOOKEEPER-45 Restructure the SVN repository after initial import
Update the jira with your comments: https://issues.apache.org/jira/browse/ZOOKEEPER-45 Patrick
need your help - review patch to fix hudson tests
Please review the fix_solaris_tests.patch file and provide feedback so that I can commit and get hudson running. See comments here: https://issues.apache.org/jira/browse/ZOOKEEPER-45?focusedCommentId=12608933#action_12608933 Thanks, Patrick
REVIEW: skeleton of ZK ASF docs and site
Please review/comment on the following Jiras: https://issues.apache.org/jira/browse/ZOOKEEPER-70 https://issues.apache.org/jira/browse/ZOOKEEPER-72 Patrick
FYI: Hadoop social 7/22
We had a great ZK presentation from Ted/Ben last meetup. Members of the ZK team will be attending this month as well if you'd like to talk f2f: http://upcoming.yahoo.com/event/869166/ Patrick
Re: Thrift interface
FUSE support is tracked by the following JIRA https://issues.apache.org/jira/browse/ZOOKEEPER-25 a patch is attached to the JIRA containing the feature. It's a big(ger) set of code so I've given it time for review. Should be committed soon and will be part of 3.0. I started working on script support just before we got accepted to Apache, which put that onto the back burner for a bit. We have a design and I implemented a very basic prototype using SWIG that supports perl/python/ruby/php/tcl. I'll be picking scripting support up again once the Apache migration is mostly complete. Patrick Benjamin Reed wrote: Hi Chris, We have talked about using Thrift for our serialization layer (I thought we had a Jira opened for it, but I cannot find it). Thrift RPC is a different story. You would lose watch callbacks. I think it would also be hard to support sessions and thus ephemeral nodes. Finally, your client and server throughput would be quite low since clients wouldn't be able to pipeline requests (it appears you can only have one request outstanding at a time) and servers have to dedicate a thread to service a request. ZooKeeper clients are more than simple RPC dispatchers they also do session maintenance and completion processing, so you need something a bit more sophisticated at the client. I assume you want this to get bindings for other languages. We have SWIG and FUSE bindings for this. If you really want thrift, I think there is a combo solution: implement a standalone service, thrift server and zk client, and run it on each ZooKeeper server. ben Chris Smith wrote: Hi, I'm currently thinking about implementing a Thrift interface for ZooKeeper, and wanted to get some opinions on it before I start. I can basically see two possible ways to implement it: * A standalone service which implements both a ZooKeeper client and a Thrift server, and basically acts as a proxy between the two * Integrate the Thrift server with the ZooKeeper server, so that ZK servers themselves handle the Thrift connections I think the former is probably nicer from a development stand-point, but the latter seems better from an end-user stand-point. Has anyone else thought about doing this (or actually done it!), and does anyone have any preferences/comments with regard to the above options? Cheers, Chris Smith
Re: Recipe contrib -- was Re: [PATCH] a simple Leader Election or exclusive Write Lock protocol/policy
I was thinking all the recipe implementations are in a single jar. zookeeper-recipes.jar otw it's going to be a pain to carry around the right jars - esp if one recipe depends on another. We saw similar issue with Pig UDF contribs (originally there were multiple jars). Patrick Doug Cutting wrote: Patrick Hunt wrote: should we have separate contrib subdirs, one for each recipe or all recipes together? What about shared code, common code for implementing a recipe? Seems a little too much to separate them all at the top level, rather than separating them in packages Do you want to package them together, in a single jar, or separately, each in their own jar? A tree per jar is standard. Doug
Re: Recipe contrib -- was Re: [PATCH] a simple Leader Election or exclusive Write Lock protocol/policy
Doug Cutting wrote: Patrick Hunt wrote: my original intent was to have it as zookeeper/trunk/src/contrib/... where this directory will have not only recipes but also other contributions. +1 That's what I meant too. And since you've said you want to package these all as a single jar, then there should just be a single tree per language, right? Yes, that's right. zookeeper/trunk/src/contrib/recipes/src/java/org/apache/zookeeper/recipes/locking/ zookeeper/trunk/src/contrib/recipes/src/c++/locking/ Right. I don't see the strong connection between releases and new features. Am I missing something? I think that the issue is implied connotation of contrib. We were thinking of contrib as interesting stuff that users might want to re-use/share but not core to zookeeper, we'll carry it around so that users know where to find it, if it gets broken and no one fixes it it won't hold up the release If I understand correctly you are saying there is no middle ground. Patrick
New Contributors to ASF ZooKeeper
Welcome Hiram Chirino and James Strachan to the ASF ZooKeeper contributor list. Patrick
Re: New Contributors to ASF ZooKeeper
Anthony Urso has also been added to the list of contributors. Patrick Patrick Hunt wrote: Welcome Hiram Chirino and James Strachan to the ASF ZooKeeper contributor list. Patrick
Re: Website
Hi Hiram, see ZOOKEEPER-70 72. We'll be following Hadoop core/hbase wrt the docs/site/wiki. Specifically using Apache Forrest for the documentation, which also generates the site. We're in the process of moving the sourceforge wiki content over from SF to apache. I'm been working with a couple of the hadoop core team members getting their input re the site, I had actually planned to push something live today. Additional complexity is due to the fact that on SF we had all docs on the wiki and were not including the api docs in the release. Apache requires us to version our documentation proper along with the code (so has to be in SVN and not on the wiki) and we also have to include the api docs (as well as regular docs) along with each release which we are not doing on SF. Patrick Hiram Chirino wrote: Is the ZooKeeper website getting moved over to Apache? I think a good website is critical for building a community around the project. Do you guys want to generate it from wiki content? For example, the http://activemq.apache.org/ site is generated from the wiki content at http://cwiki.apache.org/confluence/display/ACTIVEMQ/Index If so I think I can create the confluence space for you guys so that we can get started with that.
Re: Website
I cloned core/hbase subprojects, so we do what's currently done by other hadoop subs. The issue has come up in the past, might be a good idea at some point but might also add even more complexity to already pretty complex process (see howtorelease page). I think we should get things setup similar to existing subs (still alot of work to be done converting the wiki over to apache wiki/docs) and review at some point in the future. Howtocontrib/howtorelease/faq is already on the wiki: http://wiki.apache.org/hadoop/ZooKeeper mailinglists/news/etc... are part of the docs (not wiki), probably more convention than anything. mailinglist never changes and I believe news (and prolly some others) is part of docs in order to be versioned, but also to enable users who have checked out the code to have direct access . Patrick Hiram Chirino wrote: On Tue, Jul 22, 2008 at 1:24 PM, Patrick Hunt [EMAIL PROTECTED] wrote: Hi Hiram, see ZOOKEEPER-70 72. We'll be following Hadoop core/hbase wrt the docs/site/wiki. Specifically using Apache Forrest for the documentation, which also generates the site. We're in the process of moving the sourceforge wiki content over from SF to apache. I'm been working with a couple of the hadoop core team members getting their input re the site, I had actually planned to push something live today. Additional complexity is due to the fact that on SF we had all docs on the wiki and were not including the api docs in the release. Apache requires us to version our documentation proper along with the code (so has to be in SVN and not on the wiki) and we also have to include the api docs (as well as regular docs) along with each release which we are not doing on SF. Yep. BTW you can still take a hybrid approach where the general site information (stuff like how to contirbute, mailing lists, news, etc. etc.) is wiki driven which then links to release documentation directories which are generated from svn sources during a release. But either way is good. Good to know the site is coming soon! Patrick Hiram Chirino wrote: Is the ZooKeeper website getting moved over to Apache? I think a good website is critical for building a community around the project. Do you guys want to generate it from wiki content? For example, the http://activemq.apache.org/ site is generated from the wiki content at http://cwiki.apache.org/confluence/display/ACTIVEMQ/Index If so I think I can create the confluence space for you guys so that we can get started with that.
ASF ZooKeeper site is now live.
The basic skeleton is now live http://hadoop.apache.org/zookeeper/ Still a bunch of holes that I'm working on. In particular API docs need to be added and the hadoop main site needs to be updated to include pointers to zk. Please review asap and let me know if you see issues -- update ZOOKEEPER-72 with comments/findings. Patrick
Re: Website
The Hadoop project, including ZooKeeper, is currently using the Apache wiki. There are no plans at this time to use confluence: http://wiki.apache.org/hadoop/ZooKeeper Patrick Hiram Chirino wrote: On Tue, Jul 22, 2008 at 2:54 PM, Patrick Hunt [EMAIL PROTECTED] wrote: I cloned core/hbase subprojects, so we do what's currently done by other hadoop subs. The issue has come up in the past, might be a good idea at some point but might also add even more complexity to already pretty complex process (see howtorelease page). I think we should get things setup similar to existing subs (still alot of work to be done converting the wiki over to apache wiki/docs) and review at some point in the future. Howtocontrib/howtorelease/faq is already on the wiki: http://wiki.apache.org/hadoop/ZooKeeper mailinglists/news/etc... are part of the docs (not wiki), probably more convention than anything. mailinglist never changes and I believe news (and prolly some others) is part of docs in order to be versioned, but also to enable users who have checked out the code to have direct access . That's cool too.. If you are going to use a Wiki it might be better to use the confluence one just cause it can easily be skinned to have the main site's look and feel. Plus it has a bunch of cool macros available to import code examples from svn or bring in a table of issues from JIRA (for example http://openejb.apache.org/ejb-3-roadmap.html ) etc. Regards, Hiram Patrick Hiram Chirino wrote: On Tue, Jul 22, 2008 at 1:24 PM, Patrick Hunt [EMAIL PROTECTED] wrote: Hi Hiram, see ZOOKEEPER-70 72. We'll be following Hadoop core/hbase wrt the docs/site/wiki. Specifically using Apache Forrest for the documentation, which also generates the site. We're in the process of moving the sourceforge wiki content over from SF to apache. I'm been working with a couple of the hadoop core team members getting their input re the site, I had actually planned to push something live today. Additional complexity is due to the fact that on SF we had all docs on the wiki and were not including the api docs in the release. Apache requires us to version our documentation proper along with the code (so has to be in SVN and not on the wiki) and we also have to include the api docs (as well as regular docs) along with each release which we are not doing on SF. Yep. BTW you can still take a hybrid approach where the general site information (stuff like how to contirbute, mailing lists, news, etc. etc.) is wiki driven which then links to release documentation directories which are generated from svn sources during a release. But either way is good. Good to know the site is coming soon! Patrick Hiram Chirino wrote: Is the ZooKeeper website getting moved over to Apache? I think a good website is critical for building a community around the project. Do you guys want to generate it from wiki content? For example, the http://activemq.apache.org/ site is generated from the wiki content at http://cwiki.apache.org/confluence/display/ACTIVEMQ/Index If so I think I can create the confluence space for you guys so that we can get started with that.
Re: do the test cases work for anyone else?
I'm on ubuntu (hardy heron) and they work. Our CI machine has intermittent failures (solaris x86): http://hudson.zones.apache.org/hudson/view/ZooKeeper/job/ZooKeeper-trunk/ there's some timing issue, what you're seeing is probably related to: https://issues.apache.org/jira/browse/ZOOKEEPER-61 Frankly tests and docs are both areas that ZooKeeper could use _a lot_ of care and feeding. Tests in particular could use some refactoring and a better implementation for launching/testing/stopping client/server tests. As you're able to reproduce the issue reliably would you like to take on 61? Feel free to assign to yourself if so. Regards, Patrick James Strachan wrote: I've always had some tests failing on most boxes I try; I wasn't sure if everyone else got those or if they do work on some platforms? On OS X I get these failures [junit] Test org.apache.zookeeper.test.AsyncTest FAILED [junit] Test org.apache.zookeeper.test.WatcherFuncTest FAILED On a linux box (an EC2 box) I get these failures [junit] Test org.apache.zookeeper.test.ClientTest FAILED [junit] Test org.apache.zookeeper.test.QuorumTest FAILED Maybe they all work on windows? :) I tried adding an explicit forkmode=perTest to the junit and I still get the same results. Can anyone else get the tests to work on linux or OS X?
Re: ZooKeeper community
Good question, according to the PoweredBy wiki not too much ;-) http://wiki.apache.org/hadoop/ZooKeeper/PoweredBy (please update this if your company is using...) So far we have 5 committers (Ben, Andrew, Flavio, Mahadev and myself) and just a few contributors. There are a number of teams using ZK inside Yahoo, Veoh is using ZK (Ted Dunning gave a talk with Ben at the last Hadoop social), spinn3r, and it looks like IONA might have some interest. :-) http://hiramchirino.com/blog/index.html A number of people attending the last Hadoop social indicated that they are using or looking at ZK. Patrick Hiram Chirino wrote: How big is the ZooKeeper developer community and what commercial associations do they have?
Re: mailing lists archived on nabble.com?
Good idea. Please enter a Jira and assign it to me. Patrick James Strachan wrote: Many apache projects including Hadoop register with nabble to host online forums great online archives of the mailing lists... http://www.nabble.com/Hadoop-f17066.html Currently there's hadoop-core, hbase and lucene on there. I often refer to mailing list posts by nabble link; they're really handy. Plus end users often prefer the forum style nabble approach to getting every single email sent to a mailing list. Does a committer on zookeeper fancy registering the ZK mailing lists too (as a child of the Hadoop list)? I'd do it myself but then I'd be the owner of the forum which doesn't feel right - a committer should probably do it. Its a pretty quick process, click here http://n2.nabble.com/more/MailingListRequest.jtp and fill out the details. The hardest part is knowing the mailing list software which AFAIK is ezmlm :)
Re: Website
Best bet is to put it to the vote - create a Jira. I'll make sure we get input from Hadoop PMC and core team members. Patrick James Strachan wrote: 2008/7/22 Hiram Chirino [EMAIL PROTECTED]: Lol.. Apache infrastructure supports multiple wiki backends. It's up to the project to pick which one you want to you. You currently have picked MoinMoin, but you could have easily picked Confluence, just like these other Apache projects did: http://cwiki.apache.org/confluence/dashboard.action FWIW I've been on plenty of apache projects that started on MoinMoin, then realised down the line how much better Confluence is - then went through the wiki migration pain. Tools like wikis are personal things; and folks tend to prefer to use the tool they know. But its worth seriously thinking about which wiki you want up front - before you realise down the line that you wanna switch. Just because the rest of hadoop uses MoinMoin doesn't mean you have to follow suit; all wikis can link to each other nicely. And there might be some hadoop folks who are not exactly head over heals about MoinMoin. All I'm gonna say is I've used both heavily and far prefer Confluence hands down; it means you can avoid all the Forest stuff; Confluence can auto-export its content to make a rather nice static HTML site for you; keeping your website and wiki clean, updated and fresh. Plus you can then reuse the same wiki markup inside your issue tracker (JIRA) - so you don't need to remember multiple wiki formats. Finally, there's great JIRA macros in confluence for creating lovely release notes in your website; or the snippet macros to link to fragments of test cases in your online documentation (so your documentation gets unit tested). Having said all that - its up to you guys to pick something you're happy with - so I'll bow out now and leave you to it :)
Re: assigning JIRAs to non committers
James Strachan wrote: Just an idle observation as I'd never seen this workflow before on JIRA so thought I'd ask :) I'm new to JIRA as well... I've been watching some of the recent JIRA activity with interest. I've seen a few JIRAs arrive, someone submits a test case who's not a committer, then the issue gets assigned to the person who submitted the patch. In some cases; when there may be many patches to assign over time, I can understand it (e.g. ZOOKEEPER-78 could take a zillion iterations before the feature is complete) - but in general if one JIRA gets one patch from a non-committer, should the JIRA be left unassigned - or assigned to a committer to review and apply or reject-with-reason the patch? I believe the workflow is that the jira is assigned to the person resolving the issue (ie submiting the patch). You/Hiram have been added as contributors to jira, this means that jiras can be assigned to you. We typically add ppl to the contributor list as soon as they submit a patch. After that point you do the back/forth in the comments trying to get everyone to agree to a resolution. If this is a patch you then change the status to patch available and ask for review/voting, after which if you get a +1 it's then up to a committer to commit to svn. full details here: http://wiki.apache.org/hadoop/ZooKeeper/HowToContribute i.e. lets say I raise a JIRA and attach a patch; once we're at that stage I can't actually do anything else, not being a committer - other than add another version of the patch :) So am not sure if its worth assigning the issue to me. I guess the person who raised the issue submitted the patch can always mark it as unassigned :) It's assigned to the person who resolved the issue. If accepted it's up the the committers to get it into svn, but you (the resolver) are still responsible. This information is also important for reporting purposes. No biggie I just thought I'd ask if this was an intentional way you guys had worked together in the past? This is generally how Hadoop core/hbase do things. Patrick
Re: do the test cases work for anyone else?
It's important to capture this type of information in jira. James Strachan wrote: FWIW I've ran the tests a few times; I think all these 4 tests have timing failures in them. I've seen all of them fail on OS X at some point. Sometimes only 2 will fail. On Linux I've seen just ClientTest fail. 2008/7/23 Patrick Hunt [EMAIL PROTECTED]: I'm on ubuntu (hardy heron) and they work. Our CI machine has intermittent failures (solaris x86): http://hudson.zones.apache.org/hudson/view/ZooKeeper/job/ZooKeeper-trunk/ there's some timing issue, what you're seeing is probably related to: https://issues.apache.org/jira/browse/ZOOKEEPER-61 Frankly tests and docs are both areas that ZooKeeper could use _a lot_ of care and feeding. Tests in particular could use some refactoring and a better implementation for launching/testing/stopping client/server tests. As you're able to reproduce the issue reliably would you like to take on 61? Feel free to assign to yourself if so. As a newbie on the project its hard enough grokking ZK itself and attempting to contribute patches, but fixing bad test cases of ZK is even harder :) I was hoping the folks who know ZK really well can fix the tests they've written :). But I'll take a look and see if I can see anything obvious I can do to help with my limited knowledge of the history of the code and internals. How about we raise a JIRA for all tests that fail?
Re: ZooKeeper community
Yes, it's documented on the web site: http://hadoop.apache.org/zookeeper/credits.html Patrick Hiram Chirino wrote: So all the commiters are Yahoo folks? On Wed, Jul 23, 2008 at 12:24 PM, Patrick Hunt [EMAIL PROTECTED] wrote: Good question, according to the PoweredBy wiki not too much ;-) http://wiki.apache.org/hadoop/ZooKeeper/PoweredBy (please update this if your company is using...) So far we have 5 committers (Ben, Andrew, Flavio, Mahadev and myself) and just a few contributors. There are a number of teams using ZK inside Yahoo, Veoh is using ZK (Ted Dunning gave a talk with Ben at the last Hadoop social), spinn3r, and it looks like IONA might have some interest. :-) http://hiramchirino.com/blog/index.html A number of people attending the last Hadoop social indicated that they are using or looking at ZK. Patrick Hiram Chirino wrote: How big is the ZooKeeper developer community and what commercial associations do they have?
Re: [jira] Commented: (ZOOKEEPER-97) The code generators should support an optional output directory
Ben you want to create a jira for marshalling then? Has anyone been tracking Google's protocol buffers, do they have C support yet/planned? Patrick Benjamin Reed (JIRA) wrote: [ https://issues.apache.org/jira/browse/ZOOKEEPER-97?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12617000#action_12617000 ] Benjamin Reed commented on ZOOKEEPER-97: One thing to keep in mind is that the jute generators in ZooKeeper really need to be replace with something like thrift, so we probably shouldn't put too much work into them. The code generators should support an optional output directory --- Key: ZOOKEEPER-97 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-97 Project: Zookeeper Issue Type: Improvement Reporter: Hiram Chirino Currently the code generators (jute compiler and version gen) generate code the current directory. If these tools are to be re-used by smart ides, ant tasks, or maven plugins, the output directory needs to be parameter to the tools.
ZooKeeper test harness needs work
I'm soliciting ideas on how to improve the reliability and usability (creating new tests in particular) of our current test harness. Hudson and some users are seeing intermittent failures, primarily due to timing related issue in test setup; the test starts a server, runs some tests, then shuts down the server, loop to the next test. There is some code in ClientBase that's supposed to provide a latch for the server startup, but we also have a number of sleeps in the test setup, without which the tests fail more frequently (so something is still busted). In particular I want to make it easier to write server tests and to remove the need for sleeps as this causes the unit tests to run slowly. Additionally we are seeing a need to tests clients in addition to the server (much of our current testing is related to verifying the correct function of the zk server). In this case we are not currently able to test any client failure handling cases (such as disconnect handling) as we are running against a fully functional zk server. I'm thinking we should do two things: 1) create a better test harness for the server 2) implement a mock ZooKeeper.java that has similar semantics as zk server proper but has the capability to inject/reproduce/verify various error scenarios for client testing. If you have any ideas/suggestions/comments/etc.. or would like to work on/with please let me know. Patrick
Re: ZooKeeper test harness needs work
Could be, but I don't think I'm seeing that in the test failure cases I've seen lately. In some cases the VM itself was crashing! Notice ZOOKEEPER-2 was seeing NPEs in the QuorumPeer. I did notice in AsyncTest that it's closing the server(s) before closing the clients, which is causing alot of noise in the logs (planning to patch that). What OS/cpu type configuration are you on? I'm on Ubuntu on 1core cpu. Patrick Hiram Chirino wrote: Lots of times when a test runs it seems like the ports are already bound. Anybody else see this? Is this the cause of the intermittent failures you see? Regards, Hiram On Wed, Jul 30, 2008 at 12:32 PM, Patrick Hunt [EMAIL PROTECTED] wrote: I'm soliciting ideas on how to improve the reliability and usability (creating new tests in particular) of our current test harness. Hudson and some users are seeing intermittent failures, primarily due to timing related issue in test setup; the test starts a server, runs some tests, then shuts down the server, loop to the next test. There is some code in ClientBase that's supposed to provide a latch for the server startup, but we also have a number of sleeps in the test setup, without which the tests fail more frequently (so something is still busted). In particular I want to make it easier to write server tests and to remove the need for sleeps as this causes the unit tests to run slowly. Additionally we are seeing a need to tests clients in addition to the server (much of our current testing is related to verifying the correct function of the zk server). In this case we are not currently able to test any client failure handling cases (such as disconnect handling) as we are running against a fully functional zk server. I'm thinking we should do two things: 1) create a better test harness for the server 2) implement a mock ZooKeeper.java that has similar semantics as zk server proper but has the capability to inject/reproduce/verify various error scenarios for client testing. If you have any ideas/suggestions/comments/etc.. or would like to work on/with please let me know. Patrick
Roadmap for ZooKeeper 3.0
The ZooKeeper JIRA is up to date and a tentative date of 9/22/2008 for the 3.0 release has been set. You can see the roadmap here: https://issues.apache.org/jira/browse/ZOOKEEPER?report=com.atlassian.jira.plugin.system.project:roadmap-panel 3.0 will be our first Apache release. As such the Java package structure has changed (com.yahoo - org.apache) which will require some work to migrate user code from pre-3.0 to 3.0. There are a few additional incompatible changes that have been lumped into 3.0, we wanted to pull any such changes into this release so that going fwd we could more easily maintain b/w compatibility in the apis/data (ie break things once, which had to be done anyway for package struct). You can see these changes by checking the appropriate checkbox in the findissues section of the JIRA. The release notes for 3.0 will contain a section detailing the migrations necessary when migrating from 2.2.1 to 3.0 If you have any questions please respond. Regards, Patrick
New ZooKeeper contributors added
Congratulations, Maxim Dementiev Stu Hood were recently added as contributors to ZooKeeper. Patrick
ZooKeeper 3.0.0 code freeze EOD thursday? (Oct16)
We're down to 6 JIRAs for 3.0.0 of ZooKeeper, that's 95 issues resolved out of 101 for this release! Great job everyone!! We are on track for our Oct22nd release date (+ a few days for the PMC voting process). I'd like to see the remaining 6 issues fixed, reviewed, and committed to SVN by EOD Thursday Oct16. Anyone see issues with this? Once all the 3.0.0 jiras are committed to SVN I'll create/publish a release candidate for everyone to review test. If there are no issues found by say Tuesday of next week I'll put it up for a PMC vote. Let me know if there are questions or comments. Patrick ps. the ZooKeeper release process page is here: http://wiki.apache.org/hadoop/ZooKeeper/HowToRelease
[VOTE] Release ZooKeeper 3.0.0 (candidate 0)
I've created a candidate build for the first Apache release of ZooKeeper - version 3.0.0. *** Please download, test and VOTE before the *** vote closes EOD on Friday, October 24.*** http://people.apache.org/~phunt/zookeeper-3.0.0-candidate-0/ Patrick
Re: [VOTE] Release ZooKeeper 3.0.0 (candidate 0)
Thanks for the feedback Doug. 1) I created ZOOKEEPER-206 to address the first 3 issues you mentioned 2) regarding the logo this is already captured at ZOOKEEPER-69. We have a preliminary logo, working to finalize (should be soon) Patrick Doug Cutting wrote: +1 The release looks great, much better than most first releases! I noticed a few minor problems in the documentation, but these should by no means not stop the release. In particular: - The Zookeeper Documentation tab should contain the version number. - Informal Documentation might better be named Other Documentation. - The Other Info page should be removed, since it contains nothing. - a logo (ZOOKEEPER-69), any logo, would sure be nice. Doug Patrick Hunt wrote: I've created a candidate build for the first Apache release of ZooKeeper - version 3.0.0. *** Please download, test and VOTE before the *** vote closes EOD on Friday, October 24.*** http://people.apache.org/~phunt/zookeeper-3.0.0-candidate-0/ Patrick
Re: [VOTE] Release ZooKeeper 3.0.0 (candidate 0)
The vote has passed, I've completed the HowToRelease process and will officially announce momentarily. http://wiki.apache.org/hadoop/HowToRelease Patrick Patrick Hunt wrote: I've created a candidate build for the first Apache release of ZooKeeper - version 3.0.0. *** Please download, test and VOTE before the *** vote closes EOD on Friday, October 24.*** http://people.apache.org/~phunt/zookeeper-3.0.0-candidate-0/ Patrick
Proposal to require Java6 in 3.1.0
I've entered a JIRA targeted for ZooKeeper 3.1.0 that will add Java 6 requirement to ZooKeeper (we will drop java5 support). If you have any feedback (pos or neg) please add comments to the issue: https://issues.apache.org/jira/browse/ZOOKEEPER-210 Regards, Patrick
Anyone expert in JMX willing to help with ZK impl/docs?
ZK 3.0.0 added support for JMX. Basic server stats are available, as are details on per-client connections. However we don't have zk specific documentation and also it would probably be a good idea for someone that's expert in JMX to review what we have and suggest changes. One thing that may factor in is that we would like to move to JDK6 only in 3.1.0. Additionally there's probably information that's missing from our JMX beans that could help debug/operate a running zk instance. Anyone an expert in JMX that could provide some help? https://issues.apache.org/jira/browse/ZOOKEEPER-177 https://issues.apache.org/jira/browse/ZOOKEEPER-94 Regards, Patrick ps. docs are in forrest (see trunk/src/docs) but we do have a tech writer that could help with the formatting, etc... the main thing is to write the content.
ZooKeeper 3.0 Fix Release slated for end of this week.
I've slated the 3.0.1 fix release of ZooKeeper for the end of this week. https://issues.apache.org/jira/browse/ZOOKEEPER?report=com.atlassian.jira.plugin.system.project:roadmap-panel Of particular interest are exists() NPE in ZOOKEEPER-226, and some perf issues ZOOKEEPER-212 ZOOKEEPER-223 If there are any questions, or issues that should be included that are not, please speak up. Patrick
[VOTE] Release ZooKeeper 3.0.1 (candidate 0)
I've created a candidate build for ZooKeeper 3.0.1. *** Please download, test and VOTE before the *** vote closes EOD on Monday, November 24.*** http://people.apache.org/~phunt/zookeeper-3.0.1-candidate-0/ Should we release this? Patrick
Re: [VOTE] Release ZooKeeper 3.0.1 (candidate 0)
-1 I noticed that src/java/test/org/apache/zookeeper/TestableZooKeeper.java is missing Apache license header. We'll have to add a header and put up a new rc for voting (tomorrow) Patrick Patrick Hunt wrote: I've created a candidate build for ZooKeeper 3.0.1. *** Please download, test and VOTE before the *** vote closes EOD on Monday, November 24.*** http://people.apache.org/~phunt/zookeeper-3.0.1-candidate-0/ Should we release this? Patrick
Re: ActiveMQ is now using ZooKeeper
That's great, very cool! Can you create ZOOKEEPER JIRAs for these items that you've identified? First look it seems like we should be able to include these in 3.1.0, perhaps even 3.0.2. Regards, Patrick Hiram Chirino wrote: FYI: ActiveMQ has now started using ZooKeeper to do master election of it's HA clusters. So zookeeper is now going to get included in every new ActiveMQ distro we cut. Which in turn this should mean that it will get included in every ServiceMix and Geronimo distro since they repackage ActiveMQ. More details at: http://cwiki.apache.org/confluence/display/ACTIVEMQ/KahaDB+Master+Slave So this might start bringing more folks/requirements to your project. For example it would be nice if: - You had a slim client only jar which we package with ActiveMQ to reduce our footprint, since we only use the client aspect of ZooKeeper - Make the sever more embeddable (for example eliminate using static vars to initialize the sever) so that it can be wrapped up in an OSGi bundle and deployed in ServiceMix.
ZooKeeper (not) on wikipedia
I noticed today that there is no ZooKeeper page on Wikipedia. Would anyone like to create it? This would be preferable: http://en.wikipedia.org/wiki/ZooKeeper But I also noticed this as well (perhaps _software could redirect? Or just change the disambiguation page to link to ZooKeeper instead?): http://en.wikipedia.org/wiki/Zookeeper_(disambiguation) http://en.wikipedia.org/wiki/ZooKeeper_(software) The zk docs home and overview should have more than enough to start: http://hadoop.apache.org/zookeeper/docs/r3.0.0/ http://hadoop.apache.org/zookeeper/docs/r3.0.0/zookeeperOver.html Regards, Patrick
Re: ZooKeeper (not) on Wikipedia
Great job, thanks! Krishna Sankar (ksankar) wrote: Thanks. Done. Good catch on DFS - now you know my secret - I used the Hadoop entry as template ! I think the best category is cloud computing, a forward looking statement; distributed computing might be more precise, but the category is very broad. So I added them both, which most probably is the real nature of the beast anyway ;o) Cheers k/ |-Original Message- |From: Patrick Hunt [mailto:[EMAIL PROTECTED] |Sent: Tuesday, December 02, 2008 11:13 PM |To: [EMAIL PROTECTED]; Krishna Sankar (ksankar) |Cc: zookeeper-dev@hadoop.apache.org |Subject: Re: ZooKeeper (not) on Wikipedia | |Very nice! | |Not sure if it's external links or references but you could include: |http://developer.yahoo.net/blogs/hadoop/2008/03/intro-to-zookeeper- |video.html |http://highscalability.com/node/426 |http://www.sdtimes.com/ZOOKEEPER_SERVICES_COORDINATOR_MOVES_TO_APACHE/A b |out_SERVICECOORDINATION_and_ZOOKEEPER_and_APACHE_and_YAHOO/33011 | |distributed file system doesn't seem right to me, but I'm not sure |what I would replace it with. ;-) |http://en.wikipedia.org/wiki/Category:Distributed_computing | |Patrick | |Krishna Sankar (ksankar) wrote: | First cut at http://en.wikipedia.org/wiki/Apache_ZooKeeper. I have | copied two images (ZooKeeper services and ZooKeeper Namespace) into | Wikipedia. | | Also added the right pointer in the disambiguation page | http://en.wikipedia.org/wiki/Zookeeper_(disambiguation) | | Should we add a pointer to the Apache Template as well? | | Cheers | k/ | P.S : I think it is a good start, so have taken down the construction | banner. Can put it up if anyone things we need to add lot more | information. | | |-Original Message- | |From: Mahadev Konar [mailto:[EMAIL PROTECTED] | |Sent: Tuesday, November 25, 2008 5:04 PM | |To: zookeeper-dev@hadoop.apache.org; [EMAIL PROTECTED] | |Subject: Re: ZooKeeper (not) on Wikipedia | | | |Great!! Go ahead. | | | |mahadev | | | | | |On 11/25/08 4:57 PM, Krishna Sankar (ksankar) [EMAIL PROTECTED] | |wrote: | | | | If no one has done it, I will take a first cut at it. | | | | |-Original Message- | | |From: Patrick Hunt [mailto:[EMAIL PROTECTED] | | |Sent: Tuesday, November 25, 2008 3:26 PM | | |To: zookeeper-dev@hadoop.apache.org; | [EMAIL PROTECTED] | | |Subject: ZooKeeper (not) on wikipedia | | | | | |I noticed today that there is no ZooKeeper page on Wikipedia. |Would | | |anyone like to create it? | | | | | |This would be preferable: | | |http://en.wikipedia.org/wiki/ZooKeeper | | | | | |But I also noticed this as well (perhaps _software could redirect? | Or | | |just change the disambiguation page to link to ZooKeeper |instead?): | | |http://en.wikipedia.org/wiki/Zookeeper_(disambiguation) | | |http://en.wikipedia.org/wiki/ZooKeeper_(software) | | | | | |The zk docs home and overview should have more than enough to |start: | | |http://hadoop.apache.org/zookeeper/docs/r3.0.0/ | | |http://hadoop.apache.org/zookeeper/docs/r3.0.0/zookeeperOver.html | | | | | |Regards, | | | | | |Patrick |
Re: Maven artifacts for 3.0.0 release
Not sure if you noticed this discussion on hadoop core-dev: http://www.nabble.com/Maven-vs-Ivy-td20786184.html In particular: it would help if someone wishing to use Maven had a look at the poms and warned of trouble before they actually went up into the central repository As I mentioned previously we will most likely follow core's lead on this (both in terms of build as well as release process). Regards, Patrick Hiram Chirino wrote: commented. Basically deploying the the maven repo is the same as deploying the final release artifact (it's just at a different directory on the same machine and has a different directory structure). On Wed, Nov 19, 2008 at 1:02 PM, Patrick Hunt [EMAIL PROTECTED] wrote: Hi Hiram, I updated 224 with a few questions, if you could provide some insight that would be helpful. Patrick Hiram Chirino wrote: I created issue: https://issues.apache.org/jira/browse/ZOOKEEPER-224 to track the request. Who's the current ZooKeeper release manager? On Fri, Nov 14, 2008 at 11:07 AM, Hiram Chirino [EMAIL PROTECTED] wrote: Hi Guys, I've create a small maven 2 repository for the ZooKeeper 3.0.0 release at: http://people.apache.org/~chirino/zk-repo/ Ideally the ZooKeeper PMC can review the artifacts validate that they match the release, GPG sign all the poms and jars and push them out to the main apache m2 maven repository so it can get synchronized into the maven central repo. That's at people.apache.org:/www/people.apache.org/repo/m2-ibiblio-rsync-repository BTW. But For now maven users of ZooKeeper can add the following to their build to automatically download the release: repositories repository idchirino-zk-repo/id namePrivate ZooKeeper Repo/name urlhttp://people.apache.org/~chirino/zk-repo//url /repository /repositories dependencies dependency groupIdorg.apache.hadoop.zookeeper/groupId artifactIdzookeeper/artifactId version3.0.0/version /dependency /dependencies -- Regards, Hiram Blog: http://hiramchirino.com Open Source SOA http://open.iona.com
[ANNOUNCE] Apache ZooKeeper 3.0.1
The Apache ZooKeeper team is proud to announce Apache ZooKeeper version 3.0.1. ZooKeeper is a high-performance coordination service for distributed applications. It exposes common services - such as naming, configuration management, synchronization, and group services - in a simple interface so you don't have to write them from scratch. You can use it off-the-shelf to implement consensus, group management, leader election, and presence protocols. And you can build on it for your own, specific needs. If you are upgrading from version 2.2.1 on SourceForge be sure to review the 3.0.1 release notes for migration instructions. For ZooKeeper release details and downloads, visit: http://hadoop.apache.org/zookeeper/releases.html ZooKeeper 3.0.1 Release Notes are at: http://hadoop.apache.org/zookeeper/docs/r3.0.1/releasenotes.html Regards, The ZooKeeper Team
Re: Maven artifacts for 3.0.0 release
NP. Right, I believe that was the gist - someone experienced in Maven to review and provide feedback on the poms. Patrick Hiram Chirino wrote: sry.. must be my eyes getting weak :) So he just want's folks to review the poms? On Fri, Dec 5, 2008 at 12:39 PM, Patrick Hunt [EMAIL PROTECTED] wrote: I'm missing the reference to double trouble, where does he say that? Hiram Chirino wrote: Double trouble? What's he mean by that? On Wed, Dec 3, 2008 at 6:04 PM, Patrick Hunt [EMAIL PROTECTED] wrote: Not sure if you noticed this discussion on hadoop core-dev: http://www.nabble.com/Maven-vs-Ivy-td20786184.html In particular: it would help if someone wishing to use Maven had a look at the poms and warned of trouble before they actually went up into the central repository As I mentioned previously we will most likely follow core's lead on this (both in terms of build as well as release process). Regards, Patrick Hiram Chirino wrote: commented. Basically deploying the the maven repo is the same as deploying the final release artifact (it's just at a different directory on the same machine and has a different directory structure). On Wed, Nov 19, 2008 at 1:02 PM, Patrick Hunt [EMAIL PROTECTED] wrote: Hi Hiram, I updated 224 with a few questions, if you could provide some insight that would be helpful. Patrick Hiram Chirino wrote: I created issue: https://issues.apache.org/jira/browse/ZOOKEEPER-224 to track the request. Who's the current ZooKeeper release manager? On Fri, Nov 14, 2008 at 11:07 AM, Hiram Chirino [EMAIL PROTECTED] wrote: Hi Guys, I've create a small maven 2 repository for the ZooKeeper 3.0.0 release at: http://people.apache.org/~chirino/zk-repo/ Ideally the ZooKeeper PMC can review the artifacts validate that they match the release, GPG sign all the poms and jars and push them out to the main apache m2 maven repository so it can get synchronized into the maven central repo. That's at people.apache.org:/www/people.apache.org/repo/m2-ibiblio-rsync-repository BTW. But For now maven users of ZooKeeper can add the following to their build to automatically download the release: repositories repository idchirino-zk-repo/id namePrivate ZooKeeper Repo/name urlhttp://people.apache.org/~chirino/zk-repo//url /repository /repositories dependencies dependency groupIdorg.apache.hadoop.zookeeper/groupId artifactIdzookeeper/artifactId version3.0.0/version /dependency /dependencies -- Regards, Hiram Blog: http://hiramchirino.com Open Source SOA http://open.iona.com
Re: ZooKeeper SVN revision
I use git and 241 fixes the issue for me as well. Mahadev Konar wrote: Hi Nitay, We already have an open jira for this. http://issues.apache.org/jira/browse/ZOOKEEPER-241 This should be fixed in the next release. thanks Mahadev On 12/8/08 12:08 AM, Nitay [EMAIL PROTECTED] wrote: Hey folks, I'm working on integrating ZooKeeper into HBase. I've become a fan of git so I've started using the unofficial Apache git repo at http://jukka.zitting.name/git/?p=zookeeper.git;a=summary instead of the raw svn. Everything works great except the lastRevision.sh script fails to write the lastRevision property because it is not an SVN repository. When I run ant I get: version-info: [java] All version-related parameters must be valid integers! [java] Exception in thread main java.lang.NumberFormatException: For input string: [java] at java.lang.NumberFormatException.forInputString(NumberFormatException.java:48) [java] at java.lang.Integer.parseInt(Integer.java:468) [java] at java.lang.Integer.parseInt(Integer.java:497) [java] at org.apache.zookeeper.version.util.VerGen.main(VerGen.java:111) [java] Java Result: 1 I fixed this by hardcoding lastRevision to some value in build.xml. Why do necessarily require SVN in order for ant to run? Can we have some sensible default if the svn info doesn't work? Should I write something for git repos that produces the current HEAD sha if there is a '.git' dir instead of a '.svn' one? Thanks, -n
Re: Delaying 3.1 release by 2 to 3 weeks?
Mahadev, can you complete quotas in 2 weeks? This includes completing the code itself, documentation, tests, and incorporating review feedback? Parick Benjamin Reed wrote: we should delay. it would be good to try out quotas for a bit before we do the release. quotas are also a key part of the release. 3 weeks seem a little long though. ben From: Mahadev Konar [maha...@yahoo-inc.com] Sent: Thursday, January 15, 2009 4:32 PM To: zookeeper-dev@hadoop.apache.org Cc: zookeeper-u...@hadoop.apache.org Subject: Re: Delaying 3.1 release by 2 to 3 weeks? That was release 3.1 and not 3.2 :) mahadev On 1/15/09 4:26 PM, Mahadev Konar maha...@yahoo-inc.com wrote: Hi all, I needed to get quotas in zookeeper 3.2.0 and wanted to see if delaying the release by 2-3 weeks is ok with everyone? Here is the jira for it - http://issues.apache.org/jira/browse/ZOOKEEPER-231 Please respond if you have any issues with the delay. thanks mahadev
Re: Delaying 3.1 release by 2 to 3 weeks?
Ok, I've updated JIRA to list the release candidate build for Feb 3. Please submit the tests and documentation asap so that we can have sufficient review. I can always cut a release candidate before the 3rd, keep me posted on status. Patrick Mahadev Konar wrote: I think it should be done in 2 weeks.. mahadev On 1/16/09 2:34 PM, Patrick Hunt ph...@apache.org wrote: Mahadev, can you complete quotas in 2 weeks? This includes completing the code itself, documentation, tests, and incorporating review feedback? Parick Benjamin Reed wrote: we should delay. it would be good to try out quotas for a bit before we do the release. quotas are also a key part of the release. 3 weeks seem a little long though. ben From: Mahadev Konar [maha...@yahoo-inc.com] Sent: Thursday, January 15, 2009 4:32 PM To: zookeeper-dev@hadoop.apache.org Cc: zookeeper-u...@hadoop.apache.org Subject: Re: Delaying 3.1 release by 2 to 3 weeks? That was release 3.1 and not 3.2 :) mahadev On 1/15/09 4:26 PM, Mahadev Konar maha...@yahoo-inc.com wrote: Hi all, I needed to get quotas in zookeeper 3.2.0 and wanted to see if delaying the release by 2-3 weeks is ok with everyone? Here is the jira for it - http://issues.apache.org/jira/browse/ZOOKEEPER-231 Please respond if you have any issues with the delay. thanks mahadev
[VOTE] Release ZooKeeper 3.1.0 (candidate 0)
I've created a candidate build for ZooKeeper 3.1.0. Over 65 JIRAs are addressed in this release, with major new features for Quota, JMX, and Bookkeeper (transactional log store) to name a few. Full details in the release notes. *** Please download, test and VOTE before the *** vote closes EOD on Friday, February 6.*** http://people.apache.org/~phunt/zookeeper-3.1.0-candidate-0/ Should we release this? Patrick
[VOTE] Release ZooKeeper 3.1.0 (candidate 1)
I've created a new candidate (rc1) that fixes an issue raised by a user (thanks Stu!) during review: https://issues.apache.org/jira/browse/ZOOKEEPER-291 The release notes were also updated to reflect this change. Otw there are no other changes. *** Please download, test and VOTE before the *** vote closes EOD on Monday February 9.*** http://people.apache.org/~phunt/zookeeper-3.1.0-candidate-1/ Should we release this? Patrick
[VOTE] Release ZooKeeper 3.1.0 (candidate 2)
I've created a new candidate (rc2) that fixes an issue raised by a user (thanks Chris!) during review: https://issues.apache.org/jira/browse/ZOOKEEPER-293 The release notes were also updated to reflect this change. Otw there are no other changes. *** Please download, test and VOTE before the *** vote closes EOD on Tuesday February 10.*** http://people.apache.org/~phunt/zookeeper-3.1.0-candidate-2/ Should we release this? Patrick
Re: [VOTE] Release ZooKeeper 3.1.0 (candidate 1)
Thanks Tom. A patch for 302 has been committed to SVN, I will create a new release candidate today. Patrick Tom White wrote: After a bit of digging I think I know what's happening. I'm running in standalone mode on a Mac with IPv6. When I try to connect to localhost using new ZooKeeper(localhost, 5000, this); I get the exception I posted earlier. When I use 127.0.0.1 it works fine. So it is an IPv6 issue (you were right!). I can fix the error by quoting the value properties, for example: org.apache.ZooKeeperService:name0=StandaloneServer_port-1,name1=Connections,name2=0:0:0:0:0:0:0:1%0,name3=0x11f4b7b4c34000e Note that name2 is in quotes. See http://java.sun.com/javase/6/docs/api/javax/management/ObjectName.html. I've created a patch for this in https://issues.apache.org/jira/browse/ZOOKEEPER-302. Not sure if this is a blocker or not for 3.1.0. It probably is if IPv6 is supported. What do you think? Thanks, Tom On Thu, Feb 5, 2009 at 11:42 PM, Patrick Hunt ph...@apache.org wrote: I don't suppose you are running on an IPv6 system? Patrick Patrick Hunt wrote: Hi Tom, this is unusual - what's your environment like? OS and java version string would help to debug (zk config?). Mahadev (mac) and I (ubuntu/redhat) both running recent versions of java6 have just tried this and we can't reproduce it. We use the IP address sessionid as the name of the connection jmx bean, so this is weird: ip=connection.getRemoteAddress().getAddress().getHostAddress(); here's an example name I see in jconsole: org.apache.ZooKeeperService:name0=StandaloneServer_port-1,name1=Connections,name2=127.0.0.1,name3=0x11f489f8126 (btw, thank you sun for including the invalid name in the error message ;-) ) One thing I do notice is that printStackTrace is being called by our jmx code which should not be happening - that's why you are seeing this in on the console rather than just in the log (we log WARN then ignore the issue btw) . I'll enter a jira for this. https://issues.apache.org/jira/browse/ZOOKEEPER-300 Patrick Tom White wrote: I installed the candidate and got the following exception when performing some basic operations. The operations succeeded - it looks like a problem with JMX integration. Has anyone else seen this? Thanks, Tom $ bin/zkServer.sh start JMX enabled by default -n Starting zookeeper ... STARTED loy:zookeeper-3.1.0 tom$ javax.management.MalformedObjectNameException: Invalid character ':' in value part of property at javax.management.ObjectName.construct(ObjectName.java:602) at javax.management.ObjectName.init(ObjectName.java:1394) at org.apache.zookeeper.jmx.MBeanRegistry.makeObjectName(MBeanRegistry.java:173) at org.apache.zookeeper.jmx.MBeanRegistry.register(MBeanRegistry.java:70) at org.apache.zookeeper.server.NIOServerCnxn.finishSessionInit(NIOServerCnxn.java:861) at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:145) at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:127) at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:75) java.lang.IllegalArgumentException: Null object name at com.sun.jmx.mbeanserver.MXBeanSupport.register(MXBeanSupport.java:138) at com.sun.jmx.mbeanserver.MBeanSupport.preRegister2(MBeanSupport.java:183) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:941) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:917) at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:312) at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:482) at org.apache.zookeeper.jmx.MBeanRegistry.register(MBeanRegistry.java:72) at org.apache.zookeeper.server.NIOServerCnxn.finishSessionInit(NIOServerCnxn.java:861) at org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:145) at org.apache.zookeeper.server.SyncRequestProcessor.flush(SyncRequestProcessor.java:127) at org.apache.zookeeper.server.SyncRequestProcessor.run(SyncRequestProcessor.java:75) On Thu, Feb 5, 2009 at 8:56 PM, Stu Hood stuh...@mailtrust.com wrote: +1 Thanks Patrick! -Original Message- From: Patrick Hunt ph...@apache.org Sent: Wednesday, February 4, 2009 6:33pm To: zookeeper-dev@hadoop.apache.org zookeeper-dev@hadoop.apache.org Subject: [VOTE] Release ZooKeeper 3.1.0 (candidate 1) I've created a new candidate (rc1) that fixes an issue raised by a user (thanks Stu!) during review: https://issues.apache.org/jira/browse/ZOOKEEPER-291 The release notes were also updated to reflect this change. Otw there are no other changes. *** Please download, test and VOTE before the *** vote closes EOD on Monday February 9.*** http://people.apache.org/~phunt/zookeeper-3.1.0-candidate-1/ Should we release this? Patrick
[VOTE] Release ZooKeeper 3.1.0 (candidate 3)
I've created a new candidate (rc3) that fixes an issue raised by a user (thanks Tom!) during review: https://issues.apache.org/jira/browse/ZOOKEEPER-302 The release notes were also updated to reflect this change. Otw there are no other changes. *** Please download, test and VOTE before the *** vote closes EOD on Wednesday February 11.*** http://people.apache.org/~phunt/zookeeper-3.1.0-candidate-3/ Should we release this? Patrick
Re: setACL semantics
Mahadev, is it true that a lagging follower, participating in the ensemble but not part of the quorum, would continue to respond to read requests until it had received/processed the update from the leader? Patrick Mahadev Konar wrote: HI Manos, The accepting of a txn for being processed does not mean that the txn has been accpeted by the quorum and has been applied to the quorum servers. So, it is very much possible that one of the client changes an acl on a node and the other client unauthorized to read (with respect to the change by client1) reads the data before the acl change is propagated and accepted by the quorum. mahadev On 2/10/09 4:34 PM, Manos Kapritsos ma...@cs.utexas.edu wrote: More or less, yes. If the requests are from the same client, then you say that there will not be a problem? I guess that is true if you always wait for the response of the first request in order to execute the second. I am not sure if that is a requirement for all Zookeeper client implementations. As for two different clients (which was the case I was thinking about), this seems to be a problem. I will agree that (if clients only have one outstanding request) the two requests are concurrent and either order of execution is considered to be acceptable, but it could be that two different replicas receive the two requests in the same order, but effectively execute them in a different order. In any case, it feels wrong (at least to me) that a getData would succeed when a setACL that prevents it has already been accepted to be processed. Manos Mahadev Konar wrote: Hi Manos, If the setAcl and getdata are from the same client then they are all handled in order. So you would get an unauthroized exception when you do a getdata. If two diff clients do setacl and getdata it might be that the getdata in your case will succeed before the setacl returns on the first client. Is that what you meant? mahadev On 2/10/09 2:15 PM, Manos Kapritsos ma...@cs.utexas.edu wrote: Hi all, I have a question about the way setACL functions. It seems that the PreRequestProcessor handles all kinds of requests the same, checks the validity of the corresponding ACL, and enqueues them to Sync and Final processors. Maybe I am missing something here, but this behaviour seems weird. What if a setACL request comes, setting the ACL of a path (e.g. / ) to an IP (e.g. 128.62.32.32) , instead of its old value (e.g. World). This request will pass the ACL check, and will be enqueued to be processed by the next processors. Assume that the next request is a getData(/) from an IP other than 128.62.32.32. If this request is processed by the PreRequestProcessor before the setACL request is processed by the FinalRequestProcessor, then it will pass the ACL check (which it should not, since it came after the setACL request). It seems that there is a race condition here that should not exist. Let me know if this is actually the case or I am missing something. I am using version 3.0.1 of the code. Thank you, Manos
Re: When is the AUTH_FAILED state entered?
You're right, it's never set. Please enter a JIRA assigned to 3.2 and we'll look into it. Patrick Tom White wrote: The AUTH_FAILED state in the ZooKeeper class never seems to be assigned to in the code. So the ZooKeeper client can never enter this state. Instead it looks like authentication errors are indicated by AuthFailedException (with code AUTHFAILED). Is this right? Either the AUTH_FAILED state should be removed, or an AuthFailedException should cause the ZooKeeper client to transition to the AUTH_FAILED state. Thanks, Tom
[ANNOUNCE] Apache ZooKeeper 3.1.0
The Apache ZooKeeper team is proud to announce Apache ZooKeeper version 3.1.0. ZooKeeper is a high-performance coordination service for distributed applications. It exposes common services - such as naming, configuration management, synchronization, and group services - in a simple interface so you don't have to write them from scratch. You can use it off-the-shelf to implement consensus, group management, leader election, and presence protocols. And you can build on it for your own, specific needs. Key features of the 3.1.0 release: * Quota support * BookKeeper - a system to reliably log streams of records * JMX for server management * many fixes, improvements, improved documentation, etc... A bit about BookKeeper: a system to reliably log streams of records. In BookKeeper, servers are bookies, log streams are ledgers, and each unit of a log (aka record) is a ledger entry. BookKeeper is designed to be reliable; bookies, the servers that store ledgers can be byzantine, which means that some subset of the bookies can fail, corrupt data, discard data, but as long as there are enough correctly behaving servers the service as a whole behaves correctly; the meta data for BookKeeper is stored in ZooKeeper. For ZooKeeper release details and downloads, visit: http://hadoop.apache.org/zookeeper/releases.html ZooKeeper 3.1.0 Release Notes are at: http://hadoop.apache.org/zookeeper/docs/r3.1.0/releasenotes.html Regards, The ZooKeeper Team
Suggestions for testing JMX code in ZK server?
Anyone have suggestions for JMX testing best practices? This is one thing we are currently lacking, I'd like to add tests in 3.2. I found this which has some good insight: http://weblogs.java.net/blog/dwalend/archive/2008/06/jmx_and_testdri_1.html any other suggestions? Patrick
ZooKeeper Roadmap - 3.2.0 and beyond.
Apache release 3.1.0 has shipped and I'm already looking forward to 3.2.0. ;-) In particular I believe we should look at the following going forward. Of course ZooKeeper is open to submissions in that aren't on this list. If you have any suggestions please feel free to enter a JIRA, submit a patch, or comment on this thread. http://wiki.apache.org/hadoop/ZooKeeper/HowToContribute 3.2 (May release) implement some basic recipes as client libs consistent c/java implementations chroot support in session connect string unified scheme for connect string? uri? expand system tests move some unit tests into system test framework (esp large quorum tests) add performance/benchmark tests integrate into hudson? (historical graphs) why are some ensemble sizes (like 7) less performant? monitor/track history optimize session tracking no expiration of session if client has no ephemerals huge scalability win if we can do this (for a very common use case) cleanup the c binding logging and general code cleanliness review jmx object naming scheme error handling, general cruft move examples out of docs into contrib/examples (like hadoop core) add more examples build changes jdiff split jars? common, server, client, test ivy? push jars to maven repo? 3.3 (August release) horizontally partitioned zk support multiple client protocols/marshalling (hadoop rpc?) quota support for limiting (rather than just reporting) 4.0 API/data breakage release (November) API changes move all counter fields from int - long (version numbers for example) requires migration of snapshots (ie persistence/marshalling will change) pass enums to callbacks remove deprecated cruft bookeeper more tightly integrated into zk? Regards, Patrick
What are you using ZooKeeper for?
If you are using ZK and can publicly share this information please update the wiki PoweredBy page: http://wiki.apache.org/hadoop/ZooKeeper/PoweredBy Patrick
JExample - Interesting junit extension
This looks like it could be very useful for unit tests - basically allows you to easily reuse fixtures: http://smallwiki.unibe.ch/jexample Patrick
Re: [VOTE] Release ZooKeeper 3.1.1 (candidate 0)
+1, passes all my tests for both c and java. Patrick Stu Hood wrote: +1 Passes our integration tests. (...which don't test the C API, which is where the majority of the changes appear to have happened: https://issues.apache.org/jira/browse/ZOOKEEPER/fixforversion/12313649) Thanks, Stu -Original Message- From: Patrick Hunt ph...@apache.org Sent: Thursday, March 12, 2009 5:00pm To: zookeeper-dev@hadoop.apache.org zookeeper-dev@hadoop.apache.org Subject: [VOTE] Release ZooKeeper 3.1.1 (candidate 0) I've created a candidate build for ZooKeeper 3.1.1. This is a bug fix release addressing a small number of issues -- see the release notes for details. *** Please download, test and VOTE before the *** vote closes EOD on Friday, March 17.*** http://people.apache.org/~phunt/zookeeper-3.1.1-candidate-0/ Should we release this? Patrick
[VOTE] Release ZooKeeper 3.1.1 (candidate 1)
I've created a new candidate (rc1) that fixes a regression found during review: https://issues.apache.org/jira/browse/ZOOKEEPER-341 The release notes were also updated to reflect this change. Otw there are no other changes. *** Please download, test and VOTE before the *** vote closes EOD on Monday March 23.*** http://people.apache.org/~phunt/zookeeper-3.1.1-candidate-1/ Should we release this? Patrick
Re: [VOTE] Release ZooKeeper 3.1.1 (candidate 1)
The vote passes with +3 and no -1s. I will publish the release tomorrow. Patrick Patrick Hunt wrote: I've created a new candidate (rc1) that fixes a regression found during review: https://issues.apache.org/jira/browse/ZOOKEEPER-341 The release notes were also updated to reflect this change. Otw there are no other changes. *** Please download, test and VOTE before the *** vote closes EOD on Monday March 23.*** http://people.apache.org/~phunt/zookeeper-3.1.1-candidate-1/ Should we release this? Patrick
Re: Socket is not connected
Hi Stefan, There's a timing issue when the client closes the connection. It sends a session close to the server and then starts to close it's own sockets. If the server closes the socket before the client does you can sometimes see the client get java.net.SocketException: Socket is not connected we are a bit paranoid about making sure you/we can debug unusual situations, so we include the trace in the WARN message in the log, just in case. In this situation though it is not an issue, in other situations (say the server drops it's connection) it would help to have the trace in a debugging situation, so we include it. Btw, in general in the code if we are more concerned about an event we would output as an ERROR, not as a WARN. Patrick Stefan Groschupf wrote: Hi, I managed to upgrade my zookeeper 2 application to zookeeper 3.1.1/ All my tests run though my logs is full of following exception. In my code I basically just invoice ZooKeeper.close(); Any idea what the problem could be? Thanks, Stefan 09/03/27 02:07:51 INFO zookeeper.ZooKeeper:434 - Closing session: 0x120472f36480002 09/03/27 02:07:51 INFO zookeeper.ClientCnxn:999 - Closing ClientCnxn for session: 0x120472f36480002 09/03/27 02:07:51 INFO server.PrepRequestProcessor:360 - Processed session termination request for id: 0x120472f36480002 09/03/27 02:07:51 INFO server.NIOServerCnxn:752 - closing session:0x120472f36480002 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/0:0:0:0:0:0:0:1%0:2181 remote=/0:0:0:0:0:0:0:1%0:53636] 09/03/27 02:07:51 INFO zookeeper.ClientCnxn:892 - Exception while closing send thread for session 0x120472f36480002 : Read error rc = -1 java.nio.DirectByteBuffer[pos=0 lim=4 cap=4] 09/03/27 02:07:51 WARN zookeeper.ClientCnxn:932 - Ignoring exception during shutdown input java.net.SocketException: Socket is not connected at sun.nio.ch.SocketChannelImpl.shutdown(Native Method) at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640) at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360) at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:930) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:915) 09/03/27 02:07:51 INFO zookeeper.ClientCnxn:985 - Disconnecting ClientCnxn for session: 0x120472f36480002 09/03/27 02:07:51 INFO zookeeper.ZooKeeper:442 - Session: 0x120472f36480002 closed
[ANNOUNCE] Apache ZooKeeper 3.1.1
The Apache ZooKeeper team is proud to announce Apache ZooKeeper version 3.1.1. ZooKeeper is a high-performance coordination service for distributed applications. It exposes common services - such as naming, configuration management, synchronization, and group services - in a simple interface so you don't have to write them from scratch. You can use it off-the-shelf to implement consensus, group management, leader election, and presence protocols. And you can build on it for your own, specific needs. If you are upgrading from version 2.2.1 on SourceForge be sure to review the 3.0.1 release notes for migration instructions. For ZooKeeper release details and downloads, visit: http://hadoop.apache.org/zookeeper/releases.html ZooKeeper 3.1.1 Release Notes are at: http://hadoop.apache.org/zookeeper/docs/r3.1.1/releasenotes.html Regards, The ZooKeeper Team
Re: Socket is not connected
Thanks for the feedback. It's definitely not a clearcut decision on how to log it. As most ppl tend to run in INFO level on production systems we wanted to keep it visible for the time being. Patrick Stefan Groschupf wrote: Hi Patrick, thanks for the clarification. I can see your point but as user of zookeeper I personal would prefer to have this as debug message. Maybe something like log.debug(WARN: + msg). ... just a user feedback. Thanks, Stefan ~~~ Hadoop training and consulting http://www.scaleunlimited.com http://www.101tec.com On Mar 27, 2009, at 10:19 AM, Patrick Hunt wrote: Hi Stefan, There's a timing issue when the client closes the connection. It sends a session close to the server and then starts to close it's own sockets. If the server closes the socket before the client does you can sometimes see the client get java.net.SocketException: Socket is not connected we are a bit paranoid about making sure you/we can debug unusual situations, so we include the trace in the WARN message in the log, just in case. In this situation though it is not an issue, in other situations (say the server drops it's connection) it would help to have the trace in a debugging situation, so we include it. Btw, in general in the code if we are more concerned about an event we would output as an ERROR, not as a WARN. Patrick Stefan Groschupf wrote: Hi, I managed to upgrade my zookeeper 2 application to zookeeper 3.1.1/ All my tests run though my logs is full of following exception. In my code I basically just invoice ZooKeeper.close(); Any idea what the problem could be? Thanks, Stefan 09/03/27 02:07:51 INFO zookeeper.ZooKeeper:434 - Closing session: 0x120472f36480002 09/03/27 02:07:51 INFO zookeeper.ClientCnxn:999 - Closing ClientCnxn for session: 0x120472f36480002 09/03/27 02:07:51 INFO server.PrepRequestProcessor:360 - Processed session termination request for id: 0x120472f36480002 09/03/27 02:07:51 INFO server.NIOServerCnxn:752 - closing session:0x120472f36480002 NIOServerCnxn: java.nio.channels.SocketChannel[connected local=/0:0:0:0:0:0:0:1%0:2181 remote=/0:0:0:0:0:0:0:1%0:53636] 09/03/27 02:07:51 INFO zookeeper.ClientCnxn:892 - Exception while closing send thread for session 0x120472f36480002 : Read error rc = -1 java.nio.DirectByteBuffer[pos=0 lim=4 cap=4] 09/03/27 02:07:51 WARN zookeeper.ClientCnxn:932 - Ignoring exception during shutdown input java.net.SocketException: Socket is not connected at sun.nio.ch.SocketChannelImpl.shutdown(Native Method) at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:640) at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:360) at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:930) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:915) 09/03/27 02:07:51 INFO zookeeper.ClientCnxn:985 - Disconnecting ClientCnxn for session: 0x120472f36480002 09/03/27 02:07:51 INFO zookeeper.ZooKeeper:442 - Session: 0x120472f36480002 closed
Re: ZooKeeper Perl module
Hey Chris this is really great! Thanks for making it available to the
community, very cool.
Patrick
Chris Darroch wrote:
Hi --
The http://wiki.apache.org/hadoop/ZooKeeper page includes the
comment that someday we hope to get Python, Perl, and REST interfaces.
I hope I can help with one item from that list now, at least. I recently
put together a Perl module named Net::ZooKeeper which is now available
on CPAN:
http://cpan.org/modules/by-category/05_Networking_Devices_IPC/Net/Net-ZooKeeper-0.32.tar.gz
http://search.cpan.org/~cdarroch/Net-ZooKeeper-0.32/ZooKeeper.pm
Modelled on the DBI module, it provides an interface to ZooKeeper
through the synchronous C API functions, e.g.:
my $zkh = Net::ZooKeeper-new('localhost:7000');
my $ret = $zkh-set('/foo', 'baz');
Net::ZooKeeper currently requires ZooKeeper 3.1.1 (or at least
that version of the C API code) and Perl 5.8.8 or up, including 5.10.x.
The test suite is reasonably complete, I think, and covers a
fair bit of ground. I've found it useful for testing the ZooKeeper
C API as well as learning more than I wanted to know about
XS programming.
I've licensed the module under the Apache license 2.0 so it should
be compatible with ZooKeeper itself if there's interest in including
it under src/contribs.
For those who ask why Perl 5 and not Rakudo/Ruby/Lua/Python/
[insert cool new dynamic language here], the answer is just that I
needed an old-style Perl module first. (As a thought experiment,
though, I wonder if one could write a Parrot extension that communicated
directly with ZooKeeper, handled the ping requests internally via
a Parrot scheduler/thread/whatever, and didn't need the C API at all.
You could support any language running on Parrot with that. Well,
maybe in a few years, anyway. :-)
In the meantime, please report any suggestions or bugs to
me -- thanks!
Chris.
[Fwd: [GSoC 2009] Apache is officially participating in Google Summer of Code 2009]
FYI. If anyone has ideas for ZooKeeper that might fit GSoC. Patrick ---BeginMessage--- Dear PMC, It's now official, Google has announced that The Apache Software Foundation was selected as one of the participating organization in the GSoC 2009 program. This is an excellent opportunity for Apache, as it allows projects to build/increase its community, it also help students to learn about open source, about Apache Software Foundation and how to do things the Apache Way. Please start advertising the program and discussing project ideas with your project community. Then add your project ideas to the ASF GSoC 2009 Official Project Ideas page [1], as this is going to be the most visible place for students. Per GSoC 2009 timeline, Students should officially start applying for these ideas on March 23. ASF Members and committers can volunteer to be mentors in the GSoC 2009 program (see [2] for more details about being a mentor). We invite all mentors and GSoC interested parties to subscribe to the code-awa...@a.o mailing list [3] to coordinate work between mentors, ask questions and get updates related to GSoC 2009 program. Please, feel free to forward this announce to any appropriate dev@ or users@ lists so your larger community can hear about the GSoC 2009 program. [1] http://wiki.apache.org/general/SummerOfCode2009 [2] http://wiki.apache.org/general/SummerOfCodeMentor [3] mailto:code-awards-subscr...@apache.org -- Luciano Resende http://people.apache.org/~lresende http://lresende.blogspot.com/ ---End Message---
Re: Preventing SessionExpired events
What are you running for a session timeout on your clients? Can you run with something like jvisualvm or jconsole, and watch the gc activity when the session timeouts occur? Might give you some insight. Have you tried one of the alternative GC's available in the VM? http://developer.amd.com/documentation/articles/pages/4EasyWaystodoJavaGarbageCollectionTuning.aspx ie Flags for Latency Applications We are also working on the following jira: https://issues.apache.org/jira/browse/ZOOKEEPER-321 which will eliminate session expirations for clients w/o ephemerals. (is this the case for you?) Try turning on debug in your client, the client will spit out: LOG.debug(Got ping response for sessionid:0x If you turn on trace logging in the server you should see session updates there as well (c-server, which control session expiration). re HBASE-1316 - how does the jni c wrapper fix this? Isn't the code still running w/in the same (vm) process? Unfortunately I can't think of anything else if it is the GC. Basically you'd have to increase the timeout or try another gc with lower latency. Perhaps Mahadev/Ben/Flavio might have insight... Patrick Nitay wrote: Hey guys, We've recently replaced a few pieces of HBase's cluster management and coordination with ZooKeeper. One of guys, Andrew Purtell, has a cluster that he throws a lot of load at. Andrew's cluster was getting a lot of SessionExpired events which were causing some havoc. After some discussion on the hbase list and additional testing by Andrew (tweaking things like the session timeout, quorum size, and GC used), we suspect the problem is that the Java GC is starving the ZooKeeper hearbeat thread from executing. There is a JIRA open on the matter where Joey suggests a solution that has worked for him: https://issues.apache.org/jira/browse/HBASE-1316 We wanted to loop you guys in to see if you have any thoughts/suggestions on the matter. Thanks, -n
Re: Preventing SessionExpired events
This is good to know. It will allow us to try an replicate the situation, which we haven't been able to do. I'm hoping we can come up with something that we can proactively do to address this... Patrick Nitay wrote: Also, I should mention that some of the errors Andrew was seeing are related to ZOOKEEPER-344: I see this kind of stuff: 2009-04-07 17:58:13,344 - WARN [NIOServerCxn.Factory:2181: nioserverc...@417] - Exception causing close of session 0x2208296c38e due to java.io.IOException: Read error and bye bye HRS ephemeral znodes, which triggers (currently) HBASE-1314. This I think is ZOOKEEPER-344 https://issues.apache.org/jira/browse/ZOOKEEPER-344 - Andy On Wed, Apr 8, 2009 at 12:39 PM, Nitay nit...@gmail.com wrote: Hey guys, We've recently replaced a few pieces of HBase's cluster management and coordination with ZooKeeper. One of guys, Andrew Purtell, has a cluster that he throws a lot of load at. Andrew's cluster was getting a lot of SessionExpired events which were causing some havoc. After some discussion on the hbase list and additional testing by Andrew (tweaking things like the session timeout, quorum size, and GC used), we suspect the problem is that the Java GC is starving the ZooKeeper hearbeat thread from executing. There is a JIRA open on the matter where Joey suggests a solution that has worked for him: https://issues.apache.org/jira/browse/HBASE-1316 We wanted to loop you guys in to see if you have any thoughts/suggestions on the matter. Thanks, -n
New ZooKeeper wiki page on Troubleshooting
I added a new ZK page detailing how to troubleshoot the operating environment. A FAQ of sorts, it has a number of examples of situations we've seen and how to determine/resolve the issue. We'll expand on the content as new issues are seen. http://wiki.apache.org/hadoop/ZooKeeper/Troubleshooting Patrick
FYI: HBase ZooKeeper
New JIRA just up Visibility into ZooKeeper: https://issues.apache.org/jira/browse/HBASE-1329 Patrick
ZooKeeper Python binding now in SVN!
Last night the ZooKeeper Python binding, submitted by Henry Robinson of Cloudera, was committed to the SVN repository. Thanks Henry! We have binding support for the std Java/C as well as Perl, REST, and now Python. Please give these a try and provide any feedback, they will be included in the upcoming 3.2 release of ZooKeeper. If you find issues with the Python bindings, or any contrib binding for that matter, please enter a JIRA against the contrib-bindings component. Patrick
ZooKeeper 3.2 release status
We have been planning for a ZooKeeper 3.2 release at the end of May. There are currently 82 of 101 issues resolved for the release, this is after Mahadev and I scrubbed the bug list for obvious issues that can be pushed to 3.3. Looking at the remaining issues I believe we can realistically push a release if given another 4 weeks or so - many of the remaining issues are simple bug/doc fixes, however there are a few blockers that we should address for 3.2: https://issues.apache.org/jira/browse/ZOOKEEPER?report=com.atlassian.jira.plugin.system.project:roadmap-panel I will update the JIRA version information to push 3.2 release out to the end of June. Let me know if there are any questions/comments. Patrick
FYI: Building a distributed concurrent queue with Apache ZooKeeper
Great ZK article by Henry Robinson of Cloudera: http://www.cloudera.com/blog/2009/05/28/building-a-distributed-concurrent-queue-with-apache-zookeeper/ Patrick
Show your ZooKeeper pride!
The Hadoop summit is Wednesday. If you're attending please feel free to say hi -- Mahadev is presenting @4, Ben and I will be attending as well. Also, regardless of whether you're attending or not we'd appreciate any updates to the powered by page, if you're too busy to update it yourself send us a snippet and we'll update it for you ;-) http://wiki.apache.org/hadoop/ZooKeeper/PoweredBy Regards, Patrick
[VOTE] Release ZooKeeper 3.2.0 (candidate 1)
I have created a candidate build for ZooKeeper 3.2.0. Over 120 JIRAs are addressed in this release, with major new features for; perl/python/rest bindings, flexible quorum support, recipe code libraries, chroot support in connect string, and much more. *** Please download, test and VOTE before the *** vote closes EOD on Monday, July 6.*** http://people.apache.org/~phunt/zookeeper-3.2.0-candidate-1/ Should we release this? Patrick
Re: [VOTE] Release ZooKeeper 3.2.0 (candidate 1)
I'd prefer holding off for 3.3/3.2.1 unless you feel strongly. We're already a few days behind the original release schedule (jun29). Patrick Henry Robinson wrote: Hi Patrick - Looks good at first glance; I'll vote once I've tested more thoroughly. Is there any chance of getting ZOOKEEPER-447 added? It's a tiny patch which lets you specify config files on the command line or in the environment. It's blocked by Hudson at the moment due to the lack of tests; I find it very useful for bringing up an ensemble on a single machine for testing. Let me know if you think it needs dedicated tests - if it does, it's not worth holding the release for. Henry On Wed, Jul 1, 2009 at 6:39 PM, Patrick Hunt ph...@apache.org wrote: I have created a candidate build for ZooKeeper 3.2.0. Over 120 JIRAs are addressed in this release, with major new features for; perl/python/rest bindings, flexible quorum support, recipe code libraries, chroot support in connect string, and much more. *** Please download, test and VOTE before the *** vote closes EOD on Monday, July 6.*** http://people.apache.org/~phunt/zookeeper-3.2.0-candidate-1/http://people.apache.org/%7Ephunt/zookeeper-3.2.0-candidate-1/ Should we release this? Patrick
Re: [VOTE] Release ZooKeeper 3.2.0 (candidate 1)
It looks like you don't have cppunit installed afaict, but I'd feel more comfortable if you could verify. [exec] checking for cppunit-config... no [exec] checking for Cppunit - version = 1.10.2... no I have this on my ubuntu system: libcppunit-1.12-1 install libcppunit-dev install Could you try installing cppunit and re-run? Otw I'll mark you as +1. ;-) Patrick Stu Hood wrote: All of the Java unit tests passed, but the CPPUnit tests failed to build (perhaps I am missing a dependency?). Log attached. If I am just missing a dependency, +1. Otherwise, -1. Thanks, Stu -Original Message- From: Patrick Hunt ph...@apache.org Sent: Wednesday, July 1, 2009 1:39pm To: zookeeper-dev@hadoop.apache.org zookeeper-dev@hadoop.apache.org Subject: [VOTE] Release ZooKeeper 3.2.0 (candidate 1) I have created a candidate build for ZooKeeper 3.2.0. Over 120 JIRAs are addressed in this release, with major new features for; perl/python/rest bindings, flexible quorum support, recipe code libraries, chroot support in connect string, and much more. *** Please download, test and VOTE before the *** vote closes EOD on Monday, July 6.*** http://people.apache.org/~phunt/zookeeper-3.2.0-candidate-1/ Should we release this? Patrick
Re: [VOTE] Release ZooKeeper 3.2.0 (candidate 1)
+1, all tests run, main and contrib seem to be in working order. Patrick Mahadev Konar wrote: +1 Ran the tests and did some basic smoke testing. mahadev On 7/3/09 7:16 AM, Tom White t...@cloudera.com wrote: +1 I checked the signature, ran the Java tests, and tried the CLI. Cheers, Tom On Wed, Jul 1, 2009 at 6:39 PM, Patrick Huntph...@apache.org wrote: I have created a candidate build for ZooKeeper 3.2.0. Over 120 JIRAs are addressed in this release, with major new features for; perl/python/rest bindings, flexible quorum support, recipe code libraries, chroot support in connect string, and much more. *** Please download, test and VOTE before the *** vote closes EOD on Monday, July 6.*** http://people.apache.org/~phunt/zookeeper-3.2.0-candidate-1/ Should we release this? Patrick
Re: [VOTE] Release ZooKeeper 3.2.0 (candidate 1)
With 3 +1 PMC votes and no -1 the vote carries. I will publish the build to h.a.o over the next couple of days and send announcements when available. Thanks everyone who provided feedback/votes! Patrick Patrick Hunt wrote: +1, all tests run, main and contrib seem to be in working order. Patrick Mahadev Konar wrote: +1 Ran the tests and did some basic smoke testing. mahadev On 7/3/09 7:16 AM, Tom White t...@cloudera.com wrote: +1 I checked the signature, ran the Java tests, and tried the CLI. Cheers, Tom On Wed, Jul 1, 2009 at 6:39 PM, Patrick Huntph...@apache.org wrote: I have created a candidate build for ZooKeeper 3.2.0. Over 120 JIRAs are addressed in this release, with major new features for; perl/python/rest bindings, flexible quorum support, recipe code libraries, chroot support in connect string, and much more. *** Please download, test and VOTE before the *** vote closes EOD on Monday, July 6.*** http://people.apache.org/~phunt/zookeeper-3.2.0-candidate-1/ Should we release this? Patrick
[ANNOUNCE] Apache ZooKeeper 3.2.0
The Apache ZooKeeper team is proud to announce Apache ZooKeeper version 3.2.0. ZooKeeper is a high-performance coordination service for distributed applications. It exposes common services - such as naming, configuration management, synchronization, and group services - in a simple interface so you don't have to write them from scratch. You can use it off-the-shelf to implement consensus, group management, leader election, and presence protocols. And you can build on it for your own, specific needs. Key features of the 3.2.0 release: * client side bindings for; Perl, Python, REST * flexible quorum support * re-usable recipe code libraries * chroot support in connect string * many fixes, improvements, improved documentation, etc... A number of optimizations have gone into this release and our benchmarks show that read/write performance of version 3.2.0 is approximately twice that of the previous 3.1.0 version! For ZooKeeper release details and downloads, visit: http://hadoop.apache.org/zookeeper/releases.html ZooKeeper 3.2.0 Release Notes are at: http://hadoop.apache.org/zookeeper/docs/r3.2.0/releasenotes.html Regards, The ZooKeeper Team
ZooKeeper Roadmap - 3.3.0 and beyond.
It's that time again, thinking about the next release of ZK. From what I'm hearing people seem interested in improving scalability, manageability, and continue to improve client support. In particular I believe we should look at the following going forward. Of course ZooKeeper is open to submissions in that aren't on this list. If you have any suggestions please feel free to enter a JIRA, submit a patch, or comment on this thread. http://wiki.apache.org/hadoop/ZooKeeper/HowToContribute 3.3 (~4 months out) Observer support - ZOOKEEPER-368 Allow dynamic changes to ensemble w/o requiring restart - ZOOKEEPER-107 Move from direct NIO to framework (grizzly?) benefit of built-in ssl support for connections optimize session tracking no expiration of session if client has no ephemerals huge scalability win if we can do this (for a very common use case) hide connection loss from client - ZOOKEEPER-22 optimize server profiling for hotspots replace String use with binary buffer - elim conversion overhead (server only) expand system tests ensure b/w compat between current and previous version move some unit tests into system test framework (esp large quorum tests) integrate performance/benchmark tests cleanup the c binding logging and general code cleanliness review jmx object naming scheme error handling, general cruft move examples out of docs into contrib/examples (like hadoop core) add more examples build changes ivy? push jars to maven repo? split jars? common, server, client, test horizontally partitioned zk support multiple client protocols/marshalling (hadoop rpc?) AVRO is in the process of it's first release quota support for limiting (rather than just reporting) 4.0 API/data breakage release (q1 2010) API changes move all counter fields from int - long (version numbers for example) requires migration of snapshots (ie persistence/marshalling will change) pass enums to callbacks remove deprecated cruft bookeeper more tightly integrated into zk? Feel free to respond to this email with any thoughts, suggestions, etc... Regards, Patrick
Re: zookeeper patch builds.
Hi Giri, any update on this? Has infra been able to resolve the space issue? Thanks! Parick Giridharan Kesavan wrote: Zookeeper patch admin jobs running on hudson.zones are failing, as hudson.zones is running out of space. I 've sent an email to the apache infrastructure team. Once I get an update I would restart the zookeeper-patch builds. Thanks, Giri
Need to finalize a 3.2.1 release soon.
As Mahadev mentioned recently, 3.2.0 has a serious issue: https://issues.apache.org/jira/browse/ZOOKEEPER-484 we need to get out an official 3.2.1 release asap. I'm hoping mid/late next week would be possible. There are currently 18 issues on 3.2.1 in jira. Approx 12 of these have patches. Hudson has finally been fixed and is now up, running, and chugging through the patches (hudsonqabot). Committers please review the patches, provide feedback, and contribs address these issues so that we can get the patches committed to svn. I'm hoping we can get the 3.2.1 issues finalized in the next few days, which means a release candidate probably late next week. If you have an issue or know of an issue that's not slated for 3.2.1 currently please speak up ASAP! Regards, Patrick
Re: Hudson patch queue
Please send mail to the zk dev list @apache if there are issues like this the community should be aware of. Thanks, Patrick Mahadev Konar wrote: Hi Flavio, Giri had pinged me yesterday night that there is a problem with the mail servers on these machines and email cannot be sent or received. So, hudson patch process isnt working on these machines. Giri will ping us as soon as the problem is resolved. Thanks mahadev On 8/3/09 9:38 AM, Flavio Junqueira f...@yahoo-inc.com wrote: Hi, I have submitted a couple of patches, but they are not making it to the queue of pending patches to be tested. I have received the notifications that the patches have been submitted, but I can't see them on the queue. I did it on Friday, and I'm trying it again now, but still can't see anything. Does anyone know of an issue with the ZooKeeper queue? Thanks, -Flavio
Re: Unending Leader Elections in WAN deploy
It would be better to create a JIRA with configs as well as logs. Patrick Mahadev Konar wrote: Hi Todd, What is the synclimit you are using? Can you post your config? For WAN's you will have to use much bigger values for synclimit and others. Thanks mahadev On 8/4/09 1:24 PM, Todd Greenwood to...@audiencescience.com wrote: Mahadev, I just heard from IT that this build behaves in exactly the same way as previous versions, e.g. we get continuous leader elections that disconnect the followers and then get re-elected, and disconnect...etc. This is from a fresh sync to the 3.2 branch: svn co http://svn.apache.org/repos/asf/hadoop/zookeeper/branches/branch-3.2 ./branch-3.2 CHANGES.TXT show the various fixes included: to...@toddg01lt:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper /src/original$ head -n 50 branch-3.2/CHANGES.txt Release 3.2.1 Backward compatibile changes: BUGFIXES: ZOOKEEPER-468. avoid compile warning in send_auth_info(). (chris via flavio) ZOOKEEPER-469. make sure CPPUNIT_CFLAGS isn't overwritten (chris via mahadev) ZOOKEEPER-471. update zkperl for 3.2.x branch. (chris via mahadev) ZOOKEEPER-470. include unistd.h for sleep() in c tests (chris via mahadev) ZOOKEEPER-460. bad testRetry in cppunit tests (hudson failure) (giri via mahadev) ZOOKEEPER-467. Change log level in BookieHandle (flavio via mahadev) ZOOKEEPER-482. ignore sigpipe in testRetry to avoid silent immediate failure. (chris via mahadev) ZOOKEEPER-487. setdata on root (/) crashes the servers (mahadev via phunt) ZOOKEEPER-457. Make ZookeeperMain public, support for HBase (and other) embedded clients (ryan rawson via phunt) ZOOKEEPER-481. Add lastMessageSent to QuorumCnxManager. (flavio via mahadev) ZOOKEEPER-479. QuorumHierarchical does not count groups correctly (flavio via mahadev) ZOOKEEPER-466. crash on zookeeper_close() when using auth with empty cert (Chris Darroch via phunt) ZOOKEEPER-480. FLE should perform leader check when node is not leading and add vote of follower (flavio via mahadev) ZOOKEEPER-491. Prevent zero-weight servers from being elected (flavio via mahadev) What can I do to assist you with this issue? -Todd -Original Message- From: Mahadev Konar [mailto:maha...@yahoo-inc.com] Sent: Tuesday, August 04, 2009 12:43 PM To: zookeeper-dev@hadoop.apache.org Subject: Re: Unending Leader Elections in WAN deploy Hi todd, comments in line On 8/4/09 12:38 PM, Todd Greenwood to...@audiencescience.com wrote: Mahadev, Some quick questions: 1. Version I see that the CHANGES.txt calls this 3.2.1, but the build.xml is still calling this 3.2.0. Should this be rev'd, and am I correct in calling this release 3.2.1? Yes the release is 3.2.1. The build.xml will be fixed as soon as we tag the release. 2. Build targets The package target fails b/c the create-cppunit-configure target fails due to various problems w/ respect to autoconf. Are these dependencies documented somewhere ? I'd like to have a fully building system. create-cppunit-configure: [exec] Can't exec libtoolize: No such file or directory at /usr/bin/autoreconf line 188. [exec] Use of uninitialized value $libtoolize in pattern match (m//) at /usr/bin/autoreconf line 188. [exec] configure.ac:33: warning: macro `AM_PATH_CPPUNIT' not found in library [exec] configure.ac:33: error: possibly undefined macro: AM_PATH_CPPUNIT [exec] If this token and others are legitimate, please use m4_pattern_allow. [exec] See the Autoconf documentation. [exec] configure.ac:53: error: possibly undefined macro: AC_PROG_LIBTOOL [exec] autoreconf: /usr/bin/autoconf failed with exit status: 1 You need auto tools to run this. Please read the README for building c client library at src/c/ for the installation requirements. 3. Sync failure: This is still failing. svn: URL 'http://svn.apache.org/repos/asf/hadoop/common/nightly/test-patch' doesn't exist Yes this hasn't been fixed yet! Thanks mahadev -Todd -Original Message- From: Todd Greenwood Sent: Tuesday, August 04, 2009 11:26 AM To: 'zookeeper-u...@hadoop.apache.org' Subject: RE: Unending Leader Elections in WAN deploy Great news. Thank you Mahadev. I'll report our findings later today. -Todd -Original Message- From: Mahadev Konar [mailto:maha...@yahoo-inc.com] Sent: Tuesday, August 04, 2009 11:20 AM To: zookeeper-u...@hadoop.apache.org Subject: Re: Unending Leader Elections in WAN deploy Hi Todd, I just committed 480 and 491. You can checkout the 3.2 branch now. Thanks mahadev On 8/3/09 4:29 PM, Todd Greenwood to...@audiencescience.com wrote: That'd be perfect. Thanks! -Original Message- From: Mahadev Konar [mailto:maha...@yahoo-inc.com] Sent: Monday, August 03, 2009 4:24 PM To: zookeeper-u...@hadoop.apache.org Subject: Re: Unending Leader Elections in WAN deploy Hi Todd, Most of the patches that you mention should be
Re: Unending Leader Elections in WAN deploy
Todd, Mahadev and I looked at this and it turns out to be a regression. Ironically a patch I created for 3.2 branch to add quorum tests actually broke the quorum config -- a default value for a config parameter was lost. I'm going to submit a patch asap to get the default back, but for the time being you can set: electionAlg=3 in each of your config files. You should see reference to FastLeaderElection in your log files if this parameter is set correctly. Sorry for the trouble, Patrick Todd Greenwood wrote: Mahadev, I just heard from IT that this build behaves in exactly the same way as previous versions, e.g. we get continuous leader elections that disconnect the followers and then get re-elected, and disconnect...etc. This is from a fresh sync to the 3.2 branch: svn co http://svn.apache.org/repos/asf/hadoop/zookeeper/branches/branch-3.2 ./branch-3.2 CHANGES.TXT show the various fixes included: to...@toddg01lt:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper /src/original$ head -n 50 branch-3.2/CHANGES.txt Release 3.2.1 Backward compatibile changes: BUGFIXES: ZOOKEEPER-468. avoid compile warning in send_auth_info(). (chris via flavio) ZOOKEEPER-469. make sure CPPUNIT_CFLAGS isn't overwritten (chris via mahadev) ZOOKEEPER-471. update zkperl for 3.2.x branch. (chris via mahadev) ZOOKEEPER-470. include unistd.h for sleep() in c tests (chris via mahadev) ZOOKEEPER-460. bad testRetry in cppunit tests (hudson failure) (giri via mahadev) ZOOKEEPER-467. Change log level in BookieHandle (flavio via mahadev) ZOOKEEPER-482. ignore sigpipe in testRetry to avoid silent immediate failure. (chris via mahadev) ZOOKEEPER-487. setdata on root (/) crashes the servers (mahadev via phunt) ZOOKEEPER-457. Make ZookeeperMain public, support for HBase (and other) embedded clients (ryan rawson via phunt) ZOOKEEPER-481. Add lastMessageSent to QuorumCnxManager. (flavio via mahadev) ZOOKEEPER-479. QuorumHierarchical does not count groups correctly (flavio via mahadev) ZOOKEEPER-466. crash on zookeeper_close() when using auth with empty cert (Chris Darroch via phunt) ZOOKEEPER-480. FLE should perform leader check when node is not leading and add vote of follower (flavio via mahadev) ZOOKEEPER-491. Prevent zero-weight servers from being elected (flavio via mahadev) What can I do to assist you with this issue? -Todd -Original Message- From: Mahadev Konar [mailto:maha...@yahoo-inc.com] Sent: Tuesday, August 04, 2009 12:43 PM To: zookeeper-dev@hadoop.apache.org Subject: Re: Unending Leader Elections in WAN deploy Hi todd, comments in line On 8/4/09 12:38 PM, Todd Greenwood to...@audiencescience.com wrote: Mahadev, Some quick questions: 1. Version I see that the CHANGES.txt calls this 3.2.1, but the build.xml is still calling this 3.2.0. Should this be rev'd, and am I correct in calling this release 3.2.1? Yes the release is 3.2.1. The build.xml will be fixed as soon as we tag the release. 2. Build targets The package target fails b/c the create-cppunit-configure target fails due to various problems w/ respect to autoconf. Are these dependencies documented somewhere ? I'd like to have a fully building system. create-cppunit-configure: [exec] Can't exec libtoolize: No such file or directory at /usr/bin/autoreconf line 188. [exec] Use of uninitialized value $libtoolize in pattern match (m//) at /usr/bin/autoreconf line 188. [exec] configure.ac:33: warning: macro `AM_PATH_CPPUNIT' not found in library [exec] configure.ac:33: error: possibly undefined macro: AM_PATH_CPPUNIT [exec] If this token and others are legitimate, please use m4_pattern_allow. [exec] See the Autoconf documentation. [exec] configure.ac:53: error: possibly undefined macro: AC_PROG_LIBTOOL [exec] autoreconf: /usr/bin/autoconf failed with exit status: 1 You need auto tools to run this. Please read the README for building c client library at src/c/ for the installation requirements. 3. Sync failure: This is still failing. svn: URL 'http://svn.apache.org/repos/asf/hadoop/common/nightly/test-patch' doesn't exist Yes this hasn't been fixed yet! Thanks mahadev -Todd -Original Message- From: Todd Greenwood Sent: Tuesday, August 04, 2009 11:26 AM To: 'zookeeper-u...@hadoop.apache.org' Subject: RE: Unending Leader Elections in WAN deploy Great news. Thank you Mahadev. I'll report our findings later today. -Todd -Original Message- From: Mahadev Konar [mailto:maha...@yahoo-inc.com] Sent: Tuesday, August 04, 2009 11:20 AM To: zookeeper-u...@hadoop.apache.org Subject: Re: Unending Leader Elections in WAN deploy Hi Todd, I just committed 480 and 491. You can checkout the 3.2 branch now. Thanks mahadev On 8/3/09 4:29 PM, Todd Greenwood to...@audiencescience.com wrote: That'd be perfect. Thanks! -Original Message- From: Mahadev Konar
Re: Unending Leader Elections in WAN deploy
Mahadev/Flavio -- looks like 0 weight is still busted, fle0weighttest is actually failing on my machine, however it's reported as success: - Standard Error - Exception in thread Thread-108 junit.framework.AssertionFailedError: Elected zero-weight server at junit.framework.Assert.fail(Assert.java:47) at org.apache.zookeeper.test.FLEZeroWeightTest$LEThread.run(FLEZeroWeightTest.java:138) - --- this is probably due because the test is calling assert in a thread other than the main test thread - which junit will not track/knowabout. One problem I see with these tests (0weight test I looked at) -- it doesn't have a client attempt to connect to the various servers as part of declaring success. Really we should only consider successful test (ie assert that) if a client can connect to each server in the cluster and change/seechanges. As part of fixing this we really need to do a sanity check by testing the various command lines and checking that a client can connect. I'm not even sure FLEnewepochtest/fletest/etc... are passing either. new epoch seems to just thrash... Also I tried 3 5 server quorums by hand from the command line with 0 weight and they see similar issues to what Todd is seeing. I'm using the latest code in mainline btw. Patrick Mahadev Konar wrote: Hi todd, I see a lot of java.net.ConnectException: Connection refused at sun.nio.ch.Net.connect(Native Method) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507) at java.nio.channels.SocketChannel.open(SocketChannel.java:146) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxMana ger.java:324) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager. java:304) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender .process(FastLeaderElection.java:317) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender .run(FastLeaderElection.java:290) at java.lang.Thread.run(Thread.java:619) Is it possible that there is some firewall? Can all the servers 1-9 connect to all the others using ports that you specified in zoo.cfg i.e 2888/3888? Thanks mahadev On 8/4/09 4:56 PM, Todd Greenwood to...@audiencescience.com wrote: Looks like we're not getting *any* leader elected now Logs attached. -Original Message- From: Todd Greenwood [mailto:to...@audiencescience.com] Sent: Tuesday, August 04, 2009 4:07 PM To: zookeeper-dev@hadoop.apache.org Subject: RE: Unending Leader Elections in WAN deploy Patrick, thanks! I'll forward on to IT and I'll report back to you shortly... -Original Message- From: Patrick Hunt [mailto:ph...@apache.org] Sent: Tuesday, August 04, 2009 3:55 PM To: zookeeper-dev@hadoop.apache.org Subject: Re: Unending Leader Elections in WAN deploy Todd, Mahadev and I looked at this and it turns out to be a regression. Ironically a patch I created for 3.2 branch to add quorum tests actually broke the quorum config -- a default value for a config parameter was lost. I'm going to submit a patch asap to get the default back, but for the time being you can set: electionAlg=3 in each of your config files. You should see reference to FastLeaderElection in your log files if this parameter is set correctly. Sorry for the trouble, Patrick Todd Greenwood wrote: Mahadev, I just heard from IT that this build behaves in exactly the same way as previous versions, e.g. we get continuous leader elections that disconnect the followers and then get re-elected, and disconnect...etc. This is from a fresh sync to the 3.2 branch: svn co http://svn.apache.org/repos/asf/hadoop/zookeeper/branches/branch-3.2 ./branch-3.2 CHANGES.TXT show the various fixes included: to...@toddg01lt:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper /src/original$ head -n 50 branch-3.2/CHANGES.txt Release 3.2.1 Backward compatibile changes: BUGFIXES: ZOOKEEPER-468. avoid compile warning in send_auth_info(). (chris via flavio) ZOOKEEPER-469. make sure CPPUNIT_CFLAGS isn't overwritten (chris via mahadev) ZOOKEEPER-471. update zkperl for 3.2.x branch. (chris via mahadev) ZOOKEEPER-470. include unistd.h for sleep() in c tests (chris via mahadev) ZOOKEEPER-460. bad testRetry in cppunit tests (hudson failure) (giri via mahadev) ZOOKEEPER-467. Change log level in BookieHandle (flavio via mahadev) ZOOKEEPER-482. ignore sigpipe in testRetry to avoid silent immediate failure. (chris via mahadev) ZOOKEEPER-487. setdata on root (/) crashes the servers (mahadev via phunt) ZOOKEEPER-457. Make ZookeeperMain public, support for HBase (and other) embedded clients (ryan rawson via phunt) ZOOKEEPER-481. Add lastMessageSent to QuorumCnxManager. (flavio via mahadev) ZOOKEEPER-479. QuorumHierarchical does
Re: Unending Leader Elections in WAN deploy
(I see the same error in fle0weighttest using latest 3.2 btw) Patrick Hunt wrote: Mahadev/Flavio -- looks like 0 weight is still busted, fle0weighttest is actually failing on my machine, however it's reported as success: - Standard Error - Exception in thread Thread-108 junit.framework.AssertionFailedError: Elected zero-weight server at junit.framework.Assert.fail(Assert.java:47) at org.apache.zookeeper.test.FLEZeroWeightTest$LEThread.run(FLEZeroWeightTest.java:138) - --- this is probably due because the test is calling assert in a thread other than the main test thread - which junit will not track/knowabout. One problem I see with these tests (0weight test I looked at) -- it doesn't have a client attempt to connect to the various servers as part of declaring success. Really we should only consider successful test (ie assert that) if a client can connect to each server in the cluster and change/seechanges. As part of fixing this we really need to do a sanity check by testing the various command lines and checking that a client can connect. I'm not even sure FLEnewepochtest/fletest/etc... are passing either. new epoch seems to just thrash... Also I tried 3 5 server quorums by hand from the command line with 0 weight and they see similar issues to what Todd is seeing. I'm using the latest code in mainline btw. Patrick Mahadev Konar wrote: Hi todd, I see a lot of java.net.ConnectException: Connection refused at sun.nio.ch.Net.connect(Native Method) at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:507) at java.nio.channels.SocketChannel.open(SocketChannel.java:146) at org.apache.zookeeper.server.quorum.QuorumCnxManager.connectOne(QuorumCnxMana ger.java:324) at org.apache.zookeeper.server.quorum.QuorumCnxManager.toSend(QuorumCnxManager. java:304) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender .process(FastLeaderElection.java:317) at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender .run(FastLeaderElection.java:290) at java.lang.Thread.run(Thread.java:619) Is it possible that there is some firewall? Can all the servers 1-9 connect to all the others using ports that you specified in zoo.cfg i.e 2888/3888? Thanks mahadev On 8/4/09 4:56 PM, Todd Greenwood to...@audiencescience.com wrote: Looks like we're not getting *any* leader elected now Logs attached. -Original Message- From: Todd Greenwood [mailto:to...@audiencescience.com] Sent: Tuesday, August 04, 2009 4:07 PM To: zookeeper-dev@hadoop.apache.org Subject: RE: Unending Leader Elections in WAN deploy Patrick, thanks! I'll forward on to IT and I'll report back to you shortly... -Original Message- From: Patrick Hunt [mailto:ph...@apache.org] Sent: Tuesday, August 04, 2009 3:55 PM To: zookeeper-dev@hadoop.apache.org Subject: Re: Unending Leader Elections in WAN deploy Todd, Mahadev and I looked at this and it turns out to be a regression. Ironically a patch I created for 3.2 branch to add quorum tests actually broke the quorum config -- a default value for a config parameter was lost. I'm going to submit a patch asap to get the default back, but for the time being you can set: electionAlg=3 in each of your config files. You should see reference to FastLeaderElection in your log files if this parameter is set correctly. Sorry for the trouble, Patrick Todd Greenwood wrote: Mahadev, I just heard from IT that this build behaves in exactly the same way as previous versions, e.g. we get continuous leader elections that disconnect the followers and then get re-elected, and disconnect...etc. This is from a fresh sync to the 3.2 branch: svn co http://svn.apache.org/repos/asf/hadoop/zookeeper/branches/branch-3.2 ./branch-3.2 CHANGES.TXT show the various fixes included: to...@toddg01lt:~/asi/workspaces/main/Main/RSI/etc/holmes/main/zookeeper /src/original$ head -n 50 branch-3.2/CHANGES.txt Release 3.2.1 Backward compatibile changes: BUGFIXES: ZOOKEEPER-468. avoid compile warning in send_auth_info(). (chris via flavio) ZOOKEEPER-469. make sure CPPUNIT_CFLAGS isn't overwritten (chris via mahadev) ZOOKEEPER-471. update zkperl for 3.2.x branch. (chris via mahadev) ZOOKEEPER-470. include unistd.h for sleep() in c tests (chris via mahadev) ZOOKEEPER-460. bad testRetry in cppunit tests (hudson failure) (giri via mahadev) ZOOKEEPER-467. Change log level in BookieHandle (flavio via mahadev) ZOOKEEPER-482. ignore sigpipe in testRetry to avoid silent immediate failure. (chris via mahadev) ZOOKEEPER-487. setdata on root (/) crashes the servers (mahadev via phunt) ZOOKEEPER-457. Make ZookeeperMain public, support for HBase (and other) embedded clients (ryan rawson via phunt) ZOOKEEPER-481. Add
Re: hudson patch build back to normal
Thanks Giri! Patrick Giridharan Kesavan wrote: If you have changed the jira status to patch available in the last couple of days please resubmit your patch for hudson to pick your patch for testing. -Giri -Original Message- From: Giridharan Kesavan [mailto:gkesa...@yahoo-inc.com] Sent: Wednesday, August 05, 2009 7:18 PM To: zookeeper-dev@hadoop.apache.org Cc: Nigel Daley Subject: hudson patch build back to normal Sendmail issues on hudson.zones is fixed now and patch build for zookeeper is restarted. Regards, Giri
ZooKeeper ensemble configuration generator
This is currently more of a developer tool but I thought it might be useful for users as well -- a basic ZooKeeper ensemble configuration generator that takes some of the drudge work out of generating configs. I got sick of creating these by hand for the various setups I have (esp when experimenting) so I decided to build upon an existing templating system. (python/cheetah). It's up to github, feel free to check it out and fork/patches/comments/etc... http://github.com/phunt/zkconf/tree/master Patrick
[VOTE] Release ZooKeeper 3.2.1 (candidate 0)
I've created a candidate build for ZooKeeper 3.2.1. This is a bug fix release addressing a number of significant issues -- see the release notes for details. *** Please download, test and VOTE before the *** vote closes EOD on Wednesday, September 2.*** http://people.apache.org/~phunt/zookeeper-3.2.1-candidate-0/ Should we release this? Patrick
Re: [VOTE] Release ZooKeeper 3.2.1 (candidate 0)
+1 Mahadev Konar wrote: +1 the tests run fine for me. Also ran a bunch of smoke tests with killing servers and bringning them up at randome intervals. It all worked fine. Thanks mahadev On 8/31/09 2:52 PM, Flavio Junqueira f...@yahoo-inc.com wrote: +1. I've run a number of live tests, including some cross-colo tests, and it runs fine for me. -Flavio On Aug 28, 2009, at 8:21 PM, Patrick Hunt wrote: I've created a candidate build for ZooKeeper 3.2.1. This is a bug fix release addressing a number of significant issues -- see the release notes for details. *** Please download, test and VOTE before the *** vote closes EOD on Wednesday, September 2.*** http://people.apache.org/~phunt/zookeeper-3.2.1-candidate-0/ Should we release this? Patrick
Re: [Fwd: [Fwd: [VOTE] Release ZooKeeper 3.2.1 (candidate 0)]]
The configure script is free license (it's in the header of the file). IANAL but based on my understanding of apache license requirements this makes the file a TYPE A license, as there are no other stated requirements wrt notice, we are in compliance. Patrick Bernd Fondermann wrote: Hi, There is foreign code in the release tar ball, which is not attributed to in the top-level LICENSE/NOTICE file. see for example src/c/configure So I guess it is not safe to release this as-is. It would be nice if the release artifacts would be named apache-zookeeper... but this more my personal taste than a requirement. Bernd On Thu, Sep 3, 2009 at 06:02, Patrick Huntph...@apache.org wrote: Hadoop PMC, Please test and vote on this release in zookeeper-dev list. Patrick Hadoop PMC, Please test and vote on this release in zookeeper-dev list. Thanks, Patrick I've created a candidate build for ZooKeeper 3.2.1. This is a bug fix release addressing a number of significant issues -- see the release notes for details. *** Please download, test and VOTE before the *** vote closes EOD on Wednesday, September 2.*** http://people.apache.org/~phunt/zookeeper-3.2.1-candidate-0/ Should we release this? Patrick
Re: [VOTE] Release ZooKeeper 3.2.1 (candidate 0)
With 3 +1 PMC votes and no -1 the vote carries. I will publish the build to h.a.o over the next couple of days and send announcements when available. Thanks everyone who provided feedback/votes! Patrick From: Patrick Hunt [ph...@apache.org] Sent: Friday, August 28, 2009 11:21 AM To: zookeeper-dev@hadoop.apache.org Subject: [VOTE] Release ZooKeeper 3.2.1 (candidate 0) I've created a candidate build for ZooKeeper 3.2.1. This is a bug fix release addressing a number of significant issues -- see the release notes for details. *** Please download, test and VOTE before the *** vote closes EOD on Wednesday, September 2.*** http://people.apache.org/~phunt/zookeeper-3.2.1-candidate-0/ Should we release this? Patrick
Re: [Fwd: [Fwd: [VOTE] Release ZooKeeper 3.2.1 (candidate 0)]]
Bernd Fondermann wrote: Well, I'm not guessing any more than you do. :-) :-) I didn't -1, because I've no binding vote anyway on this PMC. I just checked out the release and gave feedback, whether it's appreciated or not I'm not quite sure. We take all serious votes seriously. Imperative to whom? Hurrying a release never serves well. Votes at Apache should run for at least 72 hrs. It's also convenient to give a timezone and time when the vote ends, because EOD might be at different times for different people. If this is a security fix, however, I think a shorter timeframe is ok. But it is not labelled as such. There is no hurry here. The vote started when my initial email was sent - specifically 08/28/2009 11:21 AM and closed on 09/03/2009 03:36 PM, which is greater than the requisite 72 hrs. You make a good point however, I will include timezone in subsequent emails. Patirck
Re: [Fwd: [Fwd: [VOTE] Release ZooKeeper 3.2.1 (candidate 0)]]
No worries. Thanks for pointing this out. I'm not against giving credit where it's due, so while may be disagreement on whether this is required or not I'm happy to add this to the notice. https://issues.apache.org/jira/browse/ZOOKEEPER-521 If you have any comments on the suggested text please feel free to comment on the jira. Regards, Patrick Bernd Fondermann wrote: On Fri, Sep 4, 2009 at 20:43, Patrick Huntph...@apache.org wrote: Bernd Fondermann wrote: Well, I'm not guessing any more than you do. :-) :-) I didn't -1, because I've no binding vote anyway on this PMC. I just checked out the release and gave feedback, whether it's appreciated or not I'm not quite sure. We take all serious votes seriously. Imperative to whom? Hurrying a release never serves well. Votes at Apache should run for at least 72 hrs. It's also convenient to give a timezone and time when the vote ends, because EOD might be at different times for different people. If this is a security fix, however, I think a shorter timeframe is ok. But it is not labelled as such. There is no hurry here. The vote started when my initial email was sent - specifically 08/28/2009 11:21 AM and closed on 09/03/2009 03:36 PM, which is greater than the requisite 72 hrs. You make a good point however, I will include timezone in subsequent emails. I couldn't make out when the release vote had been started. Looks like a great release then. Thanks for bearing with me, Bernd
[ANNOUNCE] Apache ZooKeeper 3.2.1
The Apache ZooKeeper team is proud to announce Apache ZooKeeper version 3.2.1. ZooKeeper is a high-performance coordination service for distributed applications. It exposes common services - such as naming, configuration management, synchronization, and group services - in a simple interface so you don't have to write them from scratch. You can use it off-the-shelf to implement consensus, group management, leader election, and presence protocols. And you can build on it for your own, specific needs. If you are upgrading from version 2.2.1 on SourceForge be sure to review the 3.0.1 release notes for migration instructions. For ZooKeeper release details and downloads, visit: http://hadoop.apache.org/zookeeper/releases.html ZooKeeper 3.2.1 Release Notes are at: http://hadoop.apache.org/zookeeper/docs/r3.2.1/releasenotes.html Detailed JIRA changlog can be viewed here: http://tinyurl.com/lxltfh Regards, The ZooKeeper Team
Apache ZooKeeper 3.2.1 available on all ASF mirrors.
ZooKeeper 3.2.1 was release over the weekend. If you find any issues with it please create a JIRA. I've attached the announce email which includes links to the release notes and detailed changelog. If you are using 3.2.0 please consider upgrading to 3.2.1 asap. Regards and thanks again to everyone who provided feedback and support on this release! Patrick ---BeginMessage--- The Apache ZooKeeper team is proud to announce Apache ZooKeeper version 3.2.1. ZooKeeper is a high-performance coordination service for distributed applications. It exposes common services - such as naming, configuration management, synchronization, and group services - in a simple interface so you don't have to write them from scratch. You can use it off-the-shelf to implement consensus, group management, leader election, and presence protocols. And you can build on it for your own, specific needs. If you are upgrading from version 2.2.1 on SourceForge be sure to review the 3.0.1 release notes for migration instructions. For ZooKeeper release details and downloads, visit: http://hadoop.apache.org/zookeeper/releases.html ZooKeeper 3.2.1 Release Notes are at: http://hadoop.apache.org/zookeeper/docs/r3.2.1/releasenotes.html Detailed JIRA changlog can be viewed here: http://tinyurl.com/lxltfh Regards, The ZooKeeper Team ---End Message---
Re: Hello, I have a question about the performance improvement from 3.1 to 3.2
I think this is one of the big ones in addition to what you mentioned: https://issues.apache.org/jira/browse/ZOOKEEPER-308 Patrick DaiLionel(???) wrote: Dear sir: It is said In version 3.2 r/w performance improved by ~2x compared to the previous 3.1 releasehttp://hadoop.apache.org/zookeeper/docs/r3.1.1/zookeeperOver.html#Performance. I have diff the src of the tow version, and find out the difference of FileTxnLog.java, which uses BufferedOutputStream. Could you tell me some detail about the performance improvement. Like algorithm, coding, data structure optimization. Thank you for your help. Lionel.Dai
Re: ACL question w/ Zookeeper 3.1.1
Todd, there were other responses as well. Are you seeing other traffic
from the lists? (perhaps a spam filtering issue?)
Patrick
Mahadev Konar wrote:
HI todd,
We did respond on zookeeper-user. Here is my response in case you didn't
see it...
HI todd,
From what I understand, you are sayin that a creator_all_acl does not work
with auth?
I tried the following with CREATOR_ALL_ACL and it seemed to work for me...
import org.apache.zookeeper.CreateMode;
import org.apache.zookeeper.WatchedEvent;
import org.apache.zookeeper.Watcher;
import org.apache.zookeeper.ZooKeeper;
import org.apache.zookeeper.data.ACL;
import org.apache.zookeeper.ZooDefs.Ids;
import java.util.ArrayList;
import java.util.List;
public class TestACl implements Watcher {
public static void main(String[] argv) throws Exception {
ListACL acls = new ArrayListACL(1);
String authentication_type = digest;
String authentication = mahadev:some;
for (ACL ids_acl : Ids.CREATOR_ALL_ACL) {
acls.add(ids_acl);
}
TestACl tacl = new TestACl();
ZooKeeper zoo = new ZooKeeper(localhost:2181, 3000, tacl);
zoo.addAuthInfo(authentication_type, authentication.getBytes());
zoo.create(/some, new byte[0], acls, CreateMode.PERSISTENT);
zoo.setData(/some, new byte[0], -1);
}
@Override
public void process(WatchedEvent event) {
}
}
And it worked on my set of zookeeper servers
And then
I tried
Without auth
Getdata(/some)
Which correctly gave me the error:
Exception in thread main
org.apache.zookeeper.KeeperException$NoAuthException: KeeperErrorCode =
NoAuth for /some
at org.apache.zookeeper.KeeperException.create(KeeperException.java:104)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:892)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:921)
at
org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:692)
at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:579)
at
org.apache.zookeeper.ZooKeeperMain.executeLine(ZooKeeperMain.java:351)
at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:309)
at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:268)
Is this what you are trying to do?
Thanks
mahadev
On 9/18/09 10:33 AM, Todd Greenwood to...@audiencescience.com wrote:
Appologies for cross posting, but I haven't received a response on this.
Quite simply, could someone point me to a working example/tutorial/docs
that describe how to use digest ACLs in zookeeper 3.1.1? The docs that I
have found (referenced below) have not clarified this for me.
-Todd
-Original Message-
From: Todd Greenwood
Sent: Thursday, September 17, 2009 5:05 PM
To: 'zookeeper-u...@hadoop.apache.org'
Subject: ACL question w/ Zookeeper 3.1.1
I'm attempting to secure a zookeeper installation using zookeeper
ACLs.
However, I'm finding that while Ids.OPEN_ACL_UNSAFE works great, my
attempts at using Ids.CREATOR_ALL_ACL are failing. Here's a code
snippet:
public class ZooWrapper
{
/*
1. Here I'm setting up my authentication. I've got an ACL list, and my
authentication strings.
*/
private final ListACL acl = new ArrayListACL( 1 );
private static final String authentication_type = digest;
private static final String authentication =
audiencescience:gravy;
public ZooWrapper( final String connection_string,
final String path,
final int connectiontimeout ) throws
ZooWrapperException
{
...
/*
2. Here I'm adding the acls
*/
// This works (creates nodes, sets data on nodes)
for ( ACL ids_acl : Ids.OPEN_ACL_UNSAFE )
{
acl.add( ids_acl);
}
/*
NOTE: This does not work (nodes are not created, cannot set data on
nodes
b/c nodes do not exist)
*/
//for ( ACL ids_acl : Ids.CREATOR_ALL_ACL )
//{
//acl.add( ids_acl );
//}
/*
3. Finally, I create a new zookeeper instance and add my authorization
info to it.
*/
zoo = new ZooKeeper( connection_string, connectiontimeout, this
);
zoo.addAuthInfo( authentication_type, authentication.getBytes() )
/*
4. Later, I try to write some data into zookeeper by first creating
the
node, and then calling setdata...
*/
zoo.create( path, new byte[0], acl, CreateMode.PERSISTENT );
zoo.setData( path, bytes, -1 )
As I mentioned above, when I add Ids.OPEN_ACL_UNSAFE to acl, then both
the
create and setData succeed. However, when I use Ids.CREATOR_ALL_ACL,
then
the nodes are not created. Am I missing something obvious w/ respect
to
configuring ACLs?
I've used the following references:
http://hadoop.apache.org/zookeeper/docs/r3.1.1/zookeeperProgrammers.html
http://mail-archives.apache.org/mod_mbox/hadoop-zookeeper-
Ivy/pom support for zookeeper
I've submitted a patch to enable use of ivy in zookeeper to pull jar dependencies and generate a pom: https://issues.apache.org/jira/browse/ZOOKEEPER-529 this change will allow us to push the release artifact to a maven repository. If anyone has suggestions or notices issues with this patch please comment on the jira. Patrick
Re: Maven deployment
Hi Hiram, we made some changes recently to support this process: https://issues.apache.org/jira/browse/ZOOKEEPER-529 as part of the patch (ant package) both sh1 md5 are generated for the jar and pom files. See attached for example pom generated during package. Is this sufficient to deploy to a maven repo? If so we should be good to go in 3.3.0 release of ZooKeeper. Patrick Hiram Chirino wrote: FYI: I've deployed your 3.2.1 release to my maven repo. It would be ideal if you guys could GPG sign these guys and then push them out to maven central. http://people.apache.org/~chirino/zk-repo/org/apache/hadoop/zookeeper/zookeeper/3.2.1/ Regards, Hiram ?xml version=1.0 encoding=UTF-8? !-- Apache Maven 2 POM generated by Apache Ivy http://ant.apache.org/ivy/ Apache Ivy version: 2.1.0-rc2 20090704004254 -- project xmlns=http://maven.apache.org/POM/4.0.0; xmlns:xsi=http://www.w3.org/2001/XMLSchema-instance; xsi:schemaLocation=http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd; modelVersion4.0.0/modelVersion groupIdorg.apache.zookeeper/groupId artifactIdzookeeper/artifactId packagingjar/packaging version3.3.0/version dependencies dependency groupIdlog4j/groupId artifactIdlog4j/artifactId version1.2.15/version scopecompile/scope /dependency dependency groupIdjline/groupId artifactIdjline/artifactId version0.9.94/version scopecompile/scope /dependency dependency groupIdjunit/groupId artifactIdjunit/artifactId version4.7/version scopetest/scope /dependency dependency groupIdcheckstyle/groupId artifactIdcheckstyle/artifactId version5.0/version scopetest/scope /dependency dependency groupIdjdiff/groupId artifactIdjdiff/artifactId version1.0.9/version optionaltrue/optional /dependency dependency groupIdxerces/groupId artifactIdxerces/artifactId version1.4.4/version optionaltrue/optional /dependency /dependencies /project
Re: Releases missing source tar balls?
Hi Hiram, you must have missed it - the source is indeed included in the release tar. d/l untar the release, it's in zookeeper-3.2.1/src/... Patrick Hiram Chirino wrote: Hi It looks like the zookeeper releases as missing the source tar balls which are used to create the binary releases. Remember that the Apache policy is to release source distributions. The binary distos are only there as a convenience to our users.
