Re: Restarting discussion on ZooKeeper as a TLP
Excited to see ZooKeeper graduate to TLP. On Wed, Oct 13, 2010 at 2:10 PM, Patrick Hunt wrote: > In March of this year we discussed a request from the Apache Board, and > Hadoop PMC, that we become a TLP rather than a subproject of Hadoop: > > Original discussion > http://markmail.org/thread/42cobkpzlgotcbin > > I originally voted against this move, my primary concern being that we were > not "ready" to move to tlp status given our small contributor base and > limited contributor diversity. However I'd now like to revisit that > discussion/decision. Since that time the team has been working hard to > attract new contributors, and we've seen significant new contributions come > in. There has also been feedback from board/pmc addressing many of these > concerns (both on the list and in private). I am now less concerned about > this issue and don't see it as a blocker for us to move to TLP status. > > A second concern was that by becoming a TLP the project would lose it's > connection with Hadoop, a big source of new users for us. I've been assured > (and you can see with the other projects that have moved to tlp status; > pig/hive/hbase/etc...) that this connection will be maintained. The Hadoop > ZooKeeper tab for example will redirect to our new homepage. > > Other Apache members also pointed out to me that we are essentially > operating as a TLP within the Hadoop PMC. Most of the other PMC members > have > little or no experience with ZooKeeper and this makes it difficult for them > to monitor and advise us. By moving to TLP status we'll be able to govern > ourselves and better set our direction. > > I believe we are ready to become a TLP. Please respond to this email with > your thoughts and any issues. I will call a vote in a few days, once > discussion settles. > > Regards, > > Patrick >
Restarting discussion on ZooKeeper as a TLP
In March of this year we discussed a request from the Apache Board, and Hadoop PMC, that we become a TLP rather than a subproject of Hadoop: Original discussion http://markmail.org/thread/42cobkpzlgotcbin I originally voted against this move, my primary concern being that we were not "ready" to move to tlp status given our small contributor base and limited contributor diversity. However I'd now like to revisit that discussion/decision. Since that time the team has been working hard to attract new contributors, and we've seen significant new contributions come in. There has also been feedback from board/pmc addressing many of these concerns (both on the list and in private). I am now less concerned about this issue and don't see it as a blocker for us to move to TLP status. A second concern was that by becoming a TLP the project would lose it's connection with Hadoop, a big source of new users for us. I've been assured (and you can see with the other projects that have moved to tlp status; pig/hive/hbase/etc...) that this connection will be maintained. The Hadoop ZooKeeper tab for example will redirect to our new homepage. Other Apache members also pointed out to me that we are essentially operating as a TLP within the Hadoop PMC. Most of the other PMC members have little or no experience with ZooKeeper and this makes it difficult for them to monitor and advise us. By moving to TLP status we'll be able to govern ourselves and better set our direction. I believe we are ready to become a TLP. Please respond to this email with your thoughts and any issues. I will call a vote in a few days, once discussion settles. Regards, Patrick
[jira] Commented: (ZOOKEEPER-823) update ZooKeeper java client to optionally use Netty for connections
[ https://issues.apache.org/jira/browse/ZOOKEEPER-823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920750#action_12920750 ] Thomas Koch commented on ZOOKEEPER-823: --- The problem with the failing testWatchAutoResetWithPending might be the following. The test uses testableZooKeeper.pauseCnxn($milliSeconds) PauseCnxn tries to block the ClientCnxn by holding a lock on the ClientCnxn instance, interrupting the connection and not releasing the lock for $milliSeconds. Actually I wonder how this could have ever worked reliable in the first place. The only thing I see that also synchronizes on ClientCnxn is the getXid() method. The solution might be to find a better implementation for testableZooKeeper.pauseCnxn > update ZooKeeper java client to optionally use Netty for connections > > > Key: ZOOKEEPER-823 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-823 > Project: Zookeeper > Issue Type: New Feature > Components: java client >Reporter: Patrick Hunt >Assignee: Patrick Hunt > Fix For: 3.4.0 > > Attachments: NettyNettySuiteTest.rtf, > TEST-org.apache.zookeeper.test.NettyNettySuiteTest.txt.gz, > testDisconnectedAddAuth_FAILURE, testWatchAutoResetWithPending_FAILURE, > ZOOKEEPER-823.patch, ZOOKEEPER-823.patch, ZOOKEEPER-823.patch, > ZOOKEEPER-823.patch, ZOOKEEPER-823.patch, ZOOKEEPER-823.patch, > ZOOKEEPER-823.patch, ZOOKEEPER-823.patch, ZOOKEEPER-823.patch, > ZOOKEEPER-823.patch > > > This jira will port the client side connection code to use netty rather than > direct nio. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-823) update ZooKeeper java client to optionally use Netty for connections
[ https://issues.apache.org/jira/browse/ZOOKEEPER-823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920737#action_12920737 ] Patrick Hunt commented on ZOOKEEPER-823: Thomas (also Mahadev and other committers) Here's an idea. What if we setup a branch in SVN with these changes (netty). It would be a short lived branch. Then we can setup a new job on apache hudson, and just keep hammering on that thing until we get all the issues (intermittent bugs) resolved. How's that sound? An alternative would be to use Git but apache hudson doesn't allow that. I'd create/maintain the svn branch (you need to be a committer). > update ZooKeeper java client to optionally use Netty for connections > > > Key: ZOOKEEPER-823 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-823 > Project: Zookeeper > Issue Type: New Feature > Components: java client >Reporter: Patrick Hunt >Assignee: Patrick Hunt > Fix For: 3.4.0 > > Attachments: NettyNettySuiteTest.rtf, > TEST-org.apache.zookeeper.test.NettyNettySuiteTest.txt.gz, > testDisconnectedAddAuth_FAILURE, testWatchAutoResetWithPending_FAILURE, > ZOOKEEPER-823.patch, ZOOKEEPER-823.patch, ZOOKEEPER-823.patch, > ZOOKEEPER-823.patch, ZOOKEEPER-823.patch, ZOOKEEPER-823.patch, > ZOOKEEPER-823.patch, ZOOKEEPER-823.patch, ZOOKEEPER-823.patch, > ZOOKEEPER-823.patch > > > This jira will port the client side connection code to use netty rather than > direct nio. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-823) update ZooKeeper java client to optionally use Netty for connections
[ https://issues.apache.org/jira/browse/ZOOKEEPER-823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920734#action_12920734 ] Patrick Hunt commented on ZOOKEEPER-823: Sorry man, but I checked that, a few things indicate that it's nio and not netty: 1) The log would have something like this in it: [junit] Running org.apache.zookeeper.test.NettyNettySuiteHammerTest which I don't see prior to this failure (running ... *Netty* 2) I can see log messages from NIO and not from netty in the log (NIO class loggers I mean) 3) I'm pretty sure (don't have one on hand) that if the *Netty*Test fails you see that in the stack trace. > update ZooKeeper java client to optionally use Netty for connections > > > Key: ZOOKEEPER-823 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-823 > Project: Zookeeper > Issue Type: New Feature > Components: java client >Reporter: Patrick Hunt >Assignee: Patrick Hunt > Fix For: 3.4.0 > > Attachments: NettyNettySuiteTest.rtf, > TEST-org.apache.zookeeper.test.NettyNettySuiteTest.txt.gz, > testDisconnectedAddAuth_FAILURE, testWatchAutoResetWithPending_FAILURE, > ZOOKEEPER-823.patch, ZOOKEEPER-823.patch, ZOOKEEPER-823.patch, > ZOOKEEPER-823.patch, ZOOKEEPER-823.patch, ZOOKEEPER-823.patch, > ZOOKEEPER-823.patch, ZOOKEEPER-823.patch, ZOOKEEPER-823.patch, > ZOOKEEPER-823.patch > > > This jira will port the client side connection code to use netty rather than > direct nio. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
Re: What's the QA strategy of ZooKeeper?
On Wed, Oct 13, 2010 at 6:18 AM, Thomas Koch wrote: > I'd propose to have a word about quality assurance. Is there already a > strategy to ensure the ongoing maintainability of ZK? Is there a code style > guide, a list of Dos-And-Donts (where I'd like to add some points)? > > This wiki page has some of the detail you're looking for (code style, testing and such) http://wiki.apache.org/hadoop/ZooKeeper/HowToContribute > Should PMD be added to Hudson? What is the level of FindBugs? Should it be > raised? > It could be, but the issue I've seen with "add tool X" is that it really needs to be "add tool X, configure it reasonably, then hunt down all the issues you just found and require that ppl not introduce any going fwd, setup automated tests to enforce this going fwd". Other wise it becomes just another report that ppl ignore. We had similar issues with findbugs, until recently hadoop decided to make it a priority, everyone went "pencils down" for a few weeks and resolved all the issues. Also put guidelines in place (and automated testing) to ensure no new issues come in. It's a good idea. but not a panacea. > Some of the points I'd like to add to a style guide: > > - Don't write methods longer then 20-40 lines of code > > - Are you sure you want to use inner classes? > > - If there is a new operator in a method? Could the method maybe already > receive the object as a parameter? > > - Are you sure you want to use system properties? They are like global > variables and the IDE does not know about them > > - Are you sure you want to extend a class? Often an aggregation is more > elegant. > > - Don't nest ifs and loops deeper then 2 or 3 levels. If you do so, you > should > better break your code into more methods. > > - Use Enums or constants instead of plain status integers > > - please document your intentions in code comments. You don't need to > comment > the what? but the why?. > > Do you agree with me, that there is a need for better code quality in > ZooKeeper? If so, it's not really scalable if a manic like me fights like > Don > Quichotte to clean up the code. All developers would need to establish a > sense > for clean code and constantly improve the code. > I suspect we are in general agreement, however there are realities to be faced: 1) small team 2) semi-large existing codebase, lots of legacy in there. 3) existing user base Are we open to fixing things, yes.. Do we want to improve things, many in line with what you suggest, yes (ask mahadev, he/I talk about it frequently) But at the same time we can't break existing functionality, we don't like to destabilize the code (and what you are suggesting will cause instability) because then users really get upset. This netty set of changes is not helping things either. I suspect if we can get that out of the way it would help alot, allowing other refactorings to be done much more smoothly. Patrick > > [1] https://issues.apache.org/jira/browse/ZOOKEEPER-835 > > Best regards, > > Thomas Koch, http://www.koch.ro >
[jira] Commented: (ZOOKEEPER-823) update ZooKeeper java client to optionally use Netty for connections
[ https://issues.apache.org/jira/browse/ZOOKEEPER-823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920715#action_12920715 ] Thomas Koch commented on ZOOKEEPER-823: --- Hi Patrick, I saw the same exceptions as you saw. They are not _thrown_ inside the netty code, but they occurred in my cases every time during the NettyNettySuiteTest. It isn't very obvious from the junit log, whether a test was run as part of the NettyNettySuiteTest or not. I do a forward search for "FAILED" and from this position a backwards search for "running" to find the name of the actual test suite. Are you perfecty sure, the failed tests were not run as part of the NettyNettySuiteTest? > update ZooKeeper java client to optionally use Netty for connections > > > Key: ZOOKEEPER-823 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-823 > Project: Zookeeper > Issue Type: New Feature > Components: java client >Reporter: Patrick Hunt >Assignee: Patrick Hunt > Fix For: 3.4.0 > > Attachments: NettyNettySuiteTest.rtf, > TEST-org.apache.zookeeper.test.NettyNettySuiteTest.txt.gz, > testDisconnectedAddAuth_FAILURE, testWatchAutoResetWithPending_FAILURE, > ZOOKEEPER-823.patch, ZOOKEEPER-823.patch, ZOOKEEPER-823.patch, > ZOOKEEPER-823.patch, ZOOKEEPER-823.patch, ZOOKEEPER-823.patch, > ZOOKEEPER-823.patch, ZOOKEEPER-823.patch, ZOOKEEPER-823.patch, > ZOOKEEPER-823.patch > > > This jira will port the client side connection code to use netty rather than > direct nio. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-885) Zookeeper drops connections under moderate IO load
[ https://issues.apache.org/jira/browse/ZOOKEEPER-885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920713#action_12920713 ] Flavio Junqueira commented on ZOOKEEPER-885: I remember a while back fixing an issue with CommitProcessor, which was being killed by a runtime exception. As Pat pointed out, it does look like the pipeline is stalling, but it is still unclear why and I couldn't find anything that can indicate the cause. Let me try to reproduce it. > Zookeeper drops connections under moderate IO load > -- > > Key: ZOOKEEPER-885 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-885 > Project: Zookeeper > Issue Type: Bug > Components: server >Affects Versions: 3.2.2, 3.3.1 > Environment: Debian (Lenny) > 1Gb RAM > swap disabled > 100Mb heap for zookeeper >Reporter: Alexandre Hardy >Priority: Critical > Attachments: tracezklogs.tar.gz, tracezklogs.tar.gz, > WatcherTest.java, zklogs.tar.gz > > > A zookeeper server under minimum load, with a number of clients watching > exactly one node will fail to maintain the connection when the machine is > subjected to moderate IO load. > In a specific test example we had three zookeeper servers running on > dedicated machines with 45 clients connected, watching exactly one node. The > clients would disconnect after moderate load was added to each of the > zookeeper servers with the command: > {noformat} > dd if=/dev/urandom of=/dev/mapper/nimbula-test > {noformat} > The {{dd}} command transferred data at a rate of about 4Mb/s. > The same thing happens with > {noformat} > dd if=/dev/zero of=/dev/mapper/nimbula-test > {noformat} > It seems strange that such a moderate load should cause instability in the > connection. > Very few other processes were running, the machines were setup to test the > connection instability we have experienced. Clients performed no other read > or mutation operations. > Although the documents state that minimal competing IO load should present on > the zookeeper server, it seems reasonable that moderate IO should not cause > problems in this case. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-804) c unit tests failing due to "assertion cptr failed"
[ https://issues.apache.org/jira/browse/ZOOKEEPER-804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920689#action_12920689 ] Jared Cantwell commented on ZOOKEEPER-804: -- It seems like zookeeper_process unnecessarily calls api_prolog() and api_epilog() to begin with. But given that api_prolog() is called at the beginning, if this new code path executes while a close is requested, then the threads will successfully exit, but the final part of zookeeper_close that releases the memory on the last reference will not execute (since the last reference will never be reached). Unless I'm missing something, this will leak the zkhandle. > c unit tests failing due to "assertion cptr failed" > --- > > Key: ZOOKEEPER-804 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-804 > Project: Zookeeper > Issue Type: Bug > Components: c client >Affects Versions: 3.4.0 > Environment: gcc 4.4.3, ubuntu lucid lynx, dual core laptop (intel) >Reporter: Patrick Hunt >Assignee: Michi Mutsuzaki >Priority: Critical > Fix For: 3.3.2, 3.4.0 > > Attachments: ZOOKEEPER-804.patch > > > I'm seeing this frequently: > [exec] Zookeeper_simpleSystem::testPing : elapsed 18006 : OK > [exec] Zookeeper_simpleSystem::testAcl : elapsed 1022 : OK > [exec] Zookeeper_simpleSystem::testChroot : elapsed 3145 : OK > [exec] Zookeeper_simpleSystem::testAuth ZooKeeper server started : > elapsed 25687 : OK > [exec] zktest-mt: > /home/phunt/dev/workspace/gitzk/src/c/src/zookeeper.c:1952: > zookeeper_process: Assertion `cptr' failed. > [exec] make: *** [run-check] Aborted > [exec] Zookeeper_simpleSystem::testHangingClient > Mahadev can you take a look? -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-823) update ZooKeeper java client to optionally use Netty for connections
[ https://issues.apache.org/jira/browse/ZOOKEEPER-823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920686#action_12920686 ] Patrick Hunt commented on ZOOKEEPER-823: I ran the latest patch 4 times on my hudson setup and saw 3 failures (1 each): java.util.concurrent.TimeoutException: Did not connect at org.apache.zookeeper.test.ClientBase$CountdownWatcher.waitForConnected(ClientBase.java:124) at org.apache.zookeeper.test.WatcherTest.testWatchAutoResetWithPending(WatcherTest.java:199) at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:51) org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for / at org.apache.zookeeper.KeeperException.create(KeeperException.java:90) at org.apache.zookeeper.KeeperException.create(KeeperException.java:42) at org.apache.zookeeper.ZooKeeper.setACL(ZooKeeper.java:1259) at org.apache.zookeeper.test.ACLTest.testDisconnectedAddAuth(ACLTest.java:67) at org.apache.zookeeper.JUnit4ZKTestRunner$LoggedInvokeMethod.evaluate(JUnit4ZKTestRunner.java:51) [junit] 2010-10-13 09:16:42,245 [myid:] - INFO [main:junit4zktestrunner$loggedinvokemet...@53] - TEST METHOD FAILED testAcls [junit] org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /0 [junit] at org.apache.zookeeper.KeeperException.create(KeeperException.java:90) [junit] at org.apache.zookeeper.KeeperException.create(KeeperException.java:42) [junit] at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:640) [junit] at org.apache.zookeeper.test.ACLTest.testAcls(ACLTest.java:104) If this was failing in the Netty code I'd be inclined to push the code to trunk and let things settle. However this is in the mainline NIO code and I'm reticent to break existing functionality. Perhaps a good idea would be to have Ben/Mahadev/Flavio review the patch and see if they notice any issues. I'd really like to get this in (esp as it's blocking other changes), so if we could review this and resolve the issues asap that would be great. > update ZooKeeper java client to optionally use Netty for connections > > > Key: ZOOKEEPER-823 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-823 > Project: Zookeeper > Issue Type: New Feature > Components: java client >Reporter: Patrick Hunt >Assignee: Patrick Hunt > Fix For: 3.4.0 > > Attachments: NettyNettySuiteTest.rtf, > TEST-org.apache.zookeeper.test.NettyNettySuiteTest.txt.gz, > testDisconnectedAddAuth_FAILURE, testWatchAutoResetWithPending_FAILURE, > ZOOKEEPER-823.patch, ZOOKEEPER-823.patch, ZOOKEEPER-823.patch, > ZOOKEEPER-823.patch, ZOOKEEPER-823.patch, ZOOKEEPER-823.patch, > ZOOKEEPER-823.patch, ZOOKEEPER-823.patch, ZOOKEEPER-823.patch, > ZOOKEEPER-823.patch > > > This jira will port the client side connection code to use netty rather than > direct nio. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-894) add Package o.a.zookeeper.client
[ https://issues.apache.org/jira/browse/ZOOKEEPER-894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Koch updated ZOOKEEPER-894: -- Description: I'd like to move classes that are not part of the API but belong to the ZK Client into a separate Client package. These classes are: - Inner classes that should become normal classes: Zookeeper.ZkWatchManager Zookeeper.WatchRegistration ClientCnxn.SendThread (should become a Runnable anyhow) ClientCnxn.EventThread ClientCnxn.Package ClientCnxn.AuthData ? - Classes now in the zookeeper package: ClientCnxn -> Client.Cnxn ClientCnxnSocket* -> Client.CnxnSocket* ... Maybe some others that can be moved without breaking the API - Classes yet to be written: PendingQueue ? OutgoingQueue ? was: I'd like to move classes that are not part of the API but belong to the ZK Client into a separate Client package. These classes are: - Inner classes that should become normal classes: Zookeeper.ZkWatchManager Zookeeper.WatchRegistration ClientCnxn.SendThread (should become a Runnable anyhow) ClientCnxn.EventThread ClientCnxn.Package - Classes now in the Zookeeper package: ClientCnxn -> Client.Cnxn ClientCnxnSocket* -> Client.CnxnSocket* ... Maybe some others that can be moved without breaking the API - Classes yet to be written: PendingQueue (?) OutgoingQueue (?) Summary: add Package o.a.zookeeper.client (was: add Package o.a.Zookeeper.Client) > add Package o.a.zookeeper.client > > > Key: ZOOKEEPER-894 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-894 > Project: Zookeeper > Issue Type: Sub-task >Reporter: Thomas Koch > > I'd like to move classes that are not part of the API but belong to the ZK > Client into a separate Client package. These classes are: > - Inner classes that should become normal classes: > Zookeeper.ZkWatchManager > Zookeeper.WatchRegistration > ClientCnxn.SendThread (should become a Runnable anyhow) > ClientCnxn.EventThread > ClientCnxn.Package > ClientCnxn.AuthData ? > - Classes now in the zookeeper package: > ClientCnxn -> Client.Cnxn > ClientCnxnSocket* -> Client.CnxnSocket* > ... Maybe some others that can be moved without breaking the API > - Classes yet to be written: > PendingQueue ? > OutgoingQueue ? -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Resolved: (ZOOKEEPER-890) C client invokes watcher callbacks multiple times
[ https://issues.apache.org/jira/browse/ZOOKEEPER-890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Austin Shoemaker resolved ZOOKEEPER-890. Resolution: Not A Problem Closing, C client works as intended. Submitted a patch in ZOOKEEPER-888 to handle this properly in zkpython. > C client invokes watcher callbacks multiple times > - > > Key: ZOOKEEPER-890 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-890 > Project: Zookeeper > Issue Type: Bug > Components: c client >Affects Versions: 3.3.1 > Environment: Mac OS X 10.6.5 >Reporter: Austin Shoemaker >Priority: Critical > Attachments: watcher_twice.c, ZOOKEEPER-890.patch > > > Code using the C client assumes that watcher callbacks are called exactly > once. If the watcher is called more than once, the process will likely > overwrite freed memory and/or crash. > collect_session_watchers (zk_hashtable.c) gathers watchers from > active_node_watchers, active_exist_watchers, and active_child_watchers > without removing them. This results in watchers being invoked more than once. > Test code is attached that reproduces the bug, along with a proposed patch. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-885) Zookeeper drops connections under moderate IO load
[ https://issues.apache.org/jira/browse/ZOOKEEPER-885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920655#action_12920655 ] Patrick Hunt commented on ZOOKEEPER-885: I notice this in the server log (continuous log): 2010-10-08 08:46:24,789 - DEBUG [ProcessThread:-1:commitproces...@169] - Processing request:: sessionid:0x32b8b03e21e000c type:ping cxid:0xfffe zxid:0xfffe txntype:unknown reqpath:n/a 2010-10-08 08:46:24,789 - DEBUG [CommitProcessor:3:finalrequestproces...@78] - Processing request:: sessionid:0x32b8b03e21e000c type:ping cxid:0xfffe zxid:0xfffe txntype:unknown reqpath:n/a 2010-10-08 08:46:24,789 - DEBUG [CommitProcessor:3:finalrequestproces...@160] - sessionid:0x32b8b03e21e000c type:ping cxid:0xfffe zxid:0xfffe txntype:unknown reqpath:n/a 2010-10-08 08:46:27,986 - DEBUG [ProcessThread:-1:commitproces...@169] - Processing request:: sessionid:0x32b8b03e21e0001 type:ping cxid:0xfffe zxid:0xfffe txntype:unknown reqpath:n/a 2010-10-08 08:46:38,000 - INFO [SessionTracker:zookeeperser...@315] - Expiring session 0x32b8b03e21e0004, timeout of 1ms exceeded 2010-10-08 08:46:46,471 - INFO [SessionTracker:zookeeperser...@315] - Expiring session 0x32b8b03e21e0003, timeout of 1ms exceeded 2010-10-08 08:47:00,083 - INFO [SessionTracker:zookeeperser...@315] - Expiring session 0x32b8b03e21e0001, timeout of 1ms exceeded It looks to me like the pipeline is getting stalled after 0x32b8b03e21e000c. Notice that 0x32b8b03e21e0001 sends in a ping which gets to the commit processor, but you never see the "final request processor" messages (before the session expires). Notice that 30seconds are elapsing here. > Zookeeper drops connections under moderate IO load > -- > > Key: ZOOKEEPER-885 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-885 > Project: Zookeeper > Issue Type: Bug > Components: server >Affects Versions: 3.2.2, 3.3.1 > Environment: Debian (Lenny) > 1Gb RAM > swap disabled > 100Mb heap for zookeeper >Reporter: Alexandre Hardy >Priority: Critical > Attachments: tracezklogs.tar.gz, tracezklogs.tar.gz, > WatcherTest.java, zklogs.tar.gz > > > A zookeeper server under minimum load, with a number of clients watching > exactly one node will fail to maintain the connection when the machine is > subjected to moderate IO load. > In a specific test example we had three zookeeper servers running on > dedicated machines with 45 clients connected, watching exactly one node. The > clients would disconnect after moderate load was added to each of the > zookeeper servers with the command: > {noformat} > dd if=/dev/urandom of=/dev/mapper/nimbula-test > {noformat} > The {{dd}} command transferred data at a rate of about 4Mb/s. > The same thing happens with > {noformat} > dd if=/dev/zero of=/dev/mapper/nimbula-test > {noformat} > It seems strange that such a moderate load should cause instability in the > connection. > Very few other processes were running, the machines were setup to test the > connection instability we have experienced. Clients performed no other read > or mutation operations. > Although the documents state that minimal competing IO load should present on > the zookeeper server, it seems reasonable that moderate IO should not cause > problems in this case. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-885) Zookeeper drops connections under moderate IO load
[ https://issues.apache.org/jira/browse/ZOOKEEPER-885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920646#action_12920646 ] Patrick Hunt commented on ZOOKEEPER-885: Flavio can you take a look at the updated logs? thanks. > Zookeeper drops connections under moderate IO load > -- > > Key: ZOOKEEPER-885 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-885 > Project: Zookeeper > Issue Type: Bug > Components: server >Affects Versions: 3.2.2, 3.3.1 > Environment: Debian (Lenny) > 1Gb RAM > swap disabled > 100Mb heap for zookeeper >Reporter: Alexandre Hardy >Priority: Critical > Attachments: tracezklogs.tar.gz, tracezklogs.tar.gz, > WatcherTest.java, zklogs.tar.gz > > > A zookeeper server under minimum load, with a number of clients watching > exactly one node will fail to maintain the connection when the machine is > subjected to moderate IO load. > In a specific test example we had three zookeeper servers running on > dedicated machines with 45 clients connected, watching exactly one node. The > clients would disconnect after moderate load was added to each of the > zookeeper servers with the command: > {noformat} > dd if=/dev/urandom of=/dev/mapper/nimbula-test > {noformat} > The {{dd}} command transferred data at a rate of about 4Mb/s. > The same thing happens with > {noformat} > dd if=/dev/zero of=/dev/mapper/nimbula-test > {noformat} > It seems strange that such a moderate load should cause instability in the > connection. > Very few other processes were running, the machines were setup to test the > connection instability we have experienced. Clients performed no other read > or mutation operations. > Although the documents state that minimal competing IO load should present on > the zookeeper server, it seems reasonable that moderate IO should not cause > problems in this case. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-894) add Package o.a.Zookeeper.Client
[ https://issues.apache.org/jira/browse/ZOOKEEPER-894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920645#action_12920645 ] Patrick Hunt commented on ZOOKEEPER-894: seems reasonable to me, however can you name the package o.a.zookeeper.client instead? (note all lower case. plz update the summary) > add Package o.a.Zookeeper.Client > > > Key: ZOOKEEPER-894 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-894 > Project: Zookeeper > Issue Type: Sub-task >Reporter: Thomas Koch > > I'd like to move classes that are not part of the API but belong to the ZK > Client into a separate Client package. These classes are: > - Inner classes that should become normal classes: > Zookeeper.ZkWatchManager > Zookeeper.WatchRegistration > ClientCnxn.SendThread (should become a Runnable anyhow) > ClientCnxn.EventThread > ClientCnxn.Package > - Classes now in the Zookeeper package: > ClientCnxn -> Client.Cnxn > ClientCnxnSocket* -> Client.CnxnSocket* > ... Maybe some others that can be moved without breaking the API > - Classes yet to be written: > PendingQueue (?) > OutgoingQueue (?) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Assigned: (ZOOKEEPER-893) ZooKeeper high cpu usage when invalid requests
[ https://issues.apache.org/jira/browse/ZOOKEEPER-893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Hunt reassigned ZOOKEEPER-893: -- Assignee: Thijs Terlouw > ZooKeeper high cpu usage when invalid requests > -- > > Key: ZOOKEEPER-893 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-893 > Project: Zookeeper > Issue Type: Bug > Components: server >Affects Versions: 3.3.1 > Environment: Linux 2.6.16 > 4x Intel(R) Xeon(R) CPU X3320 @ 2.50GHz > java version "1.6.0_17" > Java(TM) SE Runtime Environment (build 1.6.0_17-b04) > Java HotSpot(TM) Server VM (build 14.3-b01, mixed mode) >Reporter: Thijs Terlouw >Assignee: Thijs Terlouw >Priority: Critical > Fix For: 3.3.2, 3.4.0 > > Attachments: ZOOKEEPER-893.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > When ZooKeeper receives certain illegally formed messages on the internal > communication port (:4181 by default), it's possible for ZooKeeper to enter > an infinite loop which causes 100% cpu usage. It's related to ZOOKEEPER-427, > but that patch does not resolve all issues. > from: src/java/main/org/apache/zookeeper/server/quorum/QuorumCnxManager.java > the two affected parts: > === > int length = msgLength.getInt(); > > if(length <= 0) { > > throw new IOException("Invalid packet length:" + length); > > } > === > === > while (message.hasRemaining()) { > > temp_numbytes = channel.read(message); > > if(temp_numbytes < 0) { > > throw new IOException("Channel eof before end"); > > } > > numbytes += temp_numbytes; > > } > === > how to replicate this bug: > perform an nmap portscan against your zookeeper server: "nmap -sV -n > your.ip.here -p4181" > wait for a while untill you see some messages in the logfile and then you > will see 100% cpu usage. It does not recover from this situation. With my > patch, it does not occur anymore -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-893) ZooKeeper high cpu usage when invalid requests
[ https://issues.apache.org/jira/browse/ZOOKEEPER-893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Hunt updated ZOOKEEPER-893: --- Fix Version/s: 3.4.0 3.3.2 > ZooKeeper high cpu usage when invalid requests > -- > > Key: ZOOKEEPER-893 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-893 > Project: Zookeeper > Issue Type: Bug > Components: server >Affects Versions: 3.3.1 > Environment: Linux 2.6.16 > 4x Intel(R) Xeon(R) CPU X3320 @ 2.50GHz > java version "1.6.0_17" > Java(TM) SE Runtime Environment (build 1.6.0_17-b04) > Java HotSpot(TM) Server VM (build 14.3-b01, mixed mode) >Reporter: Thijs Terlouw >Priority: Critical > Fix For: 3.3.2, 3.4.0 > > Attachments: ZOOKEEPER-893.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > When ZooKeeper receives certain illegally formed messages on the internal > communication port (:4181 by default), it's possible for ZooKeeper to enter > an infinite loop which causes 100% cpu usage. It's related to ZOOKEEPER-427, > but that patch does not resolve all issues. > from: src/java/main/org/apache/zookeeper/server/quorum/QuorumCnxManager.java > the two affected parts: > === > int length = msgLength.getInt(); > > if(length <= 0) { > > throw new IOException("Invalid packet length:" + length); > > } > === > === > while (message.hasRemaining()) { > > temp_numbytes = channel.read(message); > > if(temp_numbytes < 0) { > > throw new IOException("Channel eof before end"); > > } > > numbytes += temp_numbytes; > > } > === > how to replicate this bug: > perform an nmap portscan against your zookeeper server: "nmap -sV -n > your.ip.here -p4181" > wait for a while untill you see some messages in the logfile and then you > will see 100% cpu usage. It does not recover from this situation. With my > patch, it does not occur anymore -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Reopened: (ZOOKEEPER-804) c unit tests failing due to "assertion cptr failed"
[ https://issues.apache.org/jira/browse/ZOOKEEPER-804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Hunt reopened ZOOKEEPER-804: Reopening due to issue raised with the patch (see earlier comment). Is this an issue that should be addressed or ok to resolve? > c unit tests failing due to "assertion cptr failed" > --- > > Key: ZOOKEEPER-804 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-804 > Project: Zookeeper > Issue Type: Bug > Components: c client >Affects Versions: 3.4.0 > Environment: gcc 4.4.3, ubuntu lucid lynx, dual core laptop (intel) >Reporter: Patrick Hunt >Assignee: Michi Mutsuzaki >Priority: Critical > Fix For: 3.3.2, 3.4.0 > > Attachments: ZOOKEEPER-804.patch > > > I'm seeing this frequently: > [exec] Zookeeper_simpleSystem::testPing : elapsed 18006 : OK > [exec] Zookeeper_simpleSystem::testAcl : elapsed 1022 : OK > [exec] Zookeeper_simpleSystem::testChroot : elapsed 3145 : OK > [exec] Zookeeper_simpleSystem::testAuth ZooKeeper server started : > elapsed 25687 : OK > [exec] zktest-mt: > /home/phunt/dev/workspace/gitzk/src/c/src/zookeeper.c:1952: > zookeeper_process: Assertion `cptr' failed. > [exec] make: *** [run-check] Aborted > [exec] Zookeeper_simpleSystem::testHangingClient > Mahadev can you take a look? -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-893) ZooKeeper high cpu usage when invalid requests
[ https://issues.apache.org/jira/browse/ZOOKEEPER-893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Hunt updated ZOOKEEPER-893: --- Status: Patch Available (was: Open) > ZooKeeper high cpu usage when invalid requests > -- > > Key: ZOOKEEPER-893 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-893 > Project: Zookeeper > Issue Type: Bug > Components: server >Affects Versions: 3.3.1 > Environment: Linux 2.6.16 > 4x Intel(R) Xeon(R) CPU X3320 @ 2.50GHz > java version "1.6.0_17" > Java(TM) SE Runtime Environment (build 1.6.0_17-b04) > Java HotSpot(TM) Server VM (build 14.3-b01, mixed mode) >Reporter: Thijs Terlouw >Assignee: Thijs Terlouw >Priority: Critical > Fix For: 3.3.2, 3.4.0 > > Attachments: ZOOKEEPER-893.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > When ZooKeeper receives certain illegally formed messages on the internal > communication port (:4181 by default), it's possible for ZooKeeper to enter > an infinite loop which causes 100% cpu usage. It's related to ZOOKEEPER-427, > but that patch does not resolve all issues. > from: src/java/main/org/apache/zookeeper/server/quorum/QuorumCnxManager.java > the two affected parts: > === > int length = msgLength.getInt(); > > if(length <= 0) { > > throw new IOException("Invalid packet length:" + length); > > } > === > === > while (message.hasRemaining()) { > > temp_numbytes = channel.read(message); > > if(temp_numbytes < 0) { > > throw new IOException("Channel eof before end"); > > } > > numbytes += temp_numbytes; > > } > === > how to replicate this bug: > perform an nmap portscan against your zookeeper server: "nmap -sV -n > your.ip.here -p4181" > wait for a while untill you see some messages in the logfile and then you > will see 100% cpu usage. It does not recover from this situation. With my > patch, it does not occur anymore -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-891) Allow non-numeric version strings
[ https://issues.apache.org/jira/browse/ZOOKEEPER-891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Hunt updated ZOOKEEPER-891: --- Fix Version/s: (was: 4.0.0) I noticed a similar issue when I attempted to move us to the scheme hadoop core is using - x.y.z-dev as the default version. We should address this. Slating for 3.4.0 > Allow non-numeric version strings > - > > Key: ZOOKEEPER-891 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-891 > Project: Zookeeper > Issue Type: Improvement > Components: build >Reporter: Eli Collins >Priority: Minor > Fix For: 3.4.0 > > > Non-numeric version strings (eg -dev) or -are not currently accepted, you > either get an error (Invalid version number format, must be "x.y.z") or if > you pass x.y.z-dev or x.y.z+1 you'll get a NumberFormatException. Would be > useful to allow non-numeric versions. > {noformat} > version-info: > [java] All version-related parameters must be valid integers! > [java] Exception in thread "main" java.lang.NumberFormatException: For > input string: "3-dev" > [java] at > java.lang.NumberFormatException.forInputString(NumberFormatException.java:48) > [java] at java.lang.Integer.parseInt(Integer.java:458) > [java] at java.lang.Integer.parseInt(Integer.java:499) > [java] at > org.apache.zookeeper.version.util.VerGen.main(VerGen.java:131) > [java] Java Result: 1 > {noformat} -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-890) C client invokes watcher callbacks multiple times
[ https://issues.apache.org/jira/browse/ZOOKEEPER-890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920619#action_12920619 ] Patrick Hunt commented on ZOOKEEPER-890: Should this issue be closed then? (also sounds like 2 other issues should be opened which were detailed in the comments - docs and python) > C client invokes watcher callbacks multiple times > - > > Key: ZOOKEEPER-890 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-890 > Project: Zookeeper > Issue Type: Bug > Components: c client >Affects Versions: 3.3.1 > Environment: Mac OS X 10.6.5 >Reporter: Austin Shoemaker >Priority: Critical > Attachments: watcher_twice.c, ZOOKEEPER-890.patch > > > Code using the C client assumes that watcher callbacks are called exactly > once. If the watcher is called more than once, the process will likely > overwrite freed memory and/or crash. > collect_session_watchers (zk_hashtable.c) gathers watchers from > active_node_watchers, active_exist_watchers, and active_child_watchers > without removing them. This results in watchers being invoked more than once. > Test code is attached that reproduces the bug, along with a proposed patch. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-860) Add alternative search-provider to ZK site
[ https://issues.apache.org/jira/browse/ZOOKEEPER-860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12920547#action_12920547 ] Otis Gospodnetic commented on ZOOKEEPER-860: For what it's worth, the following issues for the same functionality on other sub-projects have been committed so far: PIG-1661 HBASE-2886 HDFS-1367 > Add alternative search-provider to ZK site > -- > > Key: ZOOKEEPER-860 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-860 > Project: Zookeeper > Issue Type: Improvement > Components: documentation >Reporter: Alex Baranau >Assignee: Alex Baranau >Priority: Minor > Fix For: 3.4.0 > > Attachments: ZOOKEEPER-860.patch > > > Use search-hadoop.com service to make available search in ZK sources, MLs, > wiki, etc. > This was initially proposed on user mailing list > (http://search-hadoop.com/m/sTZ4Y1BVKWg1). The search service was already > added in site's skin (common for all Hadoop related projects) before (as a > part of [AVRO-626|https://issues.apache.org/jira/browse/AVRO-626]) so this > issue is about enabling it for ZK. The ultimate goal is to use it at all > Hadoop's sub-projects' sites. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
What's the QA strategy of ZooKeeper?
Hi, after filling 13 refactoring issues against the Java Client code[1], I started to dig into the server site code to understand the last issues with the Netty stuff. I feel bad. It's this feeling of "I don't wanna hurt you, but...". ZooKeeper is quite an important piece of the Hadoop ecosystem containing some of the most complicated pieces of code. And it'll only get more complex with more features. I'd propose to have a word about quality assurance. Is there already a strategy to ensure the ongoing maintainability of ZK? Is there a code style guide, a list of Dos-And-Donts (where I'd like to add some points)? Should PMD be added to Hudson? What is the level of FindBugs? Should it be raised? Some of the points I'd like to add to a style guide: - Don't write methods longer then 20-40 lines of code - Are you sure you want to use inner classes? - If there is a new operator in a method? Could the method maybe already receive the object as a parameter? - Are you sure you want to use system properties? They are like global variables and the IDE does not know about them - Are you sure you want to extend a class? Often an aggregation is more elegant. - Don't nest ifs and loops deeper then 2 or 3 levels. If you do so, you should better break your code into more methods. - Use Enums or constants instead of plain status integers - please document your intentions in code comments. You don't need to comment the what? but the why?. Do you agree with me, that there is a need for better code quality in ZooKeeper? If so, it's not really scalable if a manic like me fights like Don Quichotte to clean up the code. All developers would need to establish a sense for clean code and constantly improve the code. [1] https://issues.apache.org/jira/browse/ZOOKEEPER-835 Best regards, Thomas Koch, http://www.koch.ro
[jira] Updated: (ZOOKEEPER-894) add Package o.a.Zookeeper.Client
[ https://issues.apache.org/jira/browse/ZOOKEEPER-894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Koch updated ZOOKEEPER-894: -- Description: I'd like to move classes that are not part of the API but belong to the ZK Client into a separate Client package. These classes are: - Inner classes that should become normal classes: Zookeeper.ZkWatchManager Zookeeper.WatchRegistration ClientCnxn.SendThread (should become a Runnable anyhow) ClientCnxn.EventThread ClientCnxn.Package - Classes now in the Zookeeper package: ClientCnxn -> Client.Cnxn ClientCnxnSocket* -> Client.CnxnSocket* ... Maybe some others that can be moved without breaking the API - Classes yet to be written: PendingQueue (?) OutgoingQueue (?) > add Package o.a.Zookeeper.Client > > > Key: ZOOKEEPER-894 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-894 > Project: Zookeeper > Issue Type: Sub-task >Reporter: Thomas Koch > > I'd like to move classes that are not part of the API but belong to the ZK > Client into a separate Client package. These classes are: > - Inner classes that should become normal classes: > Zookeeper.ZkWatchManager > Zookeeper.WatchRegistration > ClientCnxn.SendThread (should become a Runnable anyhow) > ClientCnxn.EventThread > ClientCnxn.Package > - Classes now in the Zookeeper package: > ClientCnxn -> Client.Cnxn > ClientCnxnSocket* -> Client.CnxnSocket* > ... Maybe some others that can be moved without breaking the API > - Classes yet to be written: > PendingQueue (?) > OutgoingQueue (?) -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (ZOOKEEPER-894) add Package o.a.Zookeeper.Client
add Package o.a.Zookeeper.Client Key: ZOOKEEPER-894 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-894 Project: Zookeeper Issue Type: Sub-task Reporter: Thomas Koch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-823) update ZooKeeper java client to optionally use Netty for connections
[ https://issues.apache.org/jira/browse/ZOOKEEPER-823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Koch updated ZOOKEEPER-823: -- Attachment: testWatchAutoResetWithPending_FAILURE I did 10 builds now in total with the last patch. 8 builds were succesful, 2 failed with different test cases, see the *_FAILURE files. Both failures happened in the NettyNettySuiteTest. It's hard to debug this kind of Heisenbugs that happen only very seldom. It would make things easier, if I could start cleaning up the code to better identify spots where things could go wrong. > update ZooKeeper java client to optionally use Netty for connections > > > Key: ZOOKEEPER-823 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-823 > Project: Zookeeper > Issue Type: New Feature > Components: java client >Reporter: Patrick Hunt >Assignee: Patrick Hunt > Fix For: 3.4.0 > > Attachments: NettyNettySuiteTest.rtf, > TEST-org.apache.zookeeper.test.NettyNettySuiteTest.txt.gz, > testDisconnectedAddAuth_FAILURE, testWatchAutoResetWithPending_FAILURE, > ZOOKEEPER-823.patch, ZOOKEEPER-823.patch, ZOOKEEPER-823.patch, > ZOOKEEPER-823.patch, ZOOKEEPER-823.patch, ZOOKEEPER-823.patch, > ZOOKEEPER-823.patch, ZOOKEEPER-823.patch, ZOOKEEPER-823.patch, > ZOOKEEPER-823.patch > > > This jira will port the client side connection code to use netty rather than > direct nio. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.