[jira] Commented: (ZOOKEEPER-661) Add Ruby bindings
[ https://issues.apache.org/jira/browse/ZOOKEEPER-661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12837572#action_12837572 ] Andrew Reynhout commented on ZOOKEEPER-661: --- Patrick, I agree on the collaboration recommendation. I haven't had a chance to go through all of the changes in Eric's fork yet, so I'm not sure how the two attempts compare in terms of functional completeness. I'll get in touch with Eric and Evan and see how they feel. The primary difference in the two approaches is that we're using FFI and they are using a straight C extension. FFI should make it simpler to keep up with any ZK API changes, and make the code more conveniently portable to JRuby and other platforms, but that might not be a big deal -- ZK is at 3.3, and the C API has been pretty stable. Re: ASF vs github, I think it'd be great to have Ruby bindings in the official distribution, as soon as the bindings are worthy. Github and rubygems definitely make it easier to iterate on the bindings independently of the ZK/ASF release process, but theoretically at some point the bindings will catch up and ZK changes will drive the binding iterations. Thanks, Andrew Add Ruby bindings - Key: ZOOKEEPER-661 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-661 Project: Zookeeper Issue Type: New Feature Components: contrib-bindings Environment: MRI Ruby 1.9 JRuby 1.4 Reporter: Andrew Reynhout Priority: Minor Add Ruby bindings to the ZooKeeper distribution. Ruby presents special threading difficulties for asynchronous ZK calls (aget, watchers, etc). It looks like the simplest workaround is to patch the ZK C API. Proposed approach will be described in comment. Please use this ticket for discussion and suggestions. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-661) Add Ruby bindings
[ https://issues.apache.org/jira/browse/ZOOKEEPER-661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12836239#action_12836239 ] Andrew Reynhout commented on ZOOKEEPER-661: --- Looks like emaland forked the old myelin code. We looked at that, but ultimately decided to start over using Ruby FFI. Also, it doesn't look like they've done async calls and watchers yet, but maybe I missed it. Asynchronous calls were never high priority for my use case, but watchers were essential. Both ran into the same threading problems though, so I ended up using the same workarounds. I've been a bit distracted for the last few weeks, so I haven't made any progress on the MT vs ST idea. I've moved all of the code into a public github repo, so if it's useful please check it out. http://github.com/ce/ruby-zookeeper Add Ruby bindings - Key: ZOOKEEPER-661 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-661 Project: Zookeeper Issue Type: New Feature Components: contrib-bindings Environment: MRI Ruby 1.9 JRuby 1.4 Reporter: Andrew Reynhout Priority: Minor Add Ruby bindings to the ZooKeeper distribution. Ruby presents special threading difficulties for asynchronous ZK calls (aget, watchers, etc). It looks like the simplest workaround is to patch the ZK C API. Proposed approach will be described in comment. Please use this ticket for discussion and suggestions. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-661) Add Ruby bindings
[ https://issues.apache.org/jira/browse/ZOOKEEPER-661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12828709#action_12828709 ] Andrew Reynhout commented on ZOOKEEPER-661: --- Hi Patrick, The threading trickiness is similar to Python's: no true native threads, and a big Global Interpreter Lock (Ruby calls it the GVL). Unlike Python, Ruby's mechanisms exposed for external synchronization are (appear to be) awkward and very sparsely documented. I'm no expert on Ruby internals either, but I spent some time trolling through the 1.8 and 1.9 source trees to find a way to make this work...this is the best solution I could come up with. Your point about the ZK ST vs MT libraries is interesting. I considered that approach at one point, but abandoned it because I thought it wouldn't work for watchers. I don't remember what made me think that -- I've been working on this very sporadically for a while. If I'm following the concept, I would just manage all of the threading within Ruby and let the async calls (and watchers) wait in Ruby threads for the libzookeeper_st function to return..? I'll take a look at that and see what I can learn. Thank you for the pointers. Add Ruby bindings - Key: ZOOKEEPER-661 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-661 Project: Zookeeper Issue Type: New Feature Components: contrib-bindings Environment: MRI Ruby 1.9 JRuby 1.4 Reporter: Andrew Reynhout Priority: Minor Add Ruby bindings to the ZooKeeper distribution. Ruby presents special threading difficulties for asynchronous ZK calls (aget, watchers, etc). It looks like the simplest workaround is to patch the ZK C API. Proposed approach will be described in comment. Please use this ticket for discussion and suggestions. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-661) Add Ruby bindings
[ https://issues.apache.org/jira/browse/ZOOKEEPER-661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12828782#action_12828782 ] Andrew Reynhout commented on ZOOKEEPER-661: --- I used EventMachine an early attempt, but couldn't get it to work without also patching ZK. I tried Rev too, but not extensively because I don't know it as well as EM. I'll try again though, perhaps I've learned something important since then. :) Merci. Add Ruby bindings - Key: ZOOKEEPER-661 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-661 Project: Zookeeper Issue Type: New Feature Components: contrib-bindings Environment: MRI Ruby 1.9 JRuby 1.4 Reporter: Andrew Reynhout Priority: Minor Add Ruby bindings to the ZooKeeper distribution. Ruby presents special threading difficulties for asynchronous ZK calls (aget, watchers, etc). It looks like the simplest workaround is to patch the ZK C API. Proposed approach will be described in comment. Please use this ticket for discussion and suggestions. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (ZOOKEEPER-661) Add Ruby bindings
Add Ruby bindings - Key: ZOOKEEPER-661 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-661 Project: Zookeeper Issue Type: New Feature Components: contrib-bindings Environment: MRI Ruby 1.9 JRuby 1.4 Reporter: Andrew Reynhout Priority: Minor Add Ruby bindings to the ZooKeeper distribution. Ruby presents special threading difficulties for asynchronous ZK calls (aget, watchers, etc). It looks like the simplest workaround is to patch the ZK C API. Proposed approach will be described in comment. Please use this ticket for discussion and suggestions. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-661) Add Ruby bindings
[ https://issues.apache.org/jira/browse/ZOOKEEPER-661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12806730#action_12806730 ] Andrew Reynhout commented on ZOOKEEPER-661: --- I have a WIP patch for the ZK C API, and an alpha level Ruby library that works fairly well. The following is a diff of the ZK C API patch (excerpted for clarity), for example and discussion. The full patch is basically as below, repeated multiple times for multiple completion types, plus some autoconf tweaks to enable the ruby-specific code. I can attach the full patch, but this is much less noisy and seems much more useful at this stage. {code:title=zookeeper.c process_completions() diff} case COMPLETION_STAT: LOG_DEBUG((Calling COMPLETION_STAT for xid=%x rc=%d,cptr-xid,rc)); if (rc) { +#ifdef HAVE_LIBRUBY +if (getenv(RUBY_MRI_VERSION) != NULL) +ruby_stat_completion_wrapper(cptr-c.stat_result, rc, 0, cptr-data); +else +#endif cptr-c.stat_result(rc, 0, cptr-data); } else { struct SetDataResponse res; deserialize_SetDataResponse(ia, reply, res); +#ifdef HAVE_LIBRUBY +if (getenv(RUBY_MRI_VERSION) != NULL) +ruby_stat_completion_wrapper(cptr-c.stat_result, rc, res.stat, cptr-data); +else +#endif cptr-c.stat_result(rc, res.stat, cptr-data); deallocate_SetDataResponse(res); } {code} And this is the new code to support the Ruby bindings. {code:title=zookeeper.c ruby support functions} static struct Stat *stat_dup(struct Stat *stat) { struct Stat *stat_copy = malloc(sizeof(struct Stat)); memcpy(stat_copy,stat,sizeof(struct Stat)); return stat_copy; } typedef struct stat_completion_data { stat_completion_t dc; int rc; struct Stat *stat; void *ctx; } stat_completion_data; static void ruby_stat_completion_wrapper_2(struct stat_completion_data *cb) { cb-dc(cb-rc, cb-stat, cb-ctx); free(cb); } static void ruby_stat_completion_wrapper(stat_completion_t dc, int rc, struct Stat *stat, void *ctx) { struct stat_completion_data *cb; cb = (stat_completion_data *)malloc(sizeof(stat_completion_data)); cb-dc = dc; cb-rc = rc; cb-stat = stat ? stat_dup(stat) : 0; cb-ctx = ctx ? strdup(ctx) : 0; rb_thread_create((void *)ruby_stat_completion_wrapper_2, cb); } {code} *Known Issues* - I'm not treating the context pointer properly (should be struct buffer), and the strdup() is egregious -- but surprisingly this _works_ under MRI (standard) Ruby, and the correct approach does not...! However: - In JRuby, the context data is mangled. This might be my bug (I can't get good data out of ctx-buff and ctx-len... inspection of the pointer address doesn't look like the expected struct layout either..), or possibly a quirk of the data structure expectations in JRuby FFI. WIP, but lowered priority -- a) I don't think passing contexts into completions serve much purpose in Ruby, and b) I need to do some more reading to figure out next steps. - Memory allocations are not freed properly in a few cases -- I think I need to allocate the space with a C-Ruby call so that Ruby can GC it when appropriate, because the C thread could be long gone by then. This is small but since the Ruby code might be long-running, becomes important. - Checking an environment variable to decide whether to wrap the callback in a Ruby thread is ugly. However, JRuby doesn't require thread wrapping due to superior JVM threading (but might to resolve the context struct problem, TBD) so I can't just test a Ruby global that would be set by both, and I don't want to wrap all callbacks just because libruby was included at compile time. New versions of JRuby set a unique global that I could use in conjunction with the standard Ruby globals to test ruby-but-not-jruby, but that would cause failures on older JRuby versions. Does this code (or the vastly improved version of it that will result from the ensuing discussions!) look like something that could/should be part of the standard ZooKeeper API/contrib distribution? I'm currently bundling the ZK C API diff with the ruby-zookeeper Ruby library, but that isn't ideal. The ruby-zookeeper gem will be released under AL2.0, which matches ZooKeeper obviously. Thank you for any comments or suggestions. Add Ruby bindings - Key: ZOOKEEPER-661 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-661 Project: Zookeeper Issue Type: New Feature Components: contrib-bindings Environment: MRI Ruby 1.9 JRuby 1.4 Reporter: Andrew Reynhout Priority:
[jira] Created: (ZOOKEEPER-634) some elements of struct Stat have incorrect values when returned asynchronously
some elements of struct Stat have incorrect values when returned asynchronously --- Key: ZOOKEEPER-634 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-634 Project: Zookeeper Issue Type: Bug Components: c client Affects Versions: 3.2.1 Environment: OSX 10.5, 10.6 (32b and 64b, respectively) Reporter: Andrew Reynhout Priority: Minor I'm working on Ruby bindings for ZK (3.2.2) via the C API. This involves some painful kludgery due to MRI Ruby's threading model, but I believe this bug report is valid: Some of the elements of Stat structs, when returned via the C client, have values inconsistent with the synchronous call, zkCli output, and expectations. E.g.: {noformat:title=zkCli command line} $ zk/bin/zkCli.sh -server localhost:2182 set /lala lala234 Connecting to localhost:2182 WATCHER:: WatchedEvent: Server state change. New state: SyncConnected cZxid = 3 ctime = Sat Dec 19 14:32:03 EST 2009 mZxid = 69 mtime = Sat Dec 19 15:31:46 EST 2009 pZxid = 3 cversion = 0 dataVersion = 4 aclVersion = 0 ephemeralOwner = 0 dataLength = 7 numChildren = 0 {noformat} {noformat:title=Ruby script + verbose ZK log output} 2009-12-19 15:31:48,952:45753(0x751000):zoo_de...@process_completions@1902: Switching on cptr-completion_type (async) 2009-12-19 15:31:48,952:45753(0x751000):zoo_de...@process_completions@1916: Calling COMPLETION_STAT for xid=4b2d3837 rc=0 2009-12-19 15:31:48,952:45753(0x751000):zoo_de...@process_completions@1949: --- PRE ruby callback. --- res.stat0x00750e30 .czxid3 .mzxid69 .ctime1261251123149 .mtime1261254706180 .version 4 .cversion 0 .aversion 0 .ephemeralOwner 0 .dataLength 7 .numChildren 0 .pzxid3 === 2009-12-19 15:31:48,952:45753(0x751000):zoo_de...@ruby_stat_completion_wrapper@1702: --- creating ruby thread. 2009-12-19 15:31:48,952:45753(0x7ee000):zoo_de...@ruby_stat_completion_wrapper_2@1685: --- invoking ruby callback. --- cb-dc 0x002fe000 cb-rc 0 cb-ctx0x01a03870 cb-stat 0x00750e30 -czxid3 -mzxid69 -ctime1261251123149 -mtime140735387442616 -version 0 -cversion 0 -aversion 27264392 -ephemeralOwner 4302638816 -dataLength -2100901903 -numChildren 32767 -pzxid3 === 2009-12-19 15:31:48,952:45753(0x7ee000):zoo_de...@ruby_stat_completion_wrapper_2@1687: --- callback returned, freeing struct. 2009-12-19 15:31:48,952:45753(0x7ee000):zoo_de...@ruby_stat_completion_wrapper_2@1689: --- struct freed. sync returned #ZooKeeperFFI::Stat:0x01013783d0 async callback.return_code 0 attr ?? sync async -- - czxid OK 3 3 mzxid OK 69 69 ctime OK 1261251123149 1261251123149 mtime -- 1261254706180140735387442616 version -- 4 0 cversion OK 0 0 aversion -- 0 27264392 ephemeralOwner -- 0 4302638816 dataLength -- 7-2100901903 numChildren -- 0 32767 pzxid OK 3 3 2009-12-19 15:31:49,052:45753(0x6ce000):zoo_de...@do_io@316: IO thread terminated 2009-12-19 15:31:49,052:45753(0x751000):zoo_de...@do_completion@335: completion thread terminated 2009-12-19 15:31:49,052:45753(0x705f3be0):zoo_i...@zookeeper_close@2407: Closing zookeeper session 125a86bdc640020 to [127.0.0.1:2182] {noformat} The last set of lines, above, are output from my Ruby code, comparing the differences between sync and async calls. I added the LOG_DEBUG in zookeeper.c to dump the contents of res.stat before it gets shipped off to Ruby...as you can see, the LOG_DEBUG values match the values in the async Ruby call, but they're wrong. I've currently only tested on OSX 10.5 (32bit) and 10.6 (64bit), but sometimes different elements are wrong on each. I can get a comprehensive list if it's useful, but as examples: {noformat:title=data variation examples} ## aexists # ctime (int64_t): # 32b OK # 64b OK # dataLength (int32_t): # 32b OK # 64b NG (== -2100901903 instead of 10) ## aget # ctime (int64_t): # 32b OK # 64b NG (== 0 instead of something unixtimey e.g. 1261251123149) #
[jira] Resolved: (ZOOKEEPER-634) some elements of struct Stat have incorrect values when returned asynchronously
[ https://issues.apache.org/jira/browse/ZOOKEEPER-634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Reynhout resolved ZOOKEEPER-634. --- Resolution: Invalid Sorry, some of the data included is invalid. I'll repost when confirmed, if necessary. :-/ some elements of struct Stat have incorrect values when returned asynchronously --- Key: ZOOKEEPER-634 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-634 Project: Zookeeper Issue Type: Bug Components: c client Affects Versions: 3.2.1 Environment: OSX 10.5, 10.6 (32b and 64b, respectively) Reporter: Andrew Reynhout Priority: Minor I'm working on Ruby bindings for ZK (3.2.2) via the C API. This involves some painful kludgery due to MRI Ruby's threading model, but I believe this bug report is valid: Some of the elements of Stat structs, when returned via the C client, have values inconsistent with the synchronous call, zkCli output, and expectations. E.g.: {noformat:title=zkCli command line} $ zk/bin/zkCli.sh -server localhost:2182 set /lala lala234 Connecting to localhost:2182 WATCHER:: WatchedEvent: Server state change. New state: SyncConnected cZxid = 3 ctime = Sat Dec 19 14:32:03 EST 2009 mZxid = 69 mtime = Sat Dec 19 15:31:46 EST 2009 pZxid = 3 cversion = 0 dataVersion = 4 aclVersion = 0 ephemeralOwner = 0 dataLength = 7 numChildren = 0 {noformat} {noformat:title=Ruby script + verbose ZK log output} 2009-12-19 15:31:48,952:45753(0x751000):zoo_de...@process_completions@1902: Switching on cptr-completion_type (async) 2009-12-19 15:31:48,952:45753(0x751000):zoo_de...@process_completions@1916: Calling COMPLETION_STAT for xid=4b2d3837 rc=0 2009-12-19 15:31:48,952:45753(0x751000):zoo_de...@process_completions@1949: --- PRE ruby callback. --- res.stat0x00750e30 .czxid3 .mzxid69 .ctime1261251123149 .mtime1261254706180 .version 4 .cversion 0 .aversion 0 .ephemeralOwner 0 .dataLength 7 .numChildren 0 .pzxid3 === 2009-12-19 15:31:48,952:45753(0x751000):zoo_de...@ruby_stat_completion_wrapper@1702: --- creating ruby thread. 2009-12-19 15:31:48,952:45753(0x7ee000):zoo_de...@ruby_stat_completion_wrapper_2@1685: --- invoking ruby callback. --- cb-dc 0x002fe000 cb-rc 0 cb-ctx0x01a03870 cb-stat 0x00750e30 -czxid3 -mzxid69 -ctime1261251123149 -mtime140735387442616 -version 0 -cversion 0 -aversion 27264392 -ephemeralOwner 4302638816 -dataLength -2100901903 -numChildren 32767 -pzxid3 === 2009-12-19 15:31:48,952:45753(0x7ee000):zoo_de...@ruby_stat_completion_wrapper_2@1687: --- callback returned, freeing struct. 2009-12-19 15:31:48,952:45753(0x7ee000):zoo_de...@ruby_stat_completion_wrapper_2@1689: --- struct freed. sync returned #ZooKeeperFFI::Stat:0x01013783d0 async callback.return_code 0 attr ?? sync async -- - czxid OK 3 3 mzxid OK 69 69 ctime OK 1261251123149 1261251123149 mtime -- 1261254706180140735387442616 version -- 4 0 cversion OK 0 0 aversion -- 0 27264392 ephemeralOwner -- 0 4302638816 dataLength -- 7-2100901903 numChildren -- 0 32767 pzxid OK 3 3 2009-12-19 15:31:49,052:45753(0x6ce000):zoo_de...@do_io@316: IO thread terminated 2009-12-19 15:31:49,052:45753(0x751000):zoo_de...@do_completion@335: completion thread terminated 2009-12-19 15:31:49,052:45753(0x705f3be0):zoo_i...@zookeeper_close@2407: Closing zookeeper session 125a86bdc640020 to [127.0.0.1:2182] {noformat} The last set of lines, above, are output from my Ruby code, comparing the differences between sync and async calls. I added the LOG_DEBUG in zookeeper.c to dump the contents of res.stat before it gets shipped off to Ruby...as you can see, the LOG_DEBUG values match the values in the async Ruby call, but they're wrong. I've currently only tested on
[jira] Commented: (ZOOKEEPER-634) some elements of struct Stat have incorrect values when returned asynchronously
[ https://issues.apache.org/jira/browse/ZOOKEEPER-634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12792942#action_12792942 ] Andrew Reynhout commented on ZOOKEEPER-634: --- Epilogue: I spoke too soon, I guess I had been staring at the data for too long. Some parts of the Stat struct were getting repointed or overwritten by the time the values were shuttled into the Ruby glue. Aside from the variation between 32b and 64b, the variations were repeatable (always the same changes, always the same wrong data), so I thought I saw something that wasn't there. Properly duping the struct before passing to the Ruby thread resolved it. some elements of struct Stat have incorrect values when returned asynchronously --- Key: ZOOKEEPER-634 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-634 Project: Zookeeper Issue Type: Bug Components: c client Affects Versions: 3.2.1 Environment: OSX 10.5, 10.6 (32b and 64b, respectively) Reporter: Andrew Reynhout Priority: Minor I'm working on Ruby bindings for ZK (3.2.2) via the C API. This involves some painful kludgery due to MRI Ruby's threading model, but I believe this bug report is valid: Some of the elements of Stat structs, when returned via the C client, have values inconsistent with the synchronous call, zkCli output, and expectations. E.g.: {noformat:title=zkCli command line} $ zk/bin/zkCli.sh -server localhost:2182 set /lala lala234 Connecting to localhost:2182 WATCHER:: WatchedEvent: Server state change. New state: SyncConnected cZxid = 3 ctime = Sat Dec 19 14:32:03 EST 2009 mZxid = 69 mtime = Sat Dec 19 15:31:46 EST 2009 pZxid = 3 cversion = 0 dataVersion = 4 aclVersion = 0 ephemeralOwner = 0 dataLength = 7 numChildren = 0 {noformat} {noformat:title=Ruby script + verbose ZK log output} 2009-12-19 15:31:48,952:45753(0x751000):zoo_de...@process_completions@1902: Switching on cptr-completion_type (async) 2009-12-19 15:31:48,952:45753(0x751000):zoo_de...@process_completions@1916: Calling COMPLETION_STAT for xid=4b2d3837 rc=0 2009-12-19 15:31:48,952:45753(0x751000):zoo_de...@process_completions@1949: --- PRE ruby callback. --- res.stat0x00750e30 .czxid3 .mzxid69 .ctime1261251123149 .mtime1261254706180 .version 4 .cversion 0 .aversion 0 .ephemeralOwner 0 .dataLength 7 .numChildren 0 .pzxid3 === 2009-12-19 15:31:48,952:45753(0x751000):zoo_de...@ruby_stat_completion_wrapper@1702: --- creating ruby thread. 2009-12-19 15:31:48,952:45753(0x7ee000):zoo_de...@ruby_stat_completion_wrapper_2@1685: --- invoking ruby callback. --- cb-dc 0x002fe000 cb-rc 0 cb-ctx0x01a03870 cb-stat 0x00750e30 -czxid3 -mzxid69 -ctime1261251123149 -mtime140735387442616 -version 0 -cversion 0 -aversion 27264392 -ephemeralOwner 4302638816 -dataLength -2100901903 -numChildren 32767 -pzxid3 === 2009-12-19 15:31:48,952:45753(0x7ee000):zoo_de...@ruby_stat_completion_wrapper_2@1687: --- callback returned, freeing struct. 2009-12-19 15:31:48,952:45753(0x7ee000):zoo_de...@ruby_stat_completion_wrapper_2@1689: --- struct freed. sync returned #ZooKeeperFFI::Stat:0x01013783d0 async callback.return_code 0 attr ?? sync async -- - czxid OK 3 3 mzxid OK 69 69 ctime OK 1261251123149 1261251123149 mtime -- 1261254706180140735387442616 version -- 4 0 cversion OK 0 0 aversion -- 0 27264392 ephemeralOwner -- 0 4302638816 dataLength -- 7-2100901903 numChildren -- 0 32767 pzxid OK 3 3 2009-12-19 15:31:49,052:45753(0x6ce000):zoo_de...@do_io@316: IO thread terminated 2009-12-19 15:31:49,052:45753(0x751000):zoo_de...@do_completion@335: completion thread terminated 2009-12-19 15:31:49,052:45753(0x705f3be0):zoo_i...@zookeeper_close@2407: Closing zookeeper session