ZooKeeper Service Latency Overview

[Patrick Hunt]

Some users have asked what performance one should expect from a ZK 
cluster based on number of cores, client load, disk performance, etc... 
I put together the following survey that looks at performance on 
hardware with varying core counts using a shared 7200RPM disk. Many of 
our users don't wish to use a dedicated logging device, so I've run all 
the tests with the ZK datadir sharing space with the OS (no dedicated 
transactional log device). I hope you find this useful: http://bit.ly/4ekN8G


Regards,

Patrick

Re: Bugfix release 3.2.2

[Patrick Hunt]

+1

Henry Robinson wrote:

Will the release include all JIRAs up to 562, or a cherrypick of bugfixes?

It would be great to get zkpython fixes in:

http://issues.apache.org/jira/browse/ZOOKEEPER-538
 
http://issues.apache.org/jira/browse/ZOOKEEPER-554

are both genuine bug fixes.

http://issues.apache.org/jira/browse/ZOOKEEPER-510
http://issues.apache.org/jira/browse/ZOOKEEPER-540
 
http://issues.apache.org/jira/browse/ZOOKEEPER-541

are parts of that general patch effort and there are probably enough
dependencies for it to make sense to include all 5.

cheers,
Henry

On Fri, Oct 30, 2009 at 10:44 AM, Mahadev Konar wrote:


Hi all,
 We are planning to make a bugfix release 3.2.2 which will include a
critical bugfix in the c client code. The jira is ZOOKEEPER-562,
http://issues.apache.org/jira/browse/ZOOKEEPER-562.

 If you would like some fix to be considered for this bugfix release please
feel free to post on the zookeeper-dev list.


Thanks
Mahadev

Re: Possible race in LETest.java

[Patrick Hunt]

Closing the loop - what's the status on this? Can one of you open a 
JIRA and provide a patch for this?


Thanks,

Patrick

Flavio Junqueira wrote:
Hi Henry, Apologies for the the delay. Your observation sounds right to 
me. Here is how I'm reading it; let me know if it makes sense.


If everyone votes for 3 in the second round and 3 has crashed, then in 
countVotes we will remove all votes to 3 and there will be no vote left. 
In such a case, there will be no winner as a result of the call to 
countVotes and lookForLeader won't change the current vote 
(LeaderElection.java:201). This is a situation in which we are stuck.


Does it sound reasonable to add an "else" to the "if" statement of 
LeaderElection.java:201 to reset the vote? This modification would 
implementing resetting the vote when countVotes returns no winner, which 
should happen only when the replica itself votes for a dead leader.


-Flavio

On Oct 28, 2009, at 7:44 AM, Henry Robinson wrote:

[ Sending this direct since the Apache mailserver is rejecting my 
e-mails at the moment ]


As I understand it, 1 and 2 receive a vote for 3 in the first round, 
which causes them to vote for 3 in the second round. So in the second 
round, all votes cast are for 3. But 3 has died, so all votes for it 
are discounted. 1 and 2 continue to vote for 3 ad infinitum, never 
resetting their vote.


Does this sound plausible, or am I missing something?

cheers,
Henry

On Tue, Oct 27, 2009 at 3:48 PM, Flavio Junqueira  
wrote:
Hi Henry, I don't understand how 1 and 2 do not end up electing 2 in 
your situation. If they exclude 3 in countVotes, then countVotes will 
end up returning 2 and not 3, assuming there is a vote for 2. What am 
I missing?


The problem with QuorumPeer you're pointing at was also an issue with 
the FLE tests, and I couldn't see an easy way around it other than 
timing out and restarting leader election.


Cheers,
-Flavio


On Oct 27, 2009, at 6:35 AM, Henry Robinson wrote:

I've been working on adding a TCPResponderThread to the leader election
process so that if a deployment needs to be TCP only, it can be and still
use all election types. Testing this has exposed what might be a race
condition in the leader election code that prevents a leader from being
elected.

Here's the behaviour I see in LETest occasionally. With three nodes 
(reduced
from 30 for ease of debugging), node 3 gets elected before either node 
1 or
node 2 finish their election (there is one round where each node that 
3 has
the highest id, and then 3 completes its second round by receiving 
votes for

itself from 1 and 2, but 1 and 2 do not receive votes from 3).

Now 3 is killed by the test harness. 1 and 2 are still voting for it, but
every time they try, the vote tally excludes 3 since it hasn't been heard
from. They then spin round the voting process, unable to reset their 
vote. I

expect that the heartbeat mechanism in a running QuorumPeer takes care of
this when the leader is lost, but the associated QuorumPeers aren't 
running.


If this is the case, then there is a simple fix to reset the nodes 
vote to

themselves if they are voting for a node that hasn't been heard from. I
don't know why using TCP instead of UDP for the responder thread is
exacerbating this (and we can't rule out my introducing a bug :)); but as
it's a race condition the different timings associated with waiting on 
a TCP

socket might just be enough to expose the issue.

Can someone verify this might be possible / figure out what I missed?

cheers,
Henry

Re: Possible race in LETest.java

[Patrick Hunt]

Ok, great. Good to see you guys are on top of this one!

Patrick

Henry Robinson wrote:

The patch is simple, but a test is harder because it's a race condition. I'm
working on it.

Henry

On Wed, Nov 11, 2009 at 12:06 AM, Flavio Junqueira wrote:


Henry already opened one: ZOOKEEPER-569.

-Flavio


On Nov 11, 2009, at 7:03 AM, Patrick Hunt wrote:

 Closing the loop - what's the status on this? Can one of you open a

JIRA and provide a patch for this?

Thanks,

Patrick

Flavio Junqueira wrote:


Hi Henry, Apologies for the the delay. Your observation sounds right to
me. Here is how I'm reading it; let me know if it makes sense.

If everyone votes for 3 in the second round and 3 has crashed, then in
countVotes we will remove all votes to 3 and there will be no vote left.
In such a case, there will be no winner as a result of the call to
countVotes and lookForLeader won't change the current vote
(LeaderElection.java:201). This is a situation in which we are stuck.

Does it sound reasonable to add an "else" to the "if" statement of
LeaderElection.java:201 to reset the vote? This modification would
implementing resetting the vote when countVotes returns no winner, which
should happen only when the replica itself votes for a dead leader.

-Flavio

On Oct 28, 2009, at 7:44 AM, Henry Robinson wrote:

 [ Sending this direct since the Apache mailserver is rejecting my

e-mails at the moment ]

As I understand it, 1 and 2 receive a vote for 3 in the first round,
which causes them to vote for 3 in the second round. So in the second
round, all votes cast are for 3. But 3 has died, so all votes for it
are discounted. 1 and 2 continue to vote for 3 ad infinitum, never
resetting their vote.

Does this sound plausible, or am I missing something?

cheers,
Henry

On Tue, Oct 27, 2009 at 3:48 PM, Flavio Junqueira 
wrote:
Hi Henry, I don't understand how 1 and 2 do not end up electing 2 in
your situation. If they exclude 3 in countVotes, then countVotes will
end up returning 2 and not 3, assuming there is a vote for 2. What am
I missing?

The problem with QuorumPeer you're pointing at was also an issue with
the FLE tests, and I couldn't see an easy way around it other than
timing out and restarting leader election.

Cheers,
-Flavio


On Oct 27, 2009, at 6:35 AM, Henry Robinson wrote:

I've been working on adding a TCPResponderThread to the leader election
process so that if a deployment needs to be TCP only, it can be and
still
use all election types. Testing this has exposed what might be a race
condition in the leader election code that prevents a leader from being
elected.

Here's the behaviour I see in LETest occasionally. With three nodes
(reduced
from 30 for ease of debugging), node 3 gets elected before either node
1 or
node 2 finish their election (there is one round where each node that
3 has
the highest id, and then 3 completes its second round by receiving
votes for
itself from 1 and 2, but 1 and 2 do not receive votes from 3).

Now 3 is killed by the test harness. 1 and 2 are still voting for it,
but
every time they try, the vote tally excludes 3 since it hasn't been
heard
from. They then spin round the voting process, unable to reset their
vote. I
expect that the heartbeat mechanism in a running QuorumPeer takes care
of
this when the leader is lost, but the associated QuorumPeers aren't
running.

If this is the case, then there is a simple fix to reset the nodes
vote to
themselves if they are voting for a node that hasn't been heard from. I
don't know why using TCP instead of UDP for the responder thread is
exacerbating this (and we can't rule out my introducing a bug :)); but
as
it's a race condition the different timings associated with waiting on
a TCP
socket might just be enough to expose the issue.

Can someone verify this might be possible / figure out what I missed?

cheers,
Henry

[Fwd: Hadoop User Group (Bay Area) - next Wednesday (Nov 18th) at Yahoo!]

[Patrick Hunt]

Tomorrow is the BA HUG, if anyone is interested to talk with Mahadev or 
I f2f regarding ZooKeeper we'll both be in attendance.


Patrick
--- Begin Message ---
Hi all,

We are one week away from the next Bay Area Hadoop User Group  - Yahoo! 
Sunnyvale Campus, next Wednesday (Nov 18th) at 6PM

We have an exciting evening planed:

*Katta, Solr, Lucene and Hadoop - Searching at scale, Jason Rutherglen and 
Jason Venner

*Walking through the New File system API, Sanjay Radia, Yahoo!

*Keep your data in Jute but still use it in python, Paul Tarjan, Yahoo!


Please RSVP here:
http://www.meetup.com/hadoop/calendar/11724002/


Please note that this is the last HUG for 2009, as we will not have a meeting 
on December (due to the holidays).
We will open 2010 with a HUG on Jan 20th.

Looking forward to seeing you next week!

Dekel

--- End Message ---

[ATTN] If you upgraded from v2 of ZooKeeper to v3 please read this.

[Patrick Hunt]

We've found an issue with the migration tool used to migrate users from 
version 2 of ZooKeeper to version 3. This tool was provided to users who 
upgraded from the SourceForge v2 ZK, after we moved to being a 
subproject of Apache Hadoop (which is the same time that we incremented 
the version number from 2.x.y to 3.0.0 - 2+ years ago)


https://issues.apache.org/jira/browse/ZOOKEEPER-582

Please note, this only effects users who migrated their ZK data 
directory from version 2 to version 3. If you have only ever used ZK 
version 3.0.0 and later this does not effect you.


Patrick

Re: Observers!

[Patrick Hunt]

I'm really psyched to see this go into the 3.3 code line, kudos to Henry 
and the rest of the team in getting this landed!


Patrick

ps. gentle reminder - the user list is primarily for answering user 
questions on released software, many users don't follow future 
developments and can be confused as to what's current and what's future. 
(this comes from apache -> me -> you) I'd encourage all of you that are 
interested in things like Observers to follow the dev list, it would be 
great if you all could try out the Observers feature (and the other 
features that we are working on for 3.3.0) in the current trunk and 
provide early feedback via the dev list. Thank you.


Henry Robinson wrote:

Thanks! Also thanks are due to the entire ZK committer team who helped
enormously in getting the patch into shape.

Since the JIRA is now a long and complicated read, I want to summarise a
couple of important points here (although the docs have most of this
information).

1. Observers must currently be used with electionAlg=0; this is due to a
limitation of the other election algorithms which is being removed in
another JIRA.
2. This is only the 'core functionality' patch - you can use Observers in
your ensembles as of this commit (and I would love to hear experiences from
people who do so), but there's more to come in terms of optimisations and
the sanding down of some rough edges. In particular, the configuration is a
bit cumbersome and we already have a JIRA open to address that.

cheers,
Henry


On Wed, Nov 18, 2009 at 3:35 PM, Gustavo Niemeyer wrote:



r881882 | mahadev | 2009-11-18 13:06:39 -0600 (Wed, 18 Nov 2009) | 1 line

ZOOKEEPER-368. Observers: core functionality (henry robinson via mahadev)


Sweet!  Congratulations, and thanks Henry.


--
Gustavo Niemeyer
http://niemeyer.net

Re: Build failed in Hudson: ZooKeeper-trunk #544

[Patrick Hunt]

It looks to me to be the same issue we've been seeing lately on hudson 
trunk - testHammer fails randomly. We've tried a bunch of things to 
reproduce but so far no luck. Even running the test on the hudson 
machine with the same settings (outside hudson) we can't reproduce. We 
haven't seen this failure on the patch queue, which is even weirder.


afaict the session creation is stalling in the request processor, but 
it's not clear where. We may need to create a jira with some 
instrumented test code to see if we can get more data when it fails on 
hudson. Take a look though, maybe you'll see something I did not.


Patrick

Henry Robinson wrote:

The failed test is testObserversHammer - I'm having a look to see what might
be wrong.

Henry

On Thu, Nov 19, 2009 at 2:57 AM, Apache Hudson Server <
hud...@hudson.zones.apache.org> wrote:


See  inline with the variable.

[breed] ZOOKEEPER-519. Followerhandler should close the socket if it gets
an exception on a write.

[breed] ZOOKEEPER-532. java compiler should be target Java 1.5

--
[...truncated 98279 lines...]
   [junit] 2009-11-19 10:57:31,578 - INFO
 [main-SendThread(localhost:11225):clientcnxn$sendthr...@929] - Opening
socket connection to server localhost/127.0.0.1:11225
   [junit] 2009-11-19 10:57:31,578 - INFO
 [main-SendThread(localhost:11225):clientcnxn$sendthr...@837] - Socket
connection established to localhost/127.0.0.1:11225, initiating session
   [junit] 2009-11-19 10:57:31,579 - INFO
 [NIOServerCxn.Factory:11225:nioservercnxn$fact...@214] - Accepted socket
connection from /127.0.0.1:54182
   [junit] 2009-11-19 10:57:31,579 - INFO
 [NIOServerCxn.Factory:11225:nioserverc...@688] - Client attempting to
renew session 0x1250c16f121 at /127.0.0.1:54182
   [junit] 2009-11-19 10:57:31,580 - INFO
 [NIOServerCxn.Factory:11225:nioserverc...@1131] - Established session
0x1250c16f121 for client /127.0.0.1:54182
   [junit] 2009-11-19 10:57:31,580 - INFO
 [main-SendThread(localhost:11225):clientcnxn$sendthr...@640] - Session
establishment complete, sessionid = 0x1250c16f121
   [junit] 2009-11-19 10:57:31,588 - INFO  [main:clientb...@376] -
STOPPING server
   [junit] 2009-11-19 10:57:31,589 - INFO  [main:nioserverc...@974] -
Closed socket connection for client /127.0.0.1:54182 which had sessionid
0x1250c16f121
   [junit] 2009-11-19 10:57:31,589 - INFO
 [main-SendThread(localhost:11225):clientcnxn$sendthr...@1047] - Unable to
read additional data from server sessionid 0x1250c16f121, likely server
has closed socket, closing socket connection and attempting reconnect
   [junit] 2009-11-19 10:57:31,590 - INFO
 [NIOServerCxn.Factory:11225:nioservercnxn$fact...@241] - NIOServerCnxn
factory exited run method
   [junit] 2009-11-19 10:57:31,590 - INFO  [main:finalrequestproces...@365]
- shutdown of request processor complete
   [junit] 2009-11-19 10:57:31,590 - INFO
 [SyncThread:0:syncrequestproces...@151] - SyncRequestProcessor exited!
   [junit] 2009-11-19 10:57:31,590 - INFO
 [ProcessThread:-1:preprequestproces...@119] - PrepRequestProcessor exited
loop!
   [junit] ensureOnly:[]
   [junit] 2009-11-19 10:57:31,690 - INFO  [main:clientb...@369] -
STARTING server
   [junit] 2009-11-19 10:57:31,690 - INFO  [main:zookeeperser...@160] -
Created server
   [junit] 2009-11-19 10:57:31,691 - INFO  [main:nioservercnxn$fact...@123]
- binding to port 11225
   [junit] 2009-11-19 10:57:31,692 - INFO  [main:files...@81] - Reading
snapshot <
http://hudson.zones.apache.org/hudson/job/ZooKeeper-trunk/ws/trunk/build/test/tmp/test4406575549066528204.junit.dir/version-2/snapshot.5
   [junit] 2009-11-19 10:57:31,694 - INFO  [main:filetxnsnap...@208] -
Snapshotting: 6
   [junit] 2009-11-19 10:57:31,696 - INFO
 [NIOServerCxn.Factory:11225:nioservercnxn$fact...@214] - Accepted socket
connection from /127.0.0.1:54184
   [junit] 2009-11-19 10:57:31,696 - INFO
 [NIOServerCxn.Factory:11225:nioserverc...@782] - Processing stat command
from /127.0.0.1:54184
   [junit] 2009-11-19 10:57:31,697 - INFO
 [NIOServerCxn.Factory:11225:nioserverc...@974] - Closed socket connection
for client /127.0.0.1:54184 (no session established for client)
   [junit] ensureOnly:[InMemoryDataTree, StandaloneServer_port]
   [junit] expect:InMemoryDataTree
   [junit] found:InMemoryDataTree
org.apache.ZooKeeperService:name0=StandaloneServer_port-1,name1=InMemoryDataTree
   [junit] expect:StandaloneServer_port
   [junit] found:StandaloneServer_port
org.apache.ZooKeeperService:name0=StandaloneServer_port-1
   [junit] 2009-11-19 10:57:33,000 - INFO
 [SessionTracker:sessiontrackeri...@145] - SessionTrackerImpl exited loop!
   [junit] 2009-11-19 10:57:33,586 - INFO
 [main-SendThread(localhost:11225):clientcnxn$sendth

Fix releases 3.2.2 and 3.1.2 are in progress

[Patrick Hunt]

Due to the issue I reported a few days ago, ZOOKEEPER-582 (email 
attached), we will be creating new 3.2.2. and 3.1.2 releases to address. 
Release candidates should be put up for the vote in the next day or so.


3.2.2
https://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=true&&pid=12310801&fixfor=12314335&sorter/field=issuekey&sorter/order=DESC&sorter/field=priority&sorter/order=DESC

3.1.2
https://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=true&&pid=12310801&fixfor=12314394&sorter/field=issuekey&sorter/order=DESC&sorter/field=priority&sorter/order=DESC

Patrick
--- Begin Message ---
We've found an issue with the migration tool used to migrate users from 
version 2 of ZooKeeper to version 3. This tool was provided to users who 
upgraded from the SourceForge v2 ZK, after we moved to being a 
subproject of Apache Hadoop (which is the same time that we incremented 
the version number from 2.x.y to 3.0.0 - 2+ years ago)


https://issues.apache.org/jira/browse/ZOOKEEPER-582

Please note, this only effects users who migrated their ZK data 
directory from version 2 to version 3. If you have only ever used ZK 
version 3.0.0 and later this does not effect you.


Patrick
--- End Message ---

[Fwd: Re: 1st Hadoop India User Group meet]

[Patrick Hunt]

See attached - Hadoop India User Group meet up on 28th November 2009 in 
Noida, India. Aby Abraham will be presenting on ZooKeeper.


Patrick
--- Begin Message ---
Here is the updated agenda for Hadoop India User Group meet up on 28th November 
2009 in Noida-
- Introductions
- Sessions-
- "Enterprise perspective of Hadoop" - by Rajeev Gupta, IBM Research 
Lab, IBM
- "Apache Zookeeper" - by Aby Abraham, Yahoo! Bangalore
- "Advanced Hadoop Tuning and Optimizations"- by Sanjay Sharma, Impetus

- Establishing agenda for the next few meetings
- Information exchange: tips, tricks, problems and open discussion, sharing 
experiences on Hadoop and related technologies

The registrations are still open at http://hug-india.eventbrite.com/



Regards,
Sanjay Sharma
Impetus
t: +91-120-4363300 Extn 2761
m: +91-9958790163

-Original Message-
From: Sanjay Sharma
Sent: Tuesday, November 10, 2009 11:50 AM
To: 'gene...@hadoop.apache.org'
Subject: 1st Hadoop India User Group meet

We are planning to hold first Hadoop India user group meet up on 28th November 
2009 in Noida.

We would be talking about our experiences with Apache 
Hadoop/Hbase/Hive/PIG/Nutch/etc.

The agenda would be:
- Introductions
- Sharing experiences on Hadoop and related technologies
- Establishing agenda for the next few meetings
- Information exchange: tips, tricks, problems and open discussion
- Possible speaker TBD (invitations open!!)  {we do have something to share on 
"Hadoop for newbie" & "Hadoop Advanced Tuning"}

My company (Impetus) would be providing the meeting room and we should be able 
to accommodate around 40-60 friendly people. Coffee, Tea, and some snacks will 
be provided.

Please join the linked-in Hadoop India User Group 
(http://www.linkedin.com/groups?home=&gid=2258445&trk=anet_ug_hm) OR Yahoo 
group (http://tech.groups.yahoo.com/group/hadoopind/) and confirm your 
attendance.

Regards,
Sanjay Sharma

Follow our updates on www.twitter.com/impetuscalling.

* Impetus Technologies is exhibiting it capabilities in Mobile and Wireless in 
the GSMA Mobile Asia Congress, Hong Kong from November 16-18, 2009. Visit 
http://www.impetus.com/mlabs/GSMA_events.html for details.

NOTE: This message may contain information that is confidential, proprietary, 
privileged or otherwise protected by law. The message is intended solely for 
the named addressee. If received in error, please destroy and notify the 
sender. Any use of this email is prohibited when received in error. Impetus 
does not represent, warrant and/or guarantee, that the integrity of this 
communication has been maintained nor that the communication is free of errors, 
virus, interception or interference.
--- End Message ---

[VOTE] Release ZooKeeper 3.1.2 (candidate 0)

[Patrick Hunt]

I've created a candidate build for ZooKeeper 3.1.2. This is a bug fix 
release addressing just two critical issues -- see the release notes for 
details.


*** Please download, test and VOTE before the
*** vote closes 4pm pacific time, Friday, November 27.***

http://people.apache.org/~phunt/zookeeper-3.1.2-candidate-0/

Should we release this?

Patrick

[VOTE] Release ZooKeeper 3.2.2 (candidate 0)

[Patrick Hunt]

I've created a candidate build for ZooKeeper 3.2.2. This is a bug fix 
release addressing eleven issues (two critical) -- see the release notes 
for details.


*** Please download, test and VOTE before the
*** vote closes 5pm pacific time, Friday, November 27.***

http://people.apache.org/~phunt/zookeeper-3.2.2-candidate-0/

Should we release this?

Patrick

Re: Build failed in Hudson: ZooKeeper-trunk #576

[Patrick Hunt]

Yea, that's the one. It's not us. I love how it's "low priority" and 
only fixed in java 7. We have a JIRA on this (closed) but I can't seem 
to find it...


Patrick

Flavio Junqueira wrote:

Is this the problem?

http://bugs.sun.com/view_bug.do?bug_id=6427854

-Flavio

On Dec 4, 2009, at 5:49 PM, Benjamin Reed wrote:


Check it out, the default uncaught exception handler helped! look at
this problem:

2009-12-04 09:30:47,333 - ERROR [QuorumPeer with myid:1 and
clientPort:11221:quorumpeertestbase$mainthr...@92] - unexpected
exception in run
java.lang.NullPointerException
at sun.nio.ch.Util.atBugLevel(Util.java:326)
at sun.nio.ch.SelectorImpl.(SelectorImpl.java:40)
at sun.nio.ch.EPollSelectorImpl.(EPollSelectorImpl.java:47)
at
sun
.nio
.ch.EPollSelectorProvider.openSelector(EPollSelectorProvider.java:18)
at java.nio.channels.Selector.open(Selector.java:209)
at
org.apache.zookeeper.server.NIOServerCnxn$Factory.(NIOServerCnxn.java:87) 


at
org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:120) 


at
org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:102) 


at
org.apache.zookeeper.server.quorum.QuorumPeerTestBase$MainThread.run(QuorumPeerTestBase.java:89) 



looking at the source code (i don't have exactly the right version
because the line numbers don't match) there is no way that we could get
a NullPointerException in Util.atBugLevel. For that matter, we are just
calling Selector.open(), it doesn't seem like we should be getting an
exception at all!

ben

Apache Hudson Server wrote:

See 

--
[...truncated 94911 lines...]
   [junit] 2009-12-04 09:42:20,334 - INFO  
[main-SendThread(localhost:11225):clientcnxn$sendthr...@837] - Socket 
connection established to localhost/127.0.0.1:11225, initiating session
   [junit] 2009-12-04 09:42:20,334 - INFO  
[NIOServerCxn.Factory:11225:nioservercnxn$fact...@221] - Accepted 
socket connection from /127.0.0.1:46504
   [junit] 2009-12-04 09:42:20,334 - INFO  
[NIOServerCxn.Factory:11225:nioserverc...@695] - Client attempting to 
renew session 0x12559118220 at /127.0.0.1:46504
   [junit] 2009-12-04 09:42:20,335 - INFO  
[NIOServerCxn.Factory:11225:nioserverc...@1128] - Established session 
0x12559118220 for client /127.0.0.1:46504
   [junit] 2009-12-04 09:42:20,335 - INFO  
[main-SendThread(localhost:11225):clientcnxn$sendthr...@640] - 
Session establishment complete, sessionid = 0x12559118220
   [junit] 2009-12-04 09:42:20,347 - INFO  [main:clientb...@385] - 
STOPPING server
   [junit] 2009-12-04 09:42:20,348 - INFO  [main:nioserverc...@975] - 
Closed socket connection for client /127.0.0.1:46504 which had 
sessionid 0x12559118220
   [junit] 2009-12-04 09:42:20,348 - INFO  
[main-SendThread(localhost:11225):clientcnxn$sendthr...@1047] - 
Unable to read additional data from server sessionid 
0x12559118220, likely server has closed socket, closing socket 
connection and attempting reconnect
   [junit] 2009-12-04 09:42:20,349 - INFO  
[NIOServerCxn.Factory:11225:nioservercnxn$fact...@248] - 
NIOServerCnxn factory exited run method
   [junit] 2009-12-04 09:42:20,349 - INFO  
[main:finalrequestproces...@365] - shutdown of request processor 
complete
   [junit] 2009-12-04 09:42:20,349 - INFO  
[SyncThread:0:syncrequestproces...@151] - SyncRequestProcessor exited!
   [junit] 2009-12-04 09:42:20,349 - INFO  
[ProcessThread:-1:preprequestproces...@119] - PrepRequestProcessor 
exited loop!

   [junit] ensureOnly:[]
   [junit] 2009-12-04 09:42:20,449 - INFO  [main:clientb...@378] - 
STARTING server
   [junit] 2009-12-04 09:42:20,449 - INFO  [main:zookeeperser...@160] 
- Created server
   [junit] 2009-12-04 09:42:20,450 - INFO  
[main:nioservercnxn$fact...@130] - binding to port 11225
   [junit] 2009-12-04 09:42:20,456 - INFO  [main:files...@81] - 
Reading snapshot 
 

   [junit] 2009-12-04 09:42:20,458 - INFO  [main:filetxnsnap...@208] 
- Snapshotting: 6
   [junit] 2009-12-04 09:42:20,462 - INFO  
[NIOServerCxn.Factory:11225:nioservercnxn$fact...@221] - Accepted 
socket connection from /127.0.0.1:46506
   [junit] 2009-12-04 09:42:20,462 - INFO  
[NIOServerCxn.Factory:11225:nioserverc...@789] - Processing stat 
command from /127.0.0.1:46506
   [junit] 2009-12-04 09:42:20,462 - INFO  
[NIOServerCxn.Factory:11225:nioserverc...@975] - Closed socket 
connection for client /127.0.0.1:46506 (no session established for 
client)

   [junit] ensureOnly:[InMemoryDataTree, StandaloneServer_port]
   [junit] expect:InMemoryDataTree
   [junit] found:InMemoryDataTree 
org.apache.ZooKeeperService:name0=StandaloneServer_port-1,name1=InMemoryDataTree 


   [junit] expect:StandaloneServer_port
   [junit] found:StandaloneServer_port 
org.apache.ZooKeeperService:name0=Standa

this one is more serious: Build failed in Hudson: ZooKeeper-trunk #577

[Patrick Hunt]

This is a similar issue as to what's been fixed so far in ZOOKEEPER-597
http://hudson.zones.apache.org/hudson/view/ZooKeeper/job/ZooKeeper-trunk/577/testReport/org.apache.zookeeper.test/AsyncHammerTest/testObserversHammer/
but seems to be popping up in another area

It is good to have the handler here!

2009-12-04 13:46:13,493 - ERROR [CommitProcessor:0:commitproces...@146] 
- Unexpected exception causing CommitProcessor to exit

java.nio.channels.CancelledKeyException
at sun.nio.ch.SelectionKeyImpl.ensureValid(SelectionKeyImpl.java:55)
at sun.nio.ch.SelectionKeyImpl.interestOps(SelectionKeyImpl.java:64)
	at 
org.apache.zookeeper.server.NIOServerCnxn.sendBuffer(NIOServerCnxn.java:378)
	at 
org.apache.zookeeper.server.NIOServerCnxn.sendCloseSession(NIOServerCnxn.java:349)
	at 
org.apache.zookeeper.server.FinalRequestProcessor.processRequest(FinalRequestProcessor.java:356)
	at 
org.apache.zookeeper.server.quorum.Leader$ToBeAppliedRequestProcessor.processRequest(Leader.java:532)
	at 
org.apache.zookeeper.server.quorum.CommitProcessor.run(CommitProcessor.java:73)

Welcome Henry as ZooKeeper committer

[Patrick Hunt]

The ZooKeeper team and the Hadoop PMC would like to welcome Henry
Robinson as the newest committer on ZooKeeper! Henry, please add
yourself to the ZooKeeper website's list of committers.

Thanks for your hard work on behalf of ZooKeeper, Hadoop and the ASF,

Patrick

[VOTE] Release ZooKeeper 3.1.2 (candidate 1)

[Patrick Hunt]

I've created a second candidate build for ZooKeeper 3.1.2. This is a bug 
fix release addressing three critical issues -- see the release notes 
for details.


*** Please download, test and VOTE before the
*** vote closes 3pm pacific time, Friday, December 11.***

http://people.apache.org/~phunt/zookeeper-3.1.2-candidate-1/

Should we release this?

Patrick

[VOTE] Release ZooKeeper 3.2.2 (candidate 1)

[Patrick Hunt]

I've created a second candidate build for ZooKeeper 3.2.2. This is a bug 
fix release addressing 13 issues (three critical) -- see the release 
notes for details.


*** Please download, test and VOTE before the
*** vote closes 3pm pacific time, Friday, December 11.***

http://people.apache.org/~phunt/zookeeper-3.2.2-candidate-1/

Should we release this?

Patrick

Re: [VOTE] Release ZooKeeper 3.2.2 (candidate 1)

[Patrick Hunt]

+1 looks good to me - ran tests including testing on 3 node quorum 
(includes zkpython tests exercising the c client)


Patrick

On 12/08/2009 02:32 PM, Patrick Hunt wrote:

I've created a second candidate build for ZooKeeper 3.2.2. This is a bug
fix release addressing 13 issues (three critical) -- see the release
notes for details.

*** Please download, test and VOTE before the
*** vote closes 3pm pacific time, Friday, December 11.***

http://people.apache.org/~phunt/zookeeper-3.2.2-candidate-1/

Should we release this?

Patrick

Re: [VOTE] Release ZooKeeper 3.1.2 (candidate 1)

[Patrick Hunt]

+1 looks good to me - ran tests including testing on 3 node quorum

Patrick

On 12/08/2009 02:30 PM, Patrick Hunt wrote:

I've created a second candidate build for ZooKeeper 3.1.2. This is a bug
fix release addressing three critical issues -- see the release notes
for details.

*** Please download, test and VOTE before the
*** vote closes 3pm pacific time, Friday, December 11.***

http://people.apache.org/~phunt/zookeeper-3.1.2-candidate-1/

Should we release this?

Patrick

Re: [VOTE] Release ZooKeeper 3.1.2 (candidate 1)

[Patrick Hunt]

With 6 +1's (4 PMC) and no -1's this vote passes. I will push out this 
release asap.


Patrick

Patrick Hunt wrote:
I've created a second candidate build for ZooKeeper 3.1.2. This is a bug 
fix release addressing three critical issues -- see the release notes 
for details.


*** Please download, test and VOTE before the
*** vote closes 3pm pacific time, Friday, December 11.***

http://people.apache.org/~phunt/zookeeper-3.1.2-candidate-1/

Should we release this?

Patrick

Re: [VOTE] Release ZooKeeper 3.2.2 (candidate 1)

[Patrick Hunt]

With 6 +1's (3 PMC) and no -1's this vote passes. I will push out this 
release asap.


Patrick

Patrick Hunt wrote:
I've created a second candidate build for ZooKeeper 3.2.2. This is a bug 
fix release addressing 13 issues (three critical) -- see the release 
notes for details.


*** Please download, test and VOTE before the
*** vote closes 3pm pacific time, Friday, December 11.***

http://people.apache.org/~phunt/zookeeper-3.2.2-candidate-1/

Should we release this?

Patrick

[ANNOUNCE] Apache ZooKeeper 3.1.2

[Patrick Hunt]

The Apache ZooKeeper team is proud to announce Apache ZooKeeper version 
3.1.2.


ZooKeeper is a high-performance coordination service for distributed 
applications. It exposes common services - such as naming, configuration 
management, synchronization, and group services - in a simple interface 
so you don't have to write them from scratch. You can use it 
off-the-shelf to implement consensus, group management, leader election, 
and presence protocols. And you can build on it for your own, specific 
needs.


If you are upgrading from version 2.2.1 on SourceForge be sure to review 
the 3.0.1 release notes for migration instructions.


For ZooKeeper release details and downloads, visit:
http://hadoop.apache.org/zookeeper/releases.html

ZooKeeper 3.1.2 Release Notes are at:
http://hadoop.apache.org/zookeeper/docs/r3.1.2/releasenotes.html

Regards,

The ZooKeeper Team

[ANNOUNCE] Apache ZooKeeper 3.2.2

[Patrick Hunt]

The Apache ZooKeeper team is proud to announce Apache ZooKeeper version 
3.2.2.


ZooKeeper is a high-performance coordination service for distributed 
applications. It exposes common services - such as naming, configuration 
management, synchronization, and group services - in a simple interface 
so you don't have to write them from scratch. You can use it 
off-the-shelf to implement consensus, group management, leader election, 
and presence protocols. And you can build on it for your own, specific 
needs.


If you are upgrading from version 2.2.1 on SourceForge be sure to review 
the 3.0.1 release notes for migration instructions.


For ZooKeeper release details and downloads, visit:
http://hadoop.apache.org/zookeeper/releases.html

ZooKeeper 3.2.2 Release Notes are at:
http://hadoop.apache.org/zookeeper/docs/r3.2.2/releasenotes.html

Regards,

The ZooKeeper Team

Re: The C Client cause core dump in some situation

[Patrick Hunt]

Did you try using valgrind? That might help reproduce.

Qian Ye wrote:

Hi Mahadev:

I have created a jira for this issue
https://issues.apache.org/jira/browse/ZOOKEEPER-624.
And so far, I haven't found the way to reproduce the segment fault. I tried
about 10 times the same operations and only produced the core dump 1 time.
I would attach the way to the jira if I can find.

Thx


On Tue, Dec 15, 2009 at 1:53 AM, Mahadev Konar wrote:


Hi Qian,
 The code that you mention still exists in the trunk and does not check for
the len before calling memcpy. Please open a jira on this.

The interesting thing though is that the len is -1. Do you have any test
case or  a test scenario where it can be reproduced. It would be
interesting
to see why this is happening. We should not be getting a -1 len value from
the server.


Thanks
mahadev


On 12/14/09 6:19 AM, "Qian Ye"  wrote:


Hi guys:

I encountered a problem today that the Zookeeper C Client (version 3.2.0)
core dump when reconnected and did some operations on the zookeeper

server

which just restarted. The gdb infomation is like:

(gdb) bt
#0  0x00302af71900 in memcpy () from /lib64/tls/libc.so.6
#1  0x0047bfe4 in ia_deserialize_string (ia=Variable "ia" is not
available.) at src/recordio.c:270
#2  0x0047ed20 in deserialize_CreateResponse (in=0x9cd870,
tag=0x50a74e "reply", v=0x409ffe70) at generated/zookeeper.jute.c:679
#3  0x0047a1d0 in zookeeper_process (zh=0x9c8c70, events=Variable
"events" is not available.) at src/zookeeper.c:1895
#4  0x004815e6 in do_io (v=Variable "v" is not available.) at
src/mt_adaptor.c:310
#5  0x00302b80610a in start_thread () from /lib64/tls/libpthread.so.0
#6  0x00302afc6003 in clone () from /lib64/tls/libc.so.6
#7  0x in ?? ()
(gdb) f 1
#1  0x0047bfe4 in ia_deserialize_string (ia=Variable "ia" is not
available.) at src/recordio.c:270
270 in src/recordio.c
(gdb) info locals
priv = (struct buff_struct *) 0x9cd8d0
*len = -1*
rc = Variable "rc" is not available.

According to the source code,
int ia_deserialize_string(struct iarchive *ia, const char *name, char

**s)

{
struct buff_struct *priv = ia->priv;
int32_t len;
*int rc = ia_deserialize_int(ia, "len", &len);*
if (rc < 0)
return rc;
if ((priv->len - priv->off) < len) {
return -E2BIG;
}
*s = malloc(len+1);
if (!*s) {
return -ENOMEM;
}
memcpy(*s, priv->buffer+priv->off, len);
(*s)[len] = '\0';
priv->off += len;
return 0;
}

the variable len is set by ia_deserialize_int, and the returned len

doesn't

been checked, so the client segment fault when trying to memcpy -1 byte
data.
I'm not sure why the client got the len variable -1 when deserialize the
response from the server, I'm also not sure whether it is an known issue.
Could any
one give me some information about this problem?

3.3.0 release planning

[Patrick Hunt]

Now that the fix releases are out we should look at nailing down a date 
for 3.3.0, once we define a date we can triage the JIRA list and pare it 
down to what we think we can/should accomplish in the timeframe we have.


December is mostly done given many ppl are out for holidays. I'm 
thinking March 10th to create a release candidate and put it up for the 
vote. This will give ppl 8weeks or so after the holidays to complete any 
blockers before I cut the rel candiate. This sound good?


Patrick

Re: A very strange scenario, may due to some bug on the server side

[Patrick Hunt]

I created a jira for the cli - should output xids/sessionids/etc in hex

https://issues.apache.org/jira/browse/ZOOKEEPER-626

Patrick

Mahadev Konar wrote:

Hi Qian,
  This is quite weird. Are you sure the version is 3.2.1?
   If yes, please create a jira for this.

  Also, can you extract the server logs for the session



ephemeralOwner: 226627854640480810


And post it on a jira? Ephemeral Owner is the session id. You can convert
the above number to hex and look through the logs to see what happened to
this session and post the logs on the jira. Looks like the session close for
the session (226627854640480810) wasn't successful (a bug mostly). So we
need to trace back on what happened on a close of this session and why it
did not close.

Grepping all the server logs for session id (0x32524d5440e022a, this is the
hex of the the above decimal number) might give us some insight into this.


Thanks
mahadev

On 12/15/09 7:44 AM, "Benjamin Reed"  wrote:


does  se/diserver_tc/diserver_tc67 appear on all three servers?

ben

Qian Ye wrote:

Hi guys:

I find a very strange scenario today, I'm not sure how it happen, I just
found it like this. Maybe you can give me some information about it, my
Zookeeper Server is version 3.2.1.

My Zookeeper cluster contains three servers, with ip:
10.81.12.144,10.81.12.145,10.81.12.141. I wrote a client to create ephemeral
node under znode: *se/diserver_tc*. The client runs on the server with ip
10.81.13.173. The client can create a ephemeral node on zookeeper server and
write the host ip (10.81.13.173) in to the node as its data. There is only
one client process can be running at a time, because the client will listen
to a certain port.

It is strange that I found there were two ephemeral node with the ip
10.81.13.173 under znode se/diserver_tc.
*se/diserver_tc/diserver_tc67*
STAT:
czxid: 124554079820
mzxid: 124554079820
ctime: 1260609598547
mtime: 1260609598547
version: 0
cversion: 0
aversion: 0
ephemeralOwner: 226627854640480810
dataLength: 92
numChildren: 0
pzxid: 124554079820

*se/diserver_tc/diserver_tc95
*STAT:
czxid: 128849019107
mzxid: 128849019107
ctime: 1260772197356
mtime: 1260772197356
version: 0
cversion: 0
aversion: 0
ephemeralOwner: 154673159808876591
dataLength: 92
numChildren: 0
pzxid: 128849019107*
*
There are TWO with different session id! And after I kill the client process
on the server 10.81.13.173, the *se/diserver_tc/diserver_tc95 *node
disappear, but the *se/diserver_tc/diserver_tc67 *stay the same.
That means it is not my coding mistake to create the node twice. I checked
several times and I'm sure that there is no another client instance running.
And I use the 'stat' command to check the three zookeeper servers, and there
is no client from 10.81.13.173,

$echo stat | nc 10.81.12.144 2181
Zookeeper version: 3.2.1-808558, built on 08/27/2009 18:48 GMT
Clients:
 /10.81.13.173:35676[1](queued=0,recved=0,sent=0) *# it is caused by the nc
process*

Latency min/avg/max: 0/3/254
Received: 11081
Sent: 0
Outstanding: 0
Zxid: 0x1e01f5
Mode: follower
*Node count: 32
*
$ echo stat | nc 10.81.12.141 2181
Zookeeper version: 3.2.1-808558, built on 08/27/2009 18:48 GMT
Clients:
 /10.81.12.152:58110[1](queued=0,recved=10374,sent=0)
 /10.81.13.173:35677[1](queued=0,recved=0,sent=0) *# it is caused by the nc
process*

Latency min/avg/max: 0/0/37
Received: 37128
Sent: 0
Outstanding: 0
Zxid: 0x1e01f5
Mode: follower
*Node count: 26*

$ echo stat | nc 10.81.12.145 2181
Zookeeper version: 3.2.1-808558, built on 08/27/2009 18:48 GMT
Clients:
 /10.81.12.153:19130[1](queued=0,recved=10624,sent=0)
 /10.81.13.173:35678[1](queued=0,recved=0,sent=0) *# it is caused by the nc
process*

Latency min/avg/max: 0/2/213
Received: 26700
Sent: 0
Outstanding: 0
Zxid: 0x1e01f5
Mode: leader
*Node count: 26*

The three 'stat' commands show different Node count! Just cannot understand
how it happened, can anyone give me some explanation about it?


  

Re: A very strange scenario, may due to some bug on the server side

[Patrick Hunt]

You might also try the "dump" command for all 3 servers (similar to the 
stat command - it's a 4letterword) and look at it's output -- it 
includes information on ephemeral nodes.


Patrick

Qian Ye wrote:

Hi guys:

I find a very strange scenario today, I'm not sure how it happen, I just
found it like this. Maybe you can give me some information about it, my
Zookeeper Server is version 3.2.1.

My Zookeeper cluster contains three servers, with ip:
10.81.12.144,10.81.12.145,10.81.12.141. I wrote a client to create ephemeral
node under znode: *se/diserver_tc*. The client runs on the server with ip
10.81.13.173. The client can create a ephemeral node on zookeeper server and
write the host ip (10.81.13.173) in to the node as its data. There is only
one client process can be running at a time, because the client will listen
to a certain port.

It is strange that I found there were two ephemeral node with the ip
10.81.13.173 under znode se/diserver_tc.
*se/diserver_tc/diserver_tc67*
STAT:
czxid: 124554079820
mzxid: 124554079820
ctime: 1260609598547
mtime: 1260609598547
version: 0
cversion: 0
aversion: 0
ephemeralOwner: 226627854640480810
dataLength: 92
numChildren: 0
pzxid: 124554079820

*se/diserver_tc/diserver_tc95
*STAT:
czxid: 128849019107
mzxid: 128849019107
ctime: 1260772197356
mtime: 1260772197356
version: 0
cversion: 0
aversion: 0
ephemeralOwner: 154673159808876591
dataLength: 92
numChildren: 0
pzxid: 128849019107*
*
There are TWO with different session id! And after I kill the client process
on the server 10.81.13.173, the *se/diserver_tc/diserver_tc95 *node
disappear, but the *se/diserver_tc/diserver_tc67 *stay the same.
That means it is not my coding mistake to create the node twice. I checked
several times and I'm sure that there is no another client instance running.
And I use the 'stat' command to check the three zookeeper servers, and there
is no client from 10.81.13.173,

$echo stat | nc 10.81.12.144 2181
Zookeeper version: 3.2.1-808558, built on 08/27/2009 18:48 GMT
Clients:
 /10.81.13.173:35676[1](queued=0,recved=0,sent=0) *# it is caused by the nc
process*

Latency min/avg/max: 0/3/254
Received: 11081
Sent: 0
Outstanding: 0
Zxid: 0x1e01f5
Mode: follower
*Node count: 32
*
$ echo stat | nc 10.81.12.141 2181
Zookeeper version: 3.2.1-808558, built on 08/27/2009 18:48 GMT
Clients:
 /10.81.12.152:58110[1](queued=0,recved=10374,sent=0)
 /10.81.13.173:35677[1](queued=0,recved=0,sent=0) *# it is caused by the nc
process*

Latency min/avg/max: 0/0/37
Received: 37128
Sent: 0
Outstanding: 0
Zxid: 0x1e01f5
Mode: follower
*Node count: 26*

$ echo stat | nc 10.81.12.145 2181
Zookeeper version: 3.2.1-808558, built on 08/27/2009 18:48 GMT
Clients:
 /10.81.12.153:19130[1](queued=0,recved=10624,sent=0)
 /10.81.13.173:35678[1](queued=0,recved=0,sent=0) *# it is caused by the nc
process*

Latency min/avg/max: 0/2/213
Received: 26700
Sent: 0
Outstanding: 0
Zxid: 0x1e01f5
Mode: leader
*Node count: 26*

The three 'stat' commands show different Node count! Just cannot understand
how it happened, can anyone give me some explanation about it?

Re: 3.3.0 release planning

[Patrick Hunt]

Sounds good. Any more input sing out, for the time being I'll go with 
3/10/2010 as the planned release date.


I'm going to start triaging issues listed as "fix for 3.3.0", pushing 
some things that are not critical/blocker/assigned into 3.4.0 or 
"unscheduled". This doesn't mean people can't work on these issues, just 
that they are not planned for the release - it's really for planning 
purposes while the JIRA is open. If people come up with a patch for 
something feel free to submit it for 3.3.0 regardless of the current fix 
date listed in the JIRA. If you feel something should/must be in 3.3.0 
feel free to comment on jira or message the list - obv backing that up 
with a patch increases the chances. ;-)


If you have JIRAs assigned to you that are open and you don't plan to 
work on the JIRA it would probably be a good idea to update it to 
"unassigned" instead. Again, helps in planning and understanding the 
workload.


Once I do the initial triage I'll message the list again with some 
release status type information, esp as we get closer to the release date.


If you want to work on a JIRA that's "risky" it's obv better to work on 
it now than wait until the release date comes near.


Regards,

Patrick

Mahadev Konar wrote:

+1

mahadev


On 12/16/09 8:54 AM, "Benjamin Reed"  wrote:


March 10th sounds good to me.

ben

Patrick Hunt wrote:

Now that the fix releases are out we should look at nailing down a date
for 3.3.0, once we define a date we can triage the JIRA list and pare it
down to what we think we can/should accomplish in the timeframe we have.

December is mostly done given many ppl are out for holidays. I'm
thinking March 10th to create a release candidate and put it up for the
vote. This will give ppl 8weeks or so after the holidays to complete any
blockers before I cut the rel candiate. This sound good?

Patrick
  

Re: blog post about zk and group membership

[Patrick Hunt]

Adam, this is awesome! I've retweeted it on my twitter account:
http://twitter.com/phunt/status/7637054275

A few comments:

1) you are correct about the "off the shelf" part, but really alot of 
that is our marketing blurb, we paint with a broad stroke. I think it 
would make sense to include this sort of detail in the "recipes" page 
though, would you be interested to create a JIRA and submit some 
content? (you could re purpose alot of what you have in your post)


2) re zkclient - I have not been following that project closely of late, 
however Mahadev and I did review their code a few months ago. At that 
time we had some concerns about the semantic impact of some of the 
design choices they made (which were fine, just not well documented and 
therefor easy for users to miss). In particular (again, at the time, may 
be addressed by now) watch notifications could be lost during 
disconnected/expired sessions, and the user would not be made aware of this.


3) excellent point/suggestion on the "rogue" service

Patrick

Adam Rosien wrote:

I've written a blog post about using ZooKeeper for group membership at KaChing:

http://bit.ly/6yraP6
http://eng.kaching.com/2010/01/actually-implementing-group-management.html

Any comments or feedback is welcome.

.. Adam

Re: blog post about zk and group membership

[Patrick Hunt]

Adam Rosien wrote:

Re "off the shelf", my title of "Actually Implementing" was a *bit*
snarky to try and gain some readers, but it wasn't intended to be
sharp criticism; more documented recipes is a good thing for everyone.
I'd be happy to submit a JIRA with a group membership recipe.



Ya, I got that, no worries. It was a valid point and something I've been 
asked before. Figured I could get some (addl) free work out of the deal. ;-)


Disqus seems to be having troubles, I've tried like 3 times to comment 
on your blog post, did it finally make it through?



Re zkclient, I agree that documentation is a big thing, which I hope
to help out with as I use it more. I'm talking now with one of the
developers about zkclient's implementation of data serialization since
that is another big area; zk pushes all serialization semantics into
the clients, which is ok but requires its own practical strategies.


Yea, that's an interesting point. I suspect that a wrapper around the 
data (zk contrib? subclass/proxy of ZooKeeper?) that put content-type 
type information into the "header" of the data portion might be useful. 
The wrapper could encode/decode based on the data type in ZK, vs the 
data type the user would like. The wrapper could support whichever 
encode/decode the user would like (say xml/json to start) etc... Donno 
how useful in general but in a larger system with a bunch of 
heterogeneous clients accessing it might be interesting.


Patrick



On Mon, Jan 11, 2010 at 9:49 AM, Patrick Hunt  wrote:

Adam, this is awesome! I've retweeted it on my twitter account:
http://twitter.com/phunt/status/7637054275

A few comments:

1) you are correct about the "off the shelf" part, but really alot of that
is our marketing blurb, we paint with a broad stroke. I think it would make
sense to include this sort of detail in the "recipes" page though, would you
be interested to create a JIRA and submit some content? (you could re
purpose alot of what you have in your post)

2) re zkclient - I have not been following that project closely of late,
however Mahadev and I did review their code a few months ago. At that time
we had some concerns about the semantic impact of some of the design choices
they made (which were fine, just not well documented and therefor easy for
users to miss). In particular (again, at the time, may be addressed by now)
watch notifications could be lost during disconnected/expired sessions, and
the user would not be made aware of this.

3) excellent point/suggestion on the "rogue" service

Patrick

Adam Rosien wrote:

I've written a blog post about using ZooKeeper for group membership at
KaChing:

http://bit.ly/6yraP6
http://eng.kaching.com/2010/01/actually-implementing-group-management.html

Any comments or feedback is welcome.

.. Adam

Status on upcoming ZK 3.3.0 release

[Patrick Hunt]

ZK 3.3.0 is currently slated for March 10th. You can see a JIRA level 
overview here:

https://issues.apache.org/jira/browse/ZOOKEEPER/fixforversion/12313976

Mahadev and I did an initial triage of the 3.3.0 JIRAs today. There are 
currently 77 open issues slated for inclusion in this release, vs 110 
already addressed.


What does this mean for you? If there's a JIRA assigned to you or that 
you created that's listed for 3.3.0 please review it. If you don't plan 
on working on it for 3.3.0 reschedule it (3.4.0 or later), if you do 
plan to work on it please do so (sooner == better). If you want to get 
something into 3.3.0 that's not listed for 3.3.0 please submit a patch 
asap. As the 3.3.0 deadline approaches I will continue triaging the 
issues, in particular I will start pushing out non-blocker JIRAs that 
are not actively being worked on.


Thank you,

Patrick

Re: zookeeper.jar contains java source code. why?

[Patrick Hunt]

FYI, the release build (package target in build.xml) in 3.3.0 (our 
upcoming release, current trunk code) will generate 4 jars files - the 
legacy jar, and separate bin/src/javadoc jars. These last 3 are planned 
for deployment to maven repo.


Patrick

Mahadev Konar wrote:

Hi Thomas,
  Its just been a legacy that we have been including source code in the
jars. We have found that sometimes folks apply patches on there own and push
there own jars to production. It just helped us debugging in such cases.
That's why its always been there. It has no connection to jmx at all.

mahadev


On 1/21/10 5:11 AM, "Thomas Koch"  wrote:


Hi,

the Debian packaging of zookeeper is nearly ready. Only one question left: The
zookeeper.jar contains not only .class files but also the source code (.java).
Why? And should I include them as well, when compiling for Debian?
What's the connection to JMX and this issue?

Best regards,

Thomas Koch, http://www.koch.ro

Re: 3.3.0 tentative release schedule ?

[Patrick Hunt]

That information can typically be found in the JIRA, the RoadMap page in 
particular provides a good summary:

https://issues.apache.org/jira/browse/ZOOKEEPER?report=com.atlassian.jira.plugin.system.project:roadmap-panel


Team members also send occasional messages to this list with status. 
(see attached)


Patrick

Kay Kay wrote:
Just curious what the release schedule of 3.3.0 looks like and what is 
the roadmap for the same.


--
  K K.

--- Begin Message ---
ZK 3.3.0 is currently slated for March 10th. You can see a JIRA level 
overview here:

https://issues.apache.org/jira/browse/ZOOKEEPER/fixforversion/12313976

Mahadev and I did an initial triage of the 3.3.0 JIRAs today. There are 
currently 77 open issues slated for inclusion in this release, vs 110 
already addressed.


What does this mean for you? If there's a JIRA assigned to you or that 
you created that's listed for 3.3.0 please review it. If you don't plan 
on working on it for 3.3.0 reschedule it (3.4.0 or later), if you do 
plan to work on it please do so (sooner == better). If you want to get 
something into 3.3.0 that's not listed for 3.3.0 please submit a patch 
asap. As the 3.3.0 deadline approaches I will continue triaging the 
issues, in particular I will start pushing out non-blocker JIRAs that 
are not actively being worked on.


Thank you,

Patrick
--- End Message ---

Re: Status on upcoming ZK 3.3.0 release

[Patrick Hunt]

Just a reminder, 3.3.0 is coming up fast. I will re-triage the JIRA list 
 sometime next week with an eye towards reducing the list of "fix for 
3.3.0", pushing non-critical, non-resourced issues to the 3.4.0 release. 
So if you have something you want to get into 3.3.0 that's a non-blocker 
please submit a patch asap.


Patrick

Patrick Hunt wrote:
ZK 3.3.0 is currently slated for March 10th. You can see a JIRA level 
overview here:

https://issues.apache.org/jira/browse/ZOOKEEPER/fixforversion/12313976

Mahadev and I did an initial triage of the 3.3.0 JIRAs today. There are 
currently 77 open issues slated for inclusion in this release, vs 110 
already addressed.


What does this mean for you? If there's a JIRA assigned to you or that 
you created that's listed for 3.3.0 please review it. If you don't plan 
on working on it for 3.3.0 reschedule it (3.4.0 or later), if you do 
plan to work on it please do so (sooner == better). If you want to get 
something into 3.3.0 that's not listed for 3.3.0 please submit a patch 
asap. As the 3.3.0 deadline approaches I will continue triaging the 
issues, in particular I will start pushing out non-blocker JIRAs that 
are not actively being worked on.


Thank you,

Patrick

Re: ZOOKEEPER-22 and release 3.3

[Patrick Hunt]

While this is a very useful improvement it sounds to me like the prudent 
thing to do given the short time to 3.3.0. If you want we can shoot for 
3.4.0 soon after 3.3.0 goes out (with 22 as one of the primary features).


Patrick

Mahadev Konar wrote:

Hi all,

 I had been working on zookeeper-22 and found out that it needs quite a few
extensive changes. We will need to do some memory measurements to see if it
has any memory impacts or not.

Since we are targetting 3.3 release for early march, ZOOKEEPER-22 would be
hard to get into 3.3. I am proposing to move it to a later release (3.4), so
that it can be tested early in the release phase and gets baked in the
release. 



Thanks
mahadev

Re: Might I contribute a Node design for the Java API?

[Patrick Hunt]

This sounds pretty useful as a contrib, ff to submit a patch 
http://bit.ly/bgYpW5 . Be sure to include README type details on 
semantics of use (I don't see watches so this might be pretty simple).


Also see: http://bit.ly/bhcpwx a few ZK users are using this now afaict. 
However last I checked docs were thin and there def. were some issues 
with the semantics.


Regards,

Patrick

Aaron Crow wrote:

Dear ZooKeepers,

We recently integrated ZooKeeper into our Java project at work. This is a
very cool project... I feel like a kid with a new toy. After using the Java
API a little, I've begun to create some simple wrappers around the main
ZooKeeper object.

Specifically, I'm creating a simple Node design, that makes it simpler and
more convenient for me to do simple management of my ZooKeeper node state.
Just as a rough idea:

Node node = myZk.getNode(path);
if(node.exists()){
  String data = node.getData();
  Collection children = node.getChildren();
  node.delete();
}

I was wondering... am I re-inventing the wheel here (did I miss any existing
API like this)? Or if not, would the community have any interest if I wanted
to add this into the main project?

Many thanks in advance for any insights or guidance...


All the best,
Aaron

Re: Might I contribute a Node design for the Java API?

[Patrick Hunt]

Originally the c/java apis "grew up as sibilings" so there wasn't much 
attempt to make the java very OO friendly. What you are suggesting makes 
sense, and my personal thought is that if done correctly could 
"graduate" to main at some point (prolly as a complement to the current 
api, but lets start with contrib and see where it goes from there).


Regards,

Patrick

Aaron Crow wrote:

Hi Patrick,

Thanks! I took a look at zkclient. Since, as you say, it's not very well
documented, I'm not 100% clear on the goals of the project. However, I did
see that ZooKeeper nodes are described with (String)path throughout, so that
tells me that zkclient hasn't adopted the first-class "Node" design I'm
proposing. I think, if the "Node" design was found useful by people AND the
same people found zkclient useful, there would be some integration
possibilities so that the two could work hand-in-hand.

Thanks again for your reply, and thanks in advance for any further guidance
or thoughts...


All the best,
Aaron


On Tue, Feb 16, 2010 at 11:30 AM, Patrick Hunt  wrote:


This sounds pretty useful as a contrib, ff to submit a patch
http://bit.ly/bgYpW5 . Be sure to include README type details on semantics
of use (I don't see watches so this might be pretty simple).

Also see: http://bit.ly/bhcpwx a few ZK users are using this now afaict.
However last I checked docs were thin and there def. were some issues with
the semantics.

Regards,

Patrick


Aaron Crow wrote:


Dear ZooKeepers,

We recently integrated ZooKeeper into our Java project at work. This is a
very cool project... I feel like a kid with a new toy. After using the
Java
API a little, I've begun to create some simple wrappers around the main
ZooKeeper object.

Specifically, I'm creating a simple Node design, that makes it simpler and
more convenient for me to do simple management of my ZooKeeper node state.
Just as a rough idea:

Node node = myZk.getNode(path);
if(node.exists()){
 String data = node.getData();
 Collection children = node.getChildren();
 node.delete();
}

I was wondering... am I re-inventing the wheel here (did I miss any
existing
API like this)? Or if not, would the community have any interest if I
wanted
to add this into the main project?

Many thanks in advance for any insights or guidance...


All the best,
Aaron

Re: Reg: Status of ZOOKEEPER-107 - Allow dynamic changes to server cluster membership

[Patrick Hunt]

There are ways to "dynamically" update the cluster membership today, 
granted while they are dynamic they are not dynamic in the sense of 107: 
http://bit.ly/bTTdDK (see item 6)


107 is still pending. Henry was interested to implement this, I'm not 
sure his current status. Henry? I think this is a good feature to add, 
but it brings up a number of issues that we need to work through. Given 
that most ppl rarely change cluster membership, and the fact that there 
are options available today, there hasn't been a lot of demand for this.


Patrick

Hariharan Subramanian wrote:

Dear Zookeeper community,

I would like to know the status of issue ZOOKEEPER-107  (Allow dynamic
changes to server cluster membership) which is assigned to Henry Robinson

Is this the right forum to ask this question or should I post an update to
the bug report?

Thanks
-- Hari

Re: Reg: Status of ZOOKEEPER-107 - Allow dynamic changes to server cluster membership

[Patrick Hunt]

FYI, Ben has been working on Netty integration with ZooKeeper. One of 
the benefits of this is it provides encryption and certificate based 
connection auth - which would be a great way (cert) to solve some of the 
security issues highlighted in 107. So the pieces are there, interest is 
there, support is there, as Henry mentioned we need to find the resources.


Patrick

Henry Robinson wrote:

Hi -

Yes, I'm still interested in this JIRA but it's a significant change to ZK
which would be best suited for probably the 4.0 release. I have a rough
implementation that 'works' but still leaves some issues unresolved. I don't
have a great deal of time for this issue at the moment though, which is why
it's stalled a bit - other ZK issues are more pressing. This would be a
really great project for anyone looking for a meaty distributed systems
project; I'm happy to offer guidance and advice if anyone wants to take it
on!

Henry

On 18 February 2010 09:58, Patrick Hunt  wrote:


There are ways to "dynamically" update the cluster membership today,
granted while they are dynamic they are not dynamic in the sense of 107:
http://bit.ly/bTTdDK (see item 6)

107 is still pending. Henry was interested to implement this, I'm not sure
his current status. Henry? I think this is a good feature to add, but it
brings up a number of issues that we need to work through. Given that most
ppl rarely change cluster membership, and the fact that there are options
available today, there hasn't been a lot of demand for this.

Patrick


Hariharan Subramanian wrote:


Dear Zookeeper community,

I would like to know the status of issue ZOOKEEPER-107  (Allow dynamic
changes to server cluster membership) which is assigned to Henry Robinson

Is this the right forum to ask this question or should I post an update to
the bug report?

Thanks
-- Hari

Ruby client binding for ZooKeeper now available.

[Patrick Hunt]

Sorry if this is a dup for those of you following me on twitter 
(http://twitter.com/phunt) but I wanted to let you know that twitter 
(the company) has contributed a Ruby client binding for ZooKeeper. You 
can learn more about the gem here: http://bit.ly/b9VB6k


Regards,

Patrick

Re: Might I contribute a Node design for the Java API?

[Patrick Hunt]

I marked it for 3.4.0, that ok with you? We are all hunkered down at 
this point trying to wrap everything up for the 3/10 3.3.0 deadline. I 
marked 3.4.0 (should come out relatively soon after 3.3.0, 3/4 months 
say) so that we have enough time to vet it before exposing potential 
users (which makes it more difficult for example to make changes to the 
API).


Patrick

Aaron Crow wrote:

Thanks Patrick. I submitted a patch:
https://issues.apache.org/jira/browse/ZOOKEEPER-679

Looking forward to any feedback...

All the best,
Aaron


On Thu, Feb 18, 2010 at 9:48 AM, Patrick Hunt  wrote:


Originally the c/java apis "grew up as sibilings" so there wasn't much
attempt to make the java very OO friendly. What you are suggesting makes
sense, and my personal thought is that if done correctly could "graduate" to
main at some point (prolly as a complement to the current api, but lets
start with contrib and see where it goes from there).


Regards,

Patrick

Aaron Crow wrote:


Hi Patrick,

Thanks! I took a look at zkclient. Since, as you say, it's not very well
documented, I'm not 100% clear on the goals of the project. However, I did
see that ZooKeeper nodes are described with (String)path throughout, so
that
tells me that zkclient hasn't adopted the first-class "Node" design I'm
proposing. I think, if the "Node" design was found useful by people AND
the
same people found zkclient useful, there would be some integration
possibilities so that the two could work hand-in-hand.

Thanks again for your reply, and thanks in advance for any further
guidance
or thoughts...


All the best,
Aaron


On Tue, Feb 16, 2010 at 11:30 AM, Patrick Hunt  wrote:

 This sounds pretty useful as a contrib, ff to submit a patch

http://bit.ly/bgYpW5 . Be sure to include README type details on
semantics
of use (I don't see watches so this might be pretty simple).

Also see: http://bit.ly/bhcpwx a few ZK users are using this now afaict.
However last I checked docs were thin and there def. were some issues
with
the semantics.

Regards,

Patrick


Aaron Crow wrote:

 Dear ZooKeepers,

We recently integrated ZooKeeper into our Java project at work. This is
a
very cool project... I feel like a kid with a new toy. After using the
Java
API a little, I've begun to create some simple wrappers around the main
ZooKeeper object.

Specifically, I'm creating a simple Node design, that makes it simpler
and
more convenient for me to do simple management of my ZooKeeper node
state.
Just as a rough idea:

Node node = myZk.getNode(path);
if(node.exists()){
 String data = node.getData();
 Collection children = node.getChildren();
 node.delete();
}

I was wondering... am I re-inventing the wheel here (did I miss any
existing
API like this)? Or if not, would the community have any interest if I
wanted
to add this into the main project?

Many thanks in advance for any insights or guidance...


All the best,
Aaron

Re: Status on upcoming ZK 3.3.0 release

[Patrick Hunt]

March 10 is rapidly approaching. There aren't any blockers left, my plan 
is to commit any reviewed "patch availables" and push any remaining 
non-blockers into 3.4.0 on Weds as part of cutting the 3.3.0 release 
candidate. If you have something that you want to get into 3.3.0 you 
need to get the patch in now. Final warning. ;-)


Patrick

Patrick Hunt wrote:
Just a reminder, 3.3.0 is coming up fast. I will re-triage the JIRA list 
 sometime next week with an eye towards reducing the list of "fix for 
3.3.0", pushing non-critical, non-resourced issues to the 3.4.0 release. 
So if you have something you want to get into 3.3.0 that's a non-blocker 
please submit a patch asap.


Patrick

Patrick Hunt wrote:
ZK 3.3.0 is currently slated for March 10th. You can see a JIRA level 
overview here:

https://issues.apache.org/jira/browse/ZOOKEEPER/fixforversion/12313976

Mahadev and I did an initial triage of the 3.3.0 JIRAs today. There 
are currently 77 open issues slated for inclusion in this release, vs 
110 already addressed.


What does this mean for you? If there's a JIRA assigned to you or that 
you created that's listed for 3.3.0 please review it. If you don't 
plan on working on it for 3.3.0 reschedule it (3.4.0 or later), if you 
do plan to work on it please do so (sooner == better). If you want to 
get something into 3.3.0 that's not listed for 3.3.0 please submit a 
patch asap. As the 3.3.0 deadline approaches I will continue triaging 
the issues, in particular I will start pushing out non-blocker JIRAs 
that are not actively being worked on.


Thank you,

Patrick

Re: Status on upcoming ZK 3.3.0 release

[Patrick Hunt]

I was planning to cut a release candidate today, however there are a few 
blockers outstanding for this release that we need to address before I 
can do so. We're in the final stretch though.


There are a number of open issues still pending for 3.3.0, however most 
of these are non-blockers and I've talked with the assigned party about 
policy (as soon as the blockers are addressed I will push remaining 
non-blockers to 3.4.0 and cut a release candidate). Unless you feel 
strongly about an issue push it to 3.4.0 asap.


Recent changes have caused intermittent test failures (blocker issues) 
on hudson. Hudson runs with clover so these are probably timing related 
(clover test builds run much more slowly). We are working to address 
these with highest priority. Additionally Hudson itself has been 
unstable of late which is impacting our ability to process patches. Over 
the next few days things will stabilize and we'll cut the release 
candidate for 3.3.0.


I encourage people following this list to run the tests themselves 
(checkout trunk and type "ant test") and enter JIRAs for any issues you 
find. Better now than once the release candidate is cut. Better now than 
when you need to roll out the changes to your environment and find 
issues that will take a fix release to address.


Regards,

Patrick


Patrick Hunt wrote:
March 10 is rapidly approaching. There aren't any blockers left, my plan 
is to commit any reviewed "patch availables" and push any remaining 
non-blockers into 3.4.0 on Weds as part of cutting the 3.3.0 release 
candidate. If you have something that you want to get into 3.3.0 you 
need to get the patch in now. Final warning. ;-)


Patrick

Patrick Hunt wrote:
Just a reminder, 3.3.0 is coming up fast. I will re-triage the JIRA 
list  sometime next week with an eye towards reducing the list of "fix 
for 3.3.0", pushing non-critical, non-resourced issues to the 3.4.0 
release. So if you have something you want to get into 3.3.0 that's a 
non-blocker please submit a patch asap.


Patrick

Patrick Hunt wrote:
ZK 3.3.0 is currently slated for March 10th. You can see a JIRA level 
overview here:

https://issues.apache.org/jira/browse/ZOOKEEPER/fixforversion/12313976

Mahadev and I did an initial triage of the 3.3.0 JIRAs today. There 
are currently 77 open issues slated for inclusion in this release, vs 
110 already addressed.


What does this mean for you? If there's a JIRA assigned to you or 
that you created that's listed for 3.3.0 please review it. If you 
don't plan on working on it for 3.3.0 reschedule it (3.4.0 or later), 
if you do plan to work on it please do so (sooner == better). If you 
want to get something into 3.3.0 that's not listed for 3.3.0 please 
submit a patch asap. As the 3.3.0 deadline approaches I will continue 
triaging the issues, in particular I will start pushing out 
non-blocker JIRAs that are not actively being worked on.


Thank you,

Patrick

Re: Google Summer of Code

[Patrick Hunt]

Henry, good idea. I added an entry for BookKeeper FUSE module that I 
came up with but haven't had time to work on.


Regarding Gustavo's suggestion, last I talked with Ben he had 
implemented the netty changes (adds encryp/certauth support) but hadn't 
had a chance to chase down all the bugs yet. He really wants to finish 
it for 3.4.0 though.


> 1. An interface which can return the clients which set watcher on 
specific
> znode of the data tree. This kind of interface can be really helpful 
for the

> administrators.

This item (list watchers for admin purposes) is already in 3.3.0:
https://issues.apache.org/jira/browse/ZOOKEEPER-572

Patrick

Qian Ye wrote:

Hi Henry,

I think we should add two kinds of interface to the server:
1. An interface which can return the clients which set watcher on specific
znode of the data tree. This kind of interface can be really helpful for the
administrators.
2. An interface which can return a list of servers in a zookeeper cluster.

Maybe the students can help to do this work.

thx~

On Wed, Mar 10, 2010 at 4:46 AM, Gustavo Niemeyer wrote:


Hi Henry,


There is a wiki page here:
http://wiki.apache.org/hadoop/ZooKeeper/SoC2010Ideas that requires that

you

sign up to edit. Please post your project ideas up there - I've left one

as

an example. You can also mail me directly and I'll post them myself. On
Friday I'll tidy up the page and send in an application to Google.

Thanks a lot for organizing this.

The key things I'd like to see moving forward, as was discussed before
in the mailing list, are:

- Encryption of communication between servers
- Encryption of communication between servers and clients
- Dynamic cluster membership changes

I don't know how well these fit in GSoC.

--
Gustavo Niemeyer
http://niemeyer.net
http://niemeyer.net/blog
http://niemeyer.net/identi.ca
http://niemeyer.net/twitter

Re: Status on upcoming ZK 3.3.0 release

[Patrick Hunt]

The release candidate has been blocked by ZOOKEEPER-121. We're working 
to address, it's been failing intermittently on hudson. Additionally 
Hudson itself has been very unstable, Mahadev and I have been working to 
address this as well but that's slowing us considerably. Once the 
blockers are dealt with, which should be any time now if hudson 
cooperates, I should be publishing a release candidate.


Patrick

Patrick Hunt wrote:
I was planning to cut a release candidate today, however there are a few 
blockers outstanding for this release that we need to address before I 
can do so. We're in the final stretch though.


There are a number of open issues still pending for 3.3.0, however most 
of these are non-blockers and I've talked with the assigned party about 
policy (as soon as the blockers are addressed I will push remaining 
non-blockers to 3.4.0 and cut a release candidate). Unless you feel 
strongly about an issue push it to 3.4.0 asap.


Recent changes have caused intermittent test failures (blocker issues) 
on hudson. Hudson runs with clover so these are probably timing related 
(clover test builds run much more slowly). We are working to address 
these with highest priority. Additionally Hudson itself has been 
unstable of late which is impacting our ability to process patches. Over 
the next few days things will stabilize and we'll cut the release 
candidate for 3.3.0.


I encourage people following this list to run the tests themselves 
(checkout trunk and type "ant test") and enter JIRAs for any issues you 
find. Better now than once the release candidate is cut. Better now than 
when you need to roll out the changes to your environment and find 
issues that will take a fix release to address.


Regards,

Patrick


Patrick Hunt wrote:
March 10 is rapidly approaching. There aren't any blockers left, my 
plan is to commit any reviewed "patch availables" and push any 
remaining non-blockers into 3.4.0 on Weds as part of cutting the 3.3.0 
release candidate. If you have something that you want to get into 
3.3.0 you need to get the patch in now. Final warning. ;-)


Patrick

Patrick Hunt wrote:
Just a reminder, 3.3.0 is coming up fast. I will re-triage the JIRA 
list  sometime next week with an eye towards reducing the list of 
"fix for 3.3.0", pushing non-critical, non-resourced issues to the 
3.4.0 release. So if you have something you want to get into 3.3.0 
that's a non-blocker please submit a patch asap.


Patrick

Patrick Hunt wrote:
ZK 3.3.0 is currently slated for March 10th. You can see a JIRA 
level overview here:

https://issues.apache.org/jira/browse/ZOOKEEPER/fixforversion/12313976

Mahadev and I did an initial triage of the 3.3.0 JIRAs today. There 
are currently 77 open issues slated for inclusion in this release, 
vs 110 already addressed.


What does this mean for you? If there's a JIRA assigned to you or 
that you created that's listed for 3.3.0 please review it. If you 
don't plan on working on it for 3.3.0 reschedule it (3.4.0 or 
later), if you do plan to work on it please do so (sooner == 
better). If you want to get something into 3.3.0 that's not listed 
for 3.3.0 please submit a patch asap. As the 3.3.0 deadline 
approaches I will continue triaging the issues, in particular I will 
start pushing out non-blocker JIRAs that are not actively being 
worked on.


Thank you,

Patrick

Re: Zookeeper issues

[Patrick Hunt]

It looks like you are losing connectivity btw your client and server. 
You'd need to provide some basic information for us to do any debugging 
- for example what version of server/client are you using?


The first "connection loss" exception is probably due to the fact that 
you got disconnected while a request (getdata) was in progress. However 
it's not totally clear to me as you don't have date/time in your log, so 
hard for me to have context.


I suspect the second error is due to session expiration, either that or 
you are exceeding the "maxclientcnxns" (10 by default) from a single 
host. There's isn't enough log detail for me to be sure.


***Could it be that you are creating a ZK client connection for each 
operation of your "web interface"? (or maybe not closing the connection 
in all cases afterward?) If this is they case you are more than likely 
hitting the "maxclientcnxns" limit, try grepping the server logs (all of 
them on all servers) for "Too many connections from" and see if that 
shows up at all.


Try using the "4 letter words" against your servers to see what your 
connection status looks like - in particular the "stat" and "dump" 
commands. http://bit.ly/dglVld Hopefully you are using this for 
monitoring, if so you might be able to look back at historical 
information on this.


Patrick

Goteti Bhanu wrote:

Hi,

 We are having issues with our web interface (WI) to zookeeper. We 
deployed a java app (WI), in tomcat, which shows a (mostly read-only) UI for 
the zk tree, in our cluster. The app had been deployed quite some time ago. 
However, recently, it is going down many times, in the recent past, showing 
these errors in the catalina.out. These errors are coming continuously in the 
catalina.out, while the website is down.


[INFO] SyncZKTreeWatcher - [registerWatches] About to register watch for 
'/llf/version-1.0/serverNodes/llf-server/nodes/b3091258:8085/connected', 
recursive: false
[INFO] SyncZKTreeWatcher - Path: null, type: None, state: Disconnected
[INFO] SyncZKTreeWatcher - Path: null, type: None, state: SyncConnected
[WARN] SyncZKTreeWatcher - [run] Exception while handling event WatchedEvent: Znode 
change. Path: /llf/version-1.0/serverNodes/llf-server/nodes/b3091258:8085/connected 
Type: NodeChildrenChanged 
org.apache.zookeeper.KeeperException$ConnectionLossException: 
KeeperErrorCode = ConnectionLoss
at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:809)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:837)
at 
com.yahoo.cluster.zookeeper.SyncZKTreeWatcher.getNodeValue(SyncZKTreeWatcher.java:465)
at 
com.yahoo.cluster.zookeeper.SyncZKTreeWatcher.registerWatches(SyncZKTreeWatcher.java:376)
at 
com.yahoo.cluster.zookeeper.SyncZKTreeWatcher.access$100(SyncZKTreeWatcher.java:37)
at 
com.yahoo.cluster.zookeeper.SyncZKTreeWatcher$1.run(SyncZKTreeWatcher.java:169)
at java.lang.Thread.run(Thread.java:619)


In another log, we are getting the following errors continuously.


java.net.SocketException: Transport endpoint is not connected
at sun.nio.ch.SocketChannelImpl.shutdown(Native Method)
at 
sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:651)
at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:368)
at 
org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:935)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:901)
2010 03 16 09:49:55,645   org.apache.zookeeper.ClientCnxn ZK Event 
Handler-SendThread Attempting connection to server 
b3091279.crawl.yahoo.net/67.195.112.40:22801
2010 03 16 09:49:55,646   org.apache.zookeeper.ClientCnxn ZK Event 
Handler-SendThread Priming connection to 
java.nio.channels.SocketChannel[connected local=/67.195.37.119:38793 
remote=b3091279.crawl.yahoo.net/67.195.112.40:22801]
2010 03 16 09:49:55,739   org.apache.zookeeper.ClientCnxn ZK Event 
Handler-SendThread Server connection successful
2010 03 16 09:49:55,775   org.apache.zookeeper.ClientCnxn ZK Event Handler-SendThread 
Exception closing session 0xe727001201aa32ab to 
sun.nio.ch.selectionkeyi...@3ef92f5e
java.io.IOException: Read error rc = -1 java.nio.DirectByteBuffer[pos=0 lim=4 
cap=4]
at org.apache.zookeeper.ClientCnxn$SendThread.doIO(ClientCnxn.java:632)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:876)
2010 03 16 09:49:55,775   org.apache.zookeeper.ClientCnxn ZK Event 
Handler-SendThread Ignoring exception during shutdown input


What do these errors indicate? Always, restarting tomcat helps. But 
since the problem is occurring many times, we want to understand why is this 
happening? Is there a known ZK issue which could cause such behavior?

Thanks
Udaya

Re: Status on upcoming ZK 3.3.0 release

[Patrick Hunt]

Another update. 121 has been resolved, all of the remaining patches have 
been applied and non-blockers moved out of 3.3.0.


A new issue has recently been identified (see SESSION MOVED thread on 
the user list). One user is reporting "session moved" errors in 3.2.2 
when there shouldn't be. No jira on this yet but the user is creating a 
JIRA to track. This is a blocker for 3.3.0 until we can triage and 
identify the underlying problem. This is the last remaining issue so 
we'll be focused on this over the next day or two, once addressed I'll 
cut RC 0 for 3.3.0.


Patrick

Patrick Hunt wrote:
The release candidate has been blocked by ZOOKEEPER-121. We're working 
to address, it's been failing intermittently on hudson. Additionally 
Hudson itself has been very unstable, Mahadev and I have been working to 
address this as well but that's slowing us considerably. Once the 
blockers are dealt with, which should be any time now if hudson 
cooperates, I should be publishing a release candidate.


Patrick

Patrick Hunt wrote:
I was planning to cut a release candidate today, however there are a 
few blockers outstanding for this release that we need to address 
before I can do so. We're in the final stretch though.


There are a number of open issues still pending for 3.3.0, however 
most of these are non-blockers and I've talked with the assigned party 
about policy (as soon as the blockers are addressed I will push 
remaining non-blockers to 3.4.0 and cut a release candidate). Unless 
you feel strongly about an issue push it to 3.4.0 asap.


Recent changes have caused intermittent test failures (blocker issues) 
on hudson. Hudson runs with clover so these are probably timing 
related (clover test builds run much more slowly). We are working to 
address these with highest priority. Additionally Hudson itself has 
been unstable of late which is impacting our ability to process 
patches. Over the next few days things will stabilize and we'll cut 
the release candidate for 3.3.0.


I encourage people following this list to run the tests themselves 
(checkout trunk and type "ant test") and enter JIRAs for any issues 
you find. Better now than once the release candidate is cut. Better 
now than when you need to roll out the changes to your environment and 
find issues that will take a fix release to address.


Regards,

Patrick


Patrick Hunt wrote:
March 10 is rapidly approaching. There aren't any blockers left, my 
plan is to commit any reviewed "patch availables" and push any 
remaining non-blockers into 3.4.0 on Weds as part of cutting the 
3.3.0 release candidate. If you have something that you want to get 
into 3.3.0 you need to get the patch in now. Final warning. ;-)


Patrick

Patrick Hunt wrote:
Just a reminder, 3.3.0 is coming up fast. I will re-triage the JIRA 
list  sometime next week with an eye towards reducing the list of 
"fix for 3.3.0", pushing non-critical, non-resourced issues to the 
3.4.0 release. So if you have something you want to get into 3.3.0 
that's a non-blocker please submit a patch asap.


Patrick

Patrick Hunt wrote:
ZK 3.3.0 is currently slated for March 10th. You can see a JIRA 
level overview here:

https://issues.apache.org/jira/browse/ZOOKEEPER/fixforversion/12313976

Mahadev and I did an initial triage of the 3.3.0 JIRAs today. There 
are currently 77 open issues slated for inclusion in this release, 
vs 110 already addressed.


What does this mean for you? If there's a JIRA assigned to you or 
that you created that's listed for 3.3.0 please review it. If you 
don't plan on working on it for 3.3.0 reschedule it (3.4.0 or 
later), if you do plan to work on it please do so (sooner == 
better). If you want to get something into 3.3.0 that's not listed 
for 3.3.0 please submit a patch asap. As the 3.3.0 deadline 
approaches I will continue triaging the issues, in particular I 
will start pushing out non-blocker JIRAs that are not actively 
being worked on.


Thank you,

Patrick

Re: Status on upcoming ZK 3.3.0 release

[Patrick Hunt]

Looks like MOVED issue has been resolved (ZOOKEEPER-710). I'm planning 
to publish a 3.3.0 release candidate tomorrow.


Patrick

Patrick Hunt wrote:
Another update. 121 has been resolved, all of the remaining patches have 
been applied and non-blockers moved out of 3.3.0.


A new issue has recently been identified (see SESSION MOVED thread on 
the user list). One user is reporting "session moved" errors in 3.2.2 
when there shouldn't be. No jira on this yet but the user is creating a 
JIRA to track. This is a blocker for 3.3.0 until we can triage and 
identify the underlying problem. This is the last remaining issue so 
we'll be focused on this over the next day or two, once addressed I'll 
cut RC 0 for 3.3.0.


Patrick

Patrick Hunt wrote:
The release candidate has been blocked by ZOOKEEPER-121. We're working 
to address, it's been failing intermittently on hudson. Additionally 
Hudson itself has been very unstable, Mahadev and I have been working 
to address this as well but that's slowing us considerably. Once the 
blockers are dealt with, which should be any time now if hudson 
cooperates, I should be publishing a release candidate.


Patrick

Patrick Hunt wrote:
I was planning to cut a release candidate today, however there are a 
few blockers outstanding for this release that we need to address 
before I can do so. We're in the final stretch though.


There are a number of open issues still pending for 3.3.0, however 
most of these are non-blockers and I've talked with the assigned 
party about policy (as soon as the blockers are addressed I will push 
remaining non-blockers to 3.4.0 and cut a release candidate). Unless 
you feel strongly about an issue push it to 3.4.0 asap.


Recent changes have caused intermittent test failures (blocker 
issues) on hudson. Hudson runs with clover so these are probably 
timing related (clover test builds run much more slowly). We are 
working to address these with highest priority. Additionally Hudson 
itself has been unstable of late which is impacting our ability to 
process patches. Over the next few days things will stabilize and 
we'll cut the release candidate for 3.3.0.


I encourage people following this list to run the tests themselves 
(checkout trunk and type "ant test") and enter JIRAs for any issues 
you find. Better now than once the release candidate is cut. Better 
now than when you need to roll out the changes to your environment 
and find issues that will take a fix release to address.


Regards,

Patrick


Patrick Hunt wrote:
March 10 is rapidly approaching. There aren't any blockers left, my 
plan is to commit any reviewed "patch availables" and push any 
remaining non-blockers into 3.4.0 on Weds as part of cutting the 
3.3.0 release candidate. If you have something that you want to get 
into 3.3.0 you need to get the patch in now. Final warning. ;-)


Patrick

Patrick Hunt wrote:
Just a reminder, 3.3.0 is coming up fast. I will re-triage the JIRA 
list  sometime next week with an eye towards reducing the list of 
"fix for 3.3.0", pushing non-critical, non-resourced issues to the 
3.4.0 release. So if you have something you want to get into 3.3.0 
that's a non-blocker please submit a patch asap.


Patrick

Patrick Hunt wrote:
ZK 3.3.0 is currently slated for March 10th. You can see a JIRA 
level overview here:
https://issues.apache.org/jira/browse/ZOOKEEPER/fixforversion/12313976 



Mahadev and I did an initial triage of the 3.3.0 JIRAs today. 
There are currently 77 open issues slated for inclusion in this 
release, vs 110 already addressed.


What does this mean for you? If there's a JIRA assigned to you or 
that you created that's listed for 3.3.0 please review it. If you 
don't plan on working on it for 3.3.0 reschedule it (3.4.0 or 
later), if you do plan to work on it please do so (sooner == 
better). If you want to get something into 3.3.0 that's not listed 
for 3.3.0 please submit a patch asap. As the 3.3.0 deadline 
approaches I will continue triaging the issues, in particular I 
will start pushing out non-blocker JIRAs that are not actively 
being worked on.


Thank you,

Patrick

[VOTE] Release ZooKeeper 3.3.0 (candidate 0)

[Patrick Hunt]

I have created a candidate build for ZooKeeper 3.3.0. Over 180 JIRAs are 
addressed in this release.


*** Please download, test and VOTE before the
*** vote closes 1pm PT on Wednesday, March 24.***

http://people.apache.org/~phunt/zookeeper-3.3.0-candidate-0/

Should we release this?

Patrick

Draft notes for 3.3.0 release announcement

[Patrick Hunt]

Once 3.3.0 is approved I will send out an announcement that historically 
contains a short list of high impact items contained in the release. 
Here is my list so far, if you have anything to add let me know and I'll 
work on incorporating it.


Patrick

3.3.0
observers - non-voting members of the ensemble, scale reads 
http://bit.ly/9wPnpq

distributed queue recipe implementation (c/java)
additional 4letterword/jmx features to support operations
maven repository support
build support for eclipse project generation
ivy based build

contrib:
zookeeper-tree: export/import znode namespace
zooinspector: gui browser for znode namespace
bookkeeper client rewrite, use of netty

Re: ZooKeeper & dynamic discovery (apache river)

[Patrick Hunt]

Hi Brian, sounds like a cool project. While I'm not super familiar with 
Apache River or Jini I do think what you are describing could be 
beneficial. I saw the following recently, is what you are describing a 
possible solution for this situation? http://bit.ly/dabIev


Historically specifying the ZK server host/port list for client 
applications (zookeeper client sessions) has been an issue for some 
users. If we could provide an option that used River to discover this I 
believe it would be useful.


We'd be happy to work with you to get something into our contrib module. 
We recently did ivy integration for BookKeeper contrib so there's prior 
art here, we just need to duplicate and change the dependencies. I'd 
help you with this.


Feel free to open a JIRA and submit a patch to get started:
https://issues.apache.org/jira/browse/ZOOKEEPER
http://wiki.apache.org/hadoop/ZooKeeper/HowToContribute

Regards,

Patrick

Brian Murphy wrote:

I've recently been assigned to a project using
ZooKeeper and so I'm just a (new) user of ZooKeeper,
rather than someone interested in making changes to
anything internal to ZooKeeper. With that in mind, I'm not
sure if this is the right list for the question I have. If it's not,
then please forgive the interruption; and I hope someone
will (gently) point me to the correct list to pose the
question.

So here's the context to the question I'd like to ask:

The project I've been assigned to would like to be able
to do the following with respect to ZooKeeper:

 - dynamically discover each peer in an ensemble
 - as part of the dynamic discovery of a given peer,
   advertise the peer's configuration; in particular, the
   clientPort, and the peer and election ports and
   peer address
 - administratively shutdown (in a graceful fashion) a
   given peer

To achieve this we've wrapped QuorumPeerMain
and ZooKeeperServerMain in an apache river
based backend impl class that registers a frontend
proxy with a river service directory, from which that
proxy (and associated configuration info attribute)
can be dynamically discovered.

None of the work above requires any changes to
the existing ZooKeeper codebase; merely the addition
of a handful of classes that can be used or ignored.

The question I have then, is whether anyone in the
ZooKeeper community would be interested in this work
being contributed. If not, no problem. Since the classes
above currently do what we want, we'll simply move
those new classes into our project's namespace, link
to ZooKeeper in the normal way, and go from there.

So before we spend any time on trying to figure out how
to use ivy to retrieve the appropriate river jars (which we
may need help with) or creating example apps & configs
to help others to understand and try out this functionality,
we thought we'd first try to determine if there's even any
interest; and, if there is, then engage the community in
how to proceed.

Anyway, if there's any interest, please let us know.

Regards,
Brian

Re: Stuck Zookeeper-trunk build

[Patrick Hunt]

Thanks for checking on this Giri, looks like it may be a real issue:

 [exec] Zookeeper_simpleSystem::testHangingClient
 [exec] make: *** [run-check] Terminated

I'll investigate with Mahadev, he was working on this recently. Thanks.

Patrick

Giridharan Kesavan wrote:

Zookeeper trunk build seem to be stuck:

hudson   21455  0.0  0.2 1202248 44484 ?   S

Re: A deficiency? You can only add children to persistent nodes

[Patrick Hunt]

Hi Lei, that's perfectly fine, however the issue would be that 
operations such as "getchildren(/path)" would return all these, vs if 
you had an explicit path structure such as 
getchildren(/path/starburst/cluster/nodes) - ie flat vs hierarchical.


Patrick

Lei Zhang wrote:

Hi Dominic,

Is it acceptable to use ephemeral nodes with hierarchical names, such as
"starburst.cluster.nodes"?

Re: A deficiency? You can only add children to persistent nodes

[Patrick Hunt]

Dominic Williams wrote:

1/ If a node crashes or something else goes wrong, you leave behind
persistent nodes. Over time these will grow and grow, rather like the old
tmp folders used to fill with files under Windows


That's true. One either needs to use ephemerals or use persistent and 
have a "garbage collector" (implicit or explicit gc). In most cases it's 
preferable to use the ephemeral.



2/ Persistent nodes = nasty scalability *bottleneck* because you're actually
having to write to disk somewhere.


This is not actually how ZK works. All znodes regardless of 
persistent/ephemeral are written to disk persistently. Ephemeral nodes 
are tied to the session that created them. As long as the session is 
alive the ephemeral node is alive. Sessions themselves are 
persistently/reliably stored by the ZK cluster. This allows the shutdown 
of the entire cluster and restart it, all sessions/ephemerals will be 
maintained. Sessions can move from server to server (if say network 
connectivity to server A fails, or server A itself fails then the client 
will move to server B). The session and all ephemerals are maintained 
(well, as long as the client moves withing the expiration timeout value).



To avoid this I'm actually thinking of writing locking system where you work
out the existing chain not by enumerating sequential children, but by
looking at the contents of each temporary lock node to see what it is
waiting on. But... that's quite horrible. Was wondering whether there is
some technical reason why you ephemeral nodes can't have children??


There are a few cases to think about.

1) obviously ephemeral nodes can't have persistent children, this just 
doesn't make sense


2) ephemeral nodes have an owner - the session that created them. so it 
would also not make sense (in my mind at least) to have an ephemeral 
/foo with another ephemeral /foo/bar with a different owner.


3) so you are left with "ephemerals can be a child of an ephemeral with 
the same owner".


4) there are also issues of order. in particular what is the "deletion 
order" depth first or breadth first, etc...


I believe the answer so far has been "we don't do this because it's 
fairly complicated and we haven't seen any use cases that require it." 
In the cases I've seen so far there was either a misunderstanding of how 
zk worked, or a simpler way available.


Does that make sense? Thoughts?

Patrick

Re: Modify ZooKeeper Java client to hold weak references to Watcher objects

[Patrick Hunt]

Would "WeakWatcher", a proxy watcher that holds a weak reference to the 
proxied watcher, not also solve this with minimal overhead? You could do 
this today.



1/ Add a useWeakReferences parameter to new constructor (sets default
behaviour)
2/ Add alternative methods, which take useWeakWatcherRef boolean.


I would prefer option 2 over 1, since I could see some users wishing to 
use both weak/strong in the same session. Couldn't we just mark the 
watcher with a "implements WeakWatcher" obviating the need for an 
explicit parameter?


An additoinal option - we could do the same thing by instead having a 
"ZooKeeperWeakWatches" subclass of zookeeper that provides this 
additional functionality (if we could overcome other objections) which 
would not impact existing users. You might be able to do this today, or 
with minimal changes to the ZooKeeper class...



I would prefer deregisterWatcher, as a result.


Seems like this would be a useful feature. Today you can get away with 
not doing this by using something like a StrongWatcher proxy similar to 
above (or some other way of ignoring the eventual trigger). Each 
deregister requires a roundtrip to the server however (also handle the 
case when disconnected of course). We might want to allow multiple paths 
to be dereg as once... (the server will kill the connection if the 
request packet size exceeds 1mb though so keep that in mind too...). Not 
sure if there are other technical issues to consider.


Patrick

Dominic Williams wrote:

Quite a few platforms make the specification of weak listeners explicit e.g.
in ActionScript you can specify a boolean in addEventListener that specifies
whether the event source should hold weak references to listeners. Therefore
I think whether you want an event source to hold a weak or strong reference
to your listener is really an application-level choice i.e. it is the
application logic's choice whether to supply an anonymous inner class
instance that will disappear unless the event source holds a strong
reference, or a named object that you want to be freed from memory when
*you* remove your references to it.

I think for many application programming problems, there is no nice way to
manage memory unless an event source offers to hold weak references to
listeners.

deregisterWatcher certainly works for things like locks where you do try {
lock.lock(); } finally { lock.unlock(); } but how about where you are
continually creating objects that maintain a copy of some items beneath
nodes (where, say, you are constantly changing your focus from one node to
another).

In this case, most probably you will want to create a primitive that
calls deregisterWatcher in finalize(). But the problem of course is that
finalize() will never get called.

For that reason, you end up with references to primitives on which you
*must* call close() before you lose a reference to them.

In practice, this is just not possible to do reliably which is why I don't
think there can be a substitute to weak references.

Best, Dominic

On 19 March 2010 18:47, Henry Robinson  wrote:


(moved to zookeeper-dev)

This API exposes internal implementation details which ties future versions
of the client to supporting a particular set of semantics.

I would prefer deregisterWatcher, as a result.

Henry

On 19 March 2010 03:11, Dominic Williams 
wrote:
Hi I can see some people might be assigning for example anonymous class
instances as watchers/handlers, and not keeping any references to them.

To avoid breaking existing use cases, two options:

1/ Add a useWeakReferences parameter to new constructor (sets default
behaviour)

2/ Add alternative methods, which take useWeakWatcherRef boolean.

Internally will be trivial to have both

private final Map> dataWatches =
   new HashMap>();

private final Map> dataWatchesWeak =
new HashMap>();

On 18 March 2010 22:47, Ted Dunning  wrote:


This kind of sounds strange to me.

My typical idiom is to create a watcher but not retain any references

to

it

outside the client.  It sounds to me like your change will cause my
watchers
to be collected and deactivated when GC happens.

On Thu, Mar 18, 2010 at 3:32 AM, Dominic Williams <
thedwilli...@googlemail.com> wrote:


The current ZooKeeper client holds strong references to Watcher

objects.

I

want to change the client so it only holds weak references. Feedback
please.



--
Henry Robinson
Software Engineer
Cloudera
415-994-6679

Re: A deficiency? You can only add children to persistent nodes

[Patrick Hunt]

Dominic Williams wrote:

What I'd suggest might work:
- when the session that created the parent ends, ownership of the parent
could either be transferred to the owner/session that created the oldest
child, or instead ownership could be transferred to some kind of nominal
system session (which would delete the parent once the last ephemeral child
disappeared)


There may be some issues with idempotency here, also it could require 
extensive locking which drives up operation latencies (essentially 
"recursive delete"). It sounds possible, but someone would have to take 
a closer look as to the technical challenges involved.



Our general philosophy is to keep things as simple as possible wrt api, 
semantics, implementation, etc... Distributed communication is hard and 
while we handle a lot of the issues for you it's still complex. 
Following our philosophy generally makes the easy things simple and the 
hard things possible, additionally it reduces the number of bugs that we 
have in the implementation itself (both user and service code).


I don't wish to discourage you as much as provide insight/background 
into some of our decisions.


Regards,

Patrick



On 22 March 2010 16:44, Patrick Hunt  wrote:


Dominic Williams wrote:


1/ If a node crashes or something else goes wrong, you leave behind
persistent nodes. Over time these will grow and grow, rather like the old
tmp folders used to fill with files under Windows


That's true. One either needs to use ephemerals or use persistent and have
a "garbage collector" (implicit or explicit gc). In most cases it's
preferable to use the ephemeral.


 2/ Persistent nodes = nasty scalability *bottleneck* because you're

actually
having to write to disk somewhere.


This is not actually how ZK works. All znodes regardless of
persistent/ephemeral are written to disk persistently. Ephemeral nodes are
tied to the session that created them. As long as the session is alive the
ephemeral node is alive. Sessions themselves are persistently/reliably
stored by the ZK cluster. This allows the shutdown of the entire cluster and
restart it, all sessions/ephemerals will be maintained. Sessions can move
from server to server (if say network connectivity to server A fails, or
server A itself fails then the client will move to server B). The session
and all ephemerals are maintained (well, as long as the client moves withing
the expiration timeout value).


 To avoid this I'm actually thinking of writing locking system where you

work
out the existing chain not by enumerating sequential children, but by
looking at the contents of each temporary lock node to see what it is
waiting on. But... that's quite horrible. Was wondering whether there is
some technical reason why you ephemeral nodes can't have children??


There are a few cases to think about.

1) obviously ephemeral nodes can't have persistent children, this just
doesn't make sense

2) ephemeral nodes have an owner - the session that created them. so it
would also not make sense (in my mind at least) to have an ephemeral /foo
with another ephemeral /foo/bar with a different owner.

3) so you are left with "ephemerals can be a child of an ephemeral with the
same owner".

4) there are also issues of order. in particular what is the "deletion
order" depth first or breadth first, etc...

I believe the answer so far has been "we don't do this because it's fairly
complicated and we haven't seen any use cases that require it." In the cases
I've seen so far there was either a misunderstanding of how zk worked, or a
simpler way available.

Does that make sense? Thoughts?

Patrick

Begin a discussion about ZooKeeper as a top level project

[Patrick Hunt]

You have probably heard by now that there is a discussion going on in
the Hadoop PMC as to whether a number of the subprojects (Hbase, Avro,
Zookeeper, Hive, and Pig) should move out from under the Hadoop
umbrella and become top level Apache projects (TLP). This discussion
has picked up recently since the Apache board has clearly communicated
to the Hadoop PMC that it is concerned that Hadoop is acting as an
umbrella project with many disjoint subprojects underneath it. They
are concerned that this gives Apache little insight into the health
and happenings of the subproject communities which in turn means
Apache cannot properly mentor those communities.

The purpose of this email is to start a discussion within the
ZooKeeper community about this topic. Let me cover first what becoming
TLP would mean for ZooKeeper, and then I'll go into what options I
think we as a community have.

Becoming a TLP would mean that ZooKeeper would itself have a PMC that
would report directly to the Apache board. Who would be on the PMC
would be something we as a community would need to decide. Common
options would be to say all active committers are on the PMC, or all
active committers who have been a committer for at least a year. We
would also need to elect a chair of the PMC. This lucky person would
have no additional power, but would have the additional responsibility
of writing quarterly reports on ZooKeeper's status for Apache board
meetings, as well as coordinating with Apache to get accounts for new
committers, etc. We currently submit these same reports, however they
are forwarded to the board through the Hadoop PMC Chair. For more
information see
http://www.apache.org/foundation/how-it-works.html#roles

Becoming a TLP would not mean that we are ostracized from the Hadoop
community. We would continue to be invited to Hadoop Summits, HUGs,
etc.

I see three ways that we as a community can respond to this:

1) Say yes, we want to be a TLP now.

2) Say yes, we want to be a TLP, but not yet. We feel we need more
time to mature. If we choose this option we need to be able to clearly
articulate how much time we need and what we hope to see change in
that time.

3) Say no, we feel the benefits for us staying with Hadoop outweigh
the drawbacks of being a disjoint subproject. If we choose this, we
need to be able to say exactly what those benefits are and why we feel
they will be compromised by leaving the Hadoop project.

There may other options that I haven't thought of. Please feel free to
suggest any you think of.

Here are the thoughts I've formed so far on the subject:

Benefits of moving to TLP:

a) Here's the boards view as communicated to me:

"we're looking to ensure that proper and effective oversight is
  reached, and umbrellas can get in the way of that. If you *also* think
  that all of your communities have proper oversight, and that you're
  communicating enough about each/all of them to the Board, so that *it*
  can provide oversight, then that's just fine. Go do the review and
  come back and say, "we're all good. no changes are necessary.""

b) setting our own course - we would have our own PMC and therefore
have more latitude (within the apache rules of course) in setting
direction. PMC members would be focused on ZooKeeper exclusively.


Serious reservations I personally have with a move to TLP today:

a) I do not think ZooKeeper currently has a sufficiently large and
diverse enough community such that it can fend for itself as a
TLP. Our community is working hard to establish a critical mass, given
our maturity level, complexity of code, and the stakes involved (ZK is
literally the linchpin of many of our user's computing
infrastructures) it has been hard to attract/promote developers. We
currently have 5 active committers, 4 from one company and 1 from
a separate one (who only recently joined the committer ranks). The
board has stated they are willing to break their own rules here (form
a TLP with less than acceptable diversity) however I don't believe that
would be prudent from our perspective.

b) Loss of branding and discover-ability - "in the land of the cloud
the elephant is king". IMO being associated with Hadoop is a huge win
for us in terms of branding and discover-ability. This is similar to
the benefits we get of being an Apache project. People who are serious
about the cloud need to look at Hadoop. In the process they discover
ZooKeeper.

c) "if ain't broke don't fix it". I have frequent interactions with
Hadoop PMC/Chair and an Apache board member. We are getting excellent
representation through this process and I don't see how visibility
"up" or support "down" could be improved.

Questions? Thoughts? Rebuttal? Let the discussion begin.

Patrick

Re: A deficiency? You can only add children to persistent nodes

[Patrick Hunt]

Excellent idea Jeff. Dominic feel free to open 2 JIRAs, one for ephem in 
ephem and a second for the weak watchers. Summarize your goals and the 
discussion so far as appropriate. BTW, either of these JIRAs would be 
good for a new contributor interested in gaining experience with 
ZooKeeper. ephem is a bit tougher, but not so extensive that it's 
insurmountable (of course our current dev base will help out).


Regards,

Patrick

Jeff Hammerbacher wrote:

Hey Ben,

Perhaps you should open a JIRA for further discussion?

Thanks,
Jeff

On Mon, Mar 22, 2010 at 7:42 PM, Benjamin Reed  wrote:


let me put out an idea that we have kicked around for a while: ephemeral
containers. the idea is that the znode disappears if it doesn't have
children. you would create the znode with create("/path", data, acl,
EPHEMERAL_CONTAINER) this would result in the creation two znodes: /path and
/path/child. (we have to create it with a child otherwise it immediately
disappears.)

i think this mechanism would address your need in a way that is easy to
implement and use. it would also allow you to do a cool barrier
implementation!

ben


On 03/22/2010 10:37 AM, Patrick Hunt wrote:


Dominic Williams wrote:



What I'd suggest might work:
- when the session that created the parent ends, ownership of the parent
could either be transferred to the owner/session that created the oldest
child, or instead ownership could be transferred to some kind of nominal
system session (which would delete the parent once the last ephemeral
child
disappeared)



There may be some issues with idempotency here, also it could require
extensive locking which drives up operation latencies (essentially
"recursive delete"). It sounds possible, but someone would have to take
a closer look as to the technical challenges involved.


Our general philosophy is to keep things as simple as possible wrt api,
semantics, implementation, etc... Distributed communication is hard and
while we handle a lot of the issues for you it's still complex.
Following our philosophy generally makes the easy things simple and the
hard things possible, additionally it reduces the number of bugs that we
have in the implementation itself (both user and service code).

I don't wish to discourage you as much as provide insight/background
into some of our decisions.

Regards,

Patrick




On 22 March 2010 16:44, Patrick Hunt  wrote:




Dominic Williams wrote:




1/ If a node crashes or something else goes wrong, you leave behind
persistent nodes. Over time these will grow and grow, rather like the
old
tmp folders used to fill with files under Windows




That's true. One either needs to use ephemerals or use persistent and
have
a "garbage collector" (implicit or explicit gc). In most cases it's
preferable to use the ephemeral.


 2/ Persistent nodes = nasty scalability *bottleneck* because you're



actually
having to write to disk somewhere.




This is not actually how ZK works. All znodes regardless of
persistent/ephemeral are written to disk persistently. Ephemeral nodes
are
tied to the session that created them. As long as the session is alive
the
ephemeral node is alive. Sessions themselves are persistently/reliably
stored by the ZK cluster. This allows the shutdown of the entire cluster
and
restart it, all sessions/ephemerals will be maintained. Sessions can
move
from server to server (if say network connectivity to server A fails, or
server A itself fails then the client will move to server B). The
session
and all ephemerals are maintained (well, as long as the client moves
withing
the expiration timeout value).


 To avoid this I'm actually thinking of writing locking system where you



work
out the existing chain not by enumerating sequential children, but by
looking at the contents of each temporary lock node to see what it is
waiting on. But... that's quite horrible. Was wondering whether there
is
some technical reason why you ephemeral nodes can't have children??




There are a few cases to think about.

1) obviously ephemeral nodes can't have persistent children, this just
doesn't make sense

2) ephemeral nodes have an owner - the session that created them. so it
would also not make sense (in my mind at least) to have an ephemeral
/foo
with another ephemeral /foo/bar with a different owner.

3) so you are left with "ephemerals can be a child of an ephemeral with
the
same owner".

4) there are also issues of order. in particular what is the "deletion
order" depth first or breadth first, etc...

I believe the answer so far has been "we don't do this because it's
fairly
complicated and we haven't seen any use cases that require it." In the
cases
I've seen so far there was either a misunderstanding of how zk worked,
or a
simpler way available.

Does that make sense? Thoughts?

Patrick

Re: A deficiency? You can only add children to persistent nodes

[Patrick Hunt]

Feel free to assign yourself to WW. I encourage you to document 
(comments in jira or wiki page) some rough approximation of the 
api/approach for WW, it's better to get the comments/concerns up front 
than after you took the time to work it all out in a patch. There are 
good tests for watchers already, you'll need to extend those for WW and 
add appropriate javadoc/forrest docs. Asking lots of questions is fine. :-)


http://wiki.apache.org/hadoop/ZooKeeper/HowToContribute

Patrick

Dominic Williams wrote:

Ok will do. Will create patch for weak watchers and corral existing
discussion into ephem in ephem.

This will happen some time this week

Thanks for all the feedback.

On 23 March 2010 16:29, Patrick Hunt  wrote:


Excellent idea Jeff. Dominic feel free to open 2 JIRAs, one for ephem in
ephem and a second for the weak watchers. Summarize your goals and the
discussion so far as appropriate. BTW, either of these JIRAs would be good
for a new contributor interested in gaining experience with ZooKeeper. ephem
is a bit tougher, but not so extensive that it's insurmountable (of course
our current dev base will help out).

Regards,

Patrick


Jeff Hammerbacher wrote:


Hey Ben,

Perhaps you should open a JIRA for further discussion?

Thanks,
Jeff

On Mon, Mar 22, 2010 at 7:42 PM, Benjamin Reed 
wrote:

 let me put out an idea that we have kicked around for a while: ephemeral

containers. the idea is that the znode disappears if it doesn't have
children. you would create the znode with create("/path", data, acl,
EPHEMERAL_CONTAINER) this would result in the creation two znodes: /path
and
/path/child. (we have to create it with a child otherwise it immediately
disappears.)

i think this mechanism would address your need in a way that is easy to
implement and use. it would also allow you to do a cool barrier
implementation!

ben


On 03/22/2010 10:37 AM, Patrick Hunt wrote:

 Dominic Williams wrote:


 What I'd suggest might work:

- when the session that created the parent ends, ownership of the
parent
could either be transferred to the owner/session that created the
oldest
child, or instead ownership could be transferred to some kind of
nominal
system session (which would delete the parent once the last ephemeral
child
disappeared)


 There may be some issues with idempotency here, also it could require

extensive locking which drives up operation latencies (essentially
"recursive delete"). It sounds possible, but someone would have to take
a closer look as to the technical challenges involved.


Our general philosophy is to keep things as simple as possible wrt api,
semantics, implementation, etc... Distributed communication is hard and
while we handle a lot of the issues for you it's still complex.
Following our philosophy generally makes the easy things simple and the
hard things possible, additionally it reduces the number of bugs that we
have in the implementation itself (both user and service code).

I don't wish to discourage you as much as provide insight/background
into some of our decisions.

Regards,

Patrick



 On 22 March 2010 16:44, Patrick Hunt  wrote:



 Dominic Williams wrote:



 1/ If a node crashes or something else goes wrong, you leave behind

persistent nodes. Over time these will grow and grow, rather like the
old
tmp folders used to fill with files under Windows



 That's true. One either needs to use ephemerals or use persistent

and
have
a "garbage collector" (implicit or explicit gc). In most cases it's
preferable to use the ephemeral.


 2/ Persistent nodes = nasty scalability *bottleneck* because you're


 actually

having to write to disk somewhere.



 This is not actually how ZK works. All znodes regardless of

persistent/ephemeral are written to disk persistently. Ephemeral nodes
are
tied to the session that created them. As long as the session is alive
the
ephemeral node is alive. Sessions themselves are persistently/reliably
stored by the ZK cluster. This allows the shutdown of the entire
cluster
and
restart it, all sessions/ephemerals will be maintained. Sessions can
move
from server to server (if say network connectivity to server A fails,
or
server A itself fails then the client will move to server B). The
session
and all ephemerals are maintained (well, as long as the client moves
withing
the expiration timeout value).


 To avoid this I'm actually thinking of writing locking system where
you


 work

out the existing chain not by enumerating sequential children, but by
looking at the contents of each temporary lock node to see what it is
waiting on. But... that's quite horrible. Was wondering whether there
is
some technical reason why you ephemeral nodes can't have children??



 There are a few cases to think about.

1) obviously ephemeral nodes can't have persistent children, this just
doesn't make sense

2) ephemeral nodes have an owner - the session that c

Re: A deficiency? You can only add children to persistent nodes

[Patrick Hunt]

A while back (3.1? basically long after the mac port of the 1.6 jvm was 
considered "stable") we stopped "officially" supporting 1.5 
http://bit.ly/9IlzUq


However there have been occasional requests from ppl trying to use 1.5 
and we do our best not to break things for them. Last I checked the code 
still compiled under 1.5 (but it's been a while). IMO it's best to use 
1.6 though, even if it compiles/runs for 1.5 there have been a number of 
significant bugs fixed in the jvm since then.


Dominic, is there some reason in particular in this case?

Patrick

Dominic Williams wrote:

Hi Patrick, re: Weak Watchers implementation. Is it ok to assume JDK 6?

On 23 March 2010 17:32, Patrick Hunt  wrote:


Feel free to assign yourself to WW. I encourage you to document (comments
in jira or wiki page) some rough approximation of the api/approach for WW,
it's better to get the comments/concerns up front than after you took the
time to work it all out in a patch. There are good tests for watchers
already, you'll need to extend those for WW and add appropriate
javadoc/forrest docs. Asking lots of questions is fine. :-)

http://wiki.apache.org/hadoop/ZooKeeper/HowToContribute

Patrick


Dominic Williams wrote:


Ok will do. Will create patch for weak watchers and corral existing
discussion into ephem in ephem.

This will happen some time this week

Thanks for all the feedback.

On 23 March 2010 16:29, Patrick Hunt  wrote:

 Excellent idea Jeff. Dominic feel free to open 2 JIRAs, one for ephem in

ephem and a second for the weak watchers. Summarize your goals and the
discussion so far as appropriate. BTW, either of these JIRAs would be
good
for a new contributor interested in gaining experience with ZooKeeper.
ephem
is a bit tougher, but not so extensive that it's insurmountable (of
course
our current dev base will help out).

Regards,

Patrick


Jeff Hammerbacher wrote:

 Hey Ben,

Perhaps you should open a JIRA for further discussion?

Thanks,
Jeff

On Mon, Mar 22, 2010 at 7:42 PM, Benjamin Reed 
wrote:

 let me put out an idea that we have kicked around for a while:
ephemeral


containers. the idea is that the znode disappears if it doesn't have
children. you would create the znode with create("/path", data, acl,
EPHEMERAL_CONTAINER) this would result in the creation two znodes:
/path
and
/path/child. (we have to create it with a child otherwise it
immediately
disappears.)

i think this mechanism would address your need in a way that is easy to
implement and use. it would also allow you to do a cool barrier
implementation!

ben


On 03/22/2010 10:37 AM, Patrick Hunt wrote:

 Dominic Williams wrote:


 What I'd suggest might work:


- when the session that created the parent ends, ownership of the
parent
could either be transferred to the owner/session that created the
oldest
child, or instead ownership could be transferred to some kind of
nominal
system session (which would delete the parent once the last ephemeral
child
disappeared)


 There may be some issues with idempotency here, also it could
require


extensive locking which drives up operation latencies (essentially
"recursive delete"). It sounds possible, but someone would have to
take
a closer look as to the technical challenges involved.


Our general philosophy is to keep things as simple as possible wrt
api,
semantics, implementation, etc... Distributed communication is hard
and
while we handle a lot of the issues for you it's still complex.
Following our philosophy generally makes the easy things simple and
the
hard things possible, additionally it reduces the number of bugs that
we
have in the implementation itself (both user and service code).

I don't wish to discourage you as much as provide insight/background
into some of our decisions.

Regards,

Patrick



 On 22 March 2010 16:44, Patrick Hunt  wrote:



 Dominic Williams wrote:



 1/ If a node crashes or something else goes wrong, you leave behind


persistent nodes. Over time these will grow and grow, rather like
the
old
tmp folders used to fill with files under Windows



 That's true. One either needs to use ephemerals or use persistent


and
have
a "garbage collector" (implicit or explicit gc). In most cases it's
preferable to use the ephemeral.


 2/ Persistent nodes = nasty scalability *bottleneck* because you're


 actually


having to write to disk somewhere.



 This is not actually how ZK works. All znodes regardless of


persistent/ephemeral are written to disk persistently. Ephemeral
nodes
are
tied to the session that created them. As long as the session is
alive
the
ephemeral node is alive. Sessions themselves are
persistently/reliably
stored by the ZK cluster. This allows the shutdown of the entire
cluster
and
restart it, all sessions/ephemerals will be maintained. Sessions can
move
from server to server (if say network connectivity to server A
fails,
or
server A itself

Re: [VOTE] Release ZooKeeper 3.3.0 (candidate 0)

[Patrick Hunt]

+1

In addition to the std stuff (unit tests etc...) I've run a number of 
deployment tests, including clusters of size 1/3/5/6/7/9/11/12/13 and 
they are all working fine.


Patrick

Patrick Hunt wrote:
I have created a candidate build for ZooKeeper 3.3.0. Over 180 JIRAs are 
addressed in this release.


*** Please download, test and VOTE before the
*** vote closes 1pm PT on Wednesday, March 24.***

http://people.apache.org/~phunt/zookeeper-3.3.0-candidate-0/

Should we release this?

Patrick

Re: ZooKeeper & dynamic discovery (apache river)

[Patrick Hunt]

Jonathan Reichhold wrote:

I was just expressing an opinion as someone who watches the list.  I'm not
trying to drive you toward any decision, but have watched how River has
stayed in incubation for 2 years.  I've used JINI on projects and was mostly
happy, but the bugs in it were never resolved last I looked.

I was attempting to ask what the policy of Zookeeper was in regards to
something that hasn't gotten out of incubation.


Sorry for the slow response but I wanted to make sure I understood the 
issues before commenting. Jonathan, I do appreciate you bringing up this 
issue, it's not something I had considered (incubation), although I did 
look at River and it's list activity before initially commenting.


According to the response that I got from the incubator community it's 
fine for TLPs to use incubator code, in particular this snippet:



I don't think this is a problem.


Incubator releases are approved by the Incubator PMC, 
and have had as much (or more) legal review as releases by other TLPs.


We would want the submitted code to depend on a released version of 
River, not code that's unreleased (not a river jar built from svn for 
example)


full details: http://bit.ly/bbqk4D

That said, I did have some concern regarding River's apparent lack of 
progress. Mature projects might have very infrequent releases (look at 
log4j, or Forrest), but it did seem a bit unusual that a community in 
incubation would have so infrequent a release cycle, at the very least 
one would expect fix releases! This was why I suggested submitting as a 
contrib. The functionality seems useful to our user base and having it 
available in contrib would be a good thing IMO.



Please don't take my comments as a statement on this project and what you
should do.  I have no commit privileges or management of the project, and
just wanted to ask the question.


I believe your comments were both justified and useful, keep them 
coming. :-)


Brian, not sure where you guys are with your thinking on this. Feel free 
to open a JIRA, collect further feedback, and submit a patch.


Regards,

Patrick



Jonathan

On Sun, Mar 21, 2010 at 9:38 AM, Brian Murphy wrote:


On Sun, Mar 21, 2010 at 12:37 AM, Jonathan Reichhold <
jonathan.reichh...@gmail.com> wrote:

Apache River is dying from lack of updates.


Hmm, I suppose "dying" is a matter of opinion, or one's
perspective.

If you're talking about the lack of an official release,
no argument there. But, for what it's worth, that might
be more a function of the many differing opinions
being voiced by the various parties interested in that
project; which has driven some away, but has been
viewed by others as healthy discourse.

For example, because the river codebase provides
an infrastructure rather than a specific application, and is
fairly mature and stable, some of the river meritocracy feel
that the project should move more slowly than application
based apache projects typically move, whereas others feel
just the opposite. I generally leave these sort of arguments
for others to worry about though. I'm usually more interested
in whether the code serves the needs of the project I'm on
(which in this case, both river and zookeeper do).

Fortunately, the project I'm currently on doesn't need any
major new features from river. The numerous patches
and updates that have been feeding into the pending 2.1.1
release (currently under vote) have been more than enough
to serve our needs, and have been put to quite good use.
But of course, your needs and experiences may be different.



not sure tying to a project which hasn't moved since 2008 is a
good idea for a contribution to Zookeeeper.


Okay, thanks for the honest answer, Jonathan. This is what
we were trying to find out by posing the original question to
the Zookeeper community. Since we don't want to be disruptive,
we'll simply continue developing the code in our own namespace.
No harm, no foul.

Thanks,
Brian

Re: [VOTE] Release ZooKeeper 3.3.0 (candidate 0)

[Patrick Hunt]

With 5 +1's (3 from PMC members) and no -1's the vote passes. 
Congratulations all!


I will work to finish the release process and send announcement emails 
when it is completed.


Regards,

Patrick

Stack wrote:

+1

All hbase tests pass with 3.3.0 in place.  I ran small loading and
nothing odd looking.  Looks like no issue having a zk 3.3.0 client
talk to a 3.2.2 ensemble.

Requires small mods to hbase other than dropping new zk jar into
hbase/lib in place of zk 3.2.2: HBASE-2380.

St.Ack

On Fri, Mar 19, 2010 at 12:43 PM, Patrick Hunt  wrote:

I have created a candidate build for ZooKeeper 3.3.0. Over 180 JIRAs are
addressed in this release.

*** Please download, test and VOTE before the
*** vote closes 1pm PT on Wednesday, March 24.***

http://people.apache.org/~phunt/zookeeper-3.3.0-candidate-0/

Should we release this?

Patrick

Re: [VOTE] Release ZooKeeper 3.3.0 (candidate 0)

[Patrick Hunt]

Stack, you can't use a new client with an old server. We support b/w 
compatibility at the server level (new server works with old client) but 
not the other way around. You would have to upgrade the server and 
client at the same time, or upgrade the servers (rolling upgrade) then 
upgrade the clients.


Patrick

Stack wrote:

+1

All hbase tests pass with 3.3.0 in place.  I ran small loading and
nothing odd looking.  Looks like no issue having a zk 3.3.0 client
talk to a 3.2.2 ensemble.

Requires small mods to hbase other than dropping new zk jar into
hbase/lib in place of zk 3.2.2: HBASE-2380.

St.Ack

On Fri, Mar 19, 2010 at 12:43 PM, Patrick Hunt  wrote:

I have created a candidate build for ZooKeeper 3.3.0. Over 180 JIRAs are
addressed in this release.

*** Please download, test and VOTE before the
*** vote closes 1pm PT on Wednesday, March 24.***

http://people.apache.org/~phunt/zookeeper-3.3.0-candidate-0/

Should we release this?

Patrick

Re: [ANNOUNCEMENT] Contributor Workshop at Yahoo!

[Patrick Hunt]

ZooKeeper dev team, new and old - I think this would be a great 
opportunity for us to meet and discuss ZooKeeper development. We could 
focus on ZK internals, future development, Q&A with team members.


What do you think? I'd be willing to coordinate our session unless 
someone else would like to do so. I'm thinking we don't need 5 hrs, how 
about 2 hours? (it's june 30th)


LMK if there's interest and what thoughts on what we might focus on.

Patrick

Owen O'Malley wrote:
Yahoo is organizing Contributor's Workshops on the day after the Hadoop 
Summit (10 June 2010) for both Hadoop Core (HDFS & MapReduce) and Pig. 
We would be happy to provide space for any of the other Hadoop 
sub-projects as well!  If you are interested in organizing such a 
workshop for one of the Hadoop sub-projects, please email us at 
hadoopcontributorr...@yahoo-inc.com with "WORKSHOP ORGANIZER (project)" 
in the subject line.


See you all at the Hadoop Summit – June 29th, http://www.hadoopsummit.org/

Thanks,
   Owen O'Malley & Eric Baldeschwieler

Re: [VOTE] Release ZooKeeper 3.3.0 (candidate 0)

[Patrick Hunt]

Some background on this: In order to add new features (sometimes to fix 
bugs) we need to change the client lib in a non-b/w compatible way, this 
is infrequent, but there's just no way around this in some cases. At the 
server level we always ensure (and even in extreme cases this might not 
be possible, but so far it has) that a new server can talk to an old (at 
least 1 version back) client. Additionally we also ensure that a new 
server can talk to an old server, this allows "rolling upgrade" of the 
ensemble. WRT this approach I'm probably not telling you anything you 
don't already know from your own/prior projects.


In zk THIS IS CRITICAL to our primary project level goals of "high 
availability". It would be laughable if we bill ourselves as highly 
available but "sorry, you need to shut everything down then upgrade 
everything to a new version". That's just not acceptable.


The roadmap has some detail on this, but it's out of date from our 
current practices. We also need to include this information in our 
release notes.

http://wiki.apache.org/hadoop/ZooKeeper/Roadmap
https://issues.apache.org/jira/browse/ZOOKEEPER-727

Stack, in your specific case you are seeing that 3.3 client works fine 
with 3.2 server. In 3.3 we added a new API method to the client, which 
sends a new message type to the server. As long as you don't use this 
method (getchildren2) it will probably work fine. However, we don't 
officially support this configuration as we don't design for this case 
(the changes) and we don't test for this. It may be that there was some 
semantic change at the protocol (client->server protocol) level, that 
may not be exposed except in unusual cases. Perhaps if we had more 
resources we could verify this case (3.3C with 3.2S) but today we do 
not, so essentially it would be "use at your own risk".


Hope this helps. If you have further insights, esp wrt HBase using ZK 
please feel free to comment.


Regards,

Patrick

Stack wrote:

Patrick just let me know that newer client talking to older server is
not supported.  I didn't know that.  Thanks for pointing it out.  Was
sort of surprised it worked at all so just noted this aspect of my zk
3.3.0 RC0 eval.

Congrats on new release lads,
St.Ack

P.S. Below is backup of my assertion 3.3.0 client basically works
against 3.2.2 ensemble:


In client log I see this:

2010-03-25 12:23:57,670 INFO org.apache.zookeeper.ZooKeeper: Client
environment:zookeeper.version=3.3.0-925362, built on 03/19/2010 18:38
GMT

...

Then this:

2010-03-25 12:23:57,672 INFO org.apache.zookeeper.ZooKeeper:
Initiating client connection,
connectString=sv2borg165:2181,sv2borg166:2181,sv2borg167:2181,sv2borg169:2181,sv2borg164:2181
sessionTimeout=6 watcher=Thread[Thread-0,5,main]
2010-03-25 12:23:57,683 INFO org.apache.zookeeper.ClientCnxn: Opening
socket connection to server sv2borg166/10.20.20.166:2181
2010-03-25 12:23:57,686 INFO org.apache.zookeeper.ClientCnxn: Socket
connection established to sv2borg166/10.20.20.166:2181, initiating
session

..

Over on 166 I see...

2010-03-25 12:23:57,697 INFO
org.apache.zookeeper.server.NIOServerCnxn: Connected to
/10.20.20.185:46331 lastZxid 0
2010-03-25 12:23:57,725 INFO
org.apache.zookeeper.server.NIOServerCnxn: Creating new session
0x3266d5140d70759
2010-03-25 12:23:57,726 INFO
org.apache.zookeeper.server.NIOServerCnxn: Finished init of
0x3266d5140d70759 valid:true
2010-03-25 12:25:07,305 INFO
org.apache.zookeeper.server.NIOServerCnxn: Processing stat command
from /10.20.20.185:60661
2010-03-25 12:25:07,306 WARN
org.apache.zookeeper.server.NIOServerCnxn: Exception causing close of
session 0x0 due to java.io.IOException: Responded to info probe
2010-03-25 12:25:07,306 INFO
org.apache.zookeeper.server.NIOServerCnxn: closing session:0x0
NIOServerCnxn: java.nio.channels.SocketChannel[connected
local=/10.20.20.166:2181 remote=/10.20.20.185:60661]

...

If I do stat over there I see Zookeeper version: 3.2.2-888565, built
on 12/08/2009 21:51 GMT...


On Thu, Mar 25, 2010 at 2:00 PM, Patrick Hunt  wrote:

Stack, you can't use a new client with an old server. We support b/w
compatibility at the server level (new server works with old client) but not
the other way around. You would have to upgrade the server and client at the
same time, or upgrade the servers (rolling upgrade) then upgrade the
clients.

Patrick

Stack wrote:

+1

All hbase tests pass with 3.3.0 in place.  I ran small loading and
nothing odd looking.  Looks like no issue having a zk 3.3.0 client
talk to a 3.2.2 ensemble.

Requires small mods to hbase other than dropping new zk jar into
hbase/lib in place of zk 3.2.2: HBASE-2380.

St.Ack

On Fri, Mar 19, 2010 at 12:43 PM, Patrick Hunt  wrote:

I have created a candidate build for ZooKeeper 3.3.0. Over 180 JIRAs are
addressed in this release.

*** Please download, test and VOTE before the
*** vote closes 1pm PT on Wednesday, 

[ANNOUNCE] Apache ZooKeeper 3.3.0

[Patrick Hunt]

The Apache ZooKeeper team is proud to announce Apache ZooKeeper version 
3.3.0.


ZooKeeper is a high-performance coordination service for distributed 
applications. It exposes common services - such as naming, configuration 
management, synchronization, and group services - in a simple interface 
so you don't have to write them from scratch. You can use it 
off-the-shelf to implement consensus, group management, leader election, 
and presence protocols. And you can build on it for your own, specific 
needs.


Key features of the 3.3.0 release:
* observers - non-voting members of the ensemble, scale reads 
http://bit.ly/9wPnpq

* distributed queue recipe implementation (c/java)
* additional 4letterword/jmx features to support operations
* maven repository support
* build support for eclipse project generation
* ivy based build

Changes in contrib:
* zookeeper-tree: export/import znode namespace
* zooinspector: gui browser for znode namespace
* bookkeeper client rewrite, use of netty

For ZooKeeper release details and downloads, visit:
http://hadoop.apache.org/zookeeper/releases.html

ZooKeeper 3.3.0 Release Notes are at:
http://hadoop.apache.org/zookeeper/docs/r3.3.0/releasenotes.html

Regards,

The ZooKeeper Team

Re: Solitication for logging/debugging requirements

[Patrick Hunt]

Take a look at the logging page in the docs:
http://hadoop.apache.org/zookeeper/docs/current/zookeeperInternals.html#sc_logging

Some good guidelines in there. Basically we log things at info level 
that are interesting/informational but not logged so frequently that 
they fill the log. WARN is for things that are bad but that we can 
handle (like network connectivity failure). ERROR is generally for 
things we don't expect and are unlikely we can handle. FATAL means 
really bad, we shutdown the server. Many end users log only at WARN 
level or higher in production, so typically we err on the side of WARN 
for issues (so that we have a shot at debugging after the fact). Over 
time, as we gain confidence in production environments, we've been 
pushing more things that were WARN down to INFO.


I fixed a number of JIRAs for 3.3 related to logging. In particular I 
cleaned up the client session logging significantly. The most fertile 
area right now to cleanup logging is in the quorum code. That code in 
particular has issues wrt providing sufficient information to debug 
error conditions. You can easily see this by starting an ensemble of 
greater than 1 machine and try killing one/more of the servers. There 
are many places where the logging is insufficient (eg. "got vote", which 
doesn't say what the vote was or what the effect of such a vote is, 
etc...) Having improved logging in this area would really help.


Try searching on the JIRA
https://issues.apache.org/jira/browse/ZOOKEEPER
of open/closed issues re "log4j" or "logging" or "log" for further insight.

I'd also suggest you move this thread to zookeeper-dev mailing list, 
that's the proper forum for development discussion.


Regards,

Patrick

Benjamin Reed wrote:
awesome! that would be great ivan. i'm sure pat has some more concrete 
suggestions, but one simple thing to do is to run the unit tests and 
look at the log messages that get output. there are a couple of 
categories of things that need to be fixed (this is in no way exhaustive):


1) messages that have useful information, but only if you look in the 
code to figure out what it means. there are some leader election 
messages that fall into this category. it would be nice to clarify them.
2) there are error messages that really aren't errors. when shutting 
down there are a bunch of errors that are expected, but still logged, 
for example.

3) misclassified error levels

welcome aboard!

ben

On 03/29/2010 10:07 AM, Ivan Kelly wrote:

Hi,

Im going to be using Zookeeper quite extensively for a project in a
few weeks, but development hasn't kicked off yet. This means I have
some time on my hands and I'd like to get familiar with zookeeper
beforehand by perhaps writing some tools to make debugging problems
with it easier so as to save myself some time in the future. Problem
is I haven't had to debug many zookeeper problems yet, so I don't know
where the pain points are.

So, without further ado,
- Are there any places that logging is deficient that sorely needs
improvement?
- Could current logs be improved any amount or presented in a more
readable fashion?
- Would some form of log visualisation be useful (for example in
something approximating a sequence diagram)?

Feel free to suggest anything which the list above doesn't allude to
which you think would be helpful.

Cheers,
Ivan

   

Monitoring ZooKeeper in 3.3.0

[Patrick Hunt]

I've posted an overview of ZooKeeper 3.3.0's low level monitoring 
commands that you might find useful. A number were added in this release 
in response to user requests: http://bit.ly/940Ag1


Regards,

Patrick

ZooKeeper Roadmap - 3.4.0 and beyond

[Patrick Hunt]

It's that time again, thinking about the next release of ZK. From what 
I'm hearing people seem interested in improving client support 
(ZOOKEEPER-22 missed 3.3) and reducing some technical debt.


In particular I believe we should look at the following going forward. 
Of course ZooKeeper is open to submissions in that aren't on this list. 
If you have any suggestions please feel free to enter a JIRA, submit a 
patch, or comment on this thread.

http://wiki.apache.org/hadoop/ZooKeeper/HowToContribute

3.4 (~3 months out)

* ZOOKEEPER-22
* Netty - allow encryption btw servers (potentially clients too)
* Testing
  * mockito
  * split out client tests, allow running against mocks as well as 
various versions of server

  * improve C client testing
* test container support
* split out zab? (might be useful as a library for third parties)


Feel free to respond to this email with any thoughts, suggestions, etc...

Regards,

Patrick

Re: per Roadmap - 3.4.0 and beyond

[Patrick Hunt]

That would be great. src/contrib holds the existing language bindings, 
please add it there. Feel free to create a JIRA and attach a patch:


http://wiki.apache.org/hadoop/ZooKeeper/HowToContribute

Patrick

Maarten Koopmans wrote:

I have a Scala wrapper around the Java class for sync communications which 
works pretty well in my project.

Wit a bit of refactoring that could be added if people are interested. Should 
be fairly easy to add the async part, but I had no need or that.

--Maarten

Op 30 mrt 2010, om 01:20 heeft Patrick Hunt het volgende geschreven:


It's that time again, thinking about the next release of ZK. From what I'm 
hearing people seem interested in improving client support (ZOOKEEPER-22 missed 
3.3) and reducing some technical debt.

In particular I believe we should look at the following going forward. Of 
course ZooKeeper is open to submissions in that aren't on this list. If you 
have any suggestions please feel free to enter a JIRA, submit a patch, or 
comment on this thread.
http://wiki.apache.org/hadoop/ZooKeeper/HowToContribute

3.4 (~3 months out)

* ZOOKEEPER-22
* Netty - allow encryption btw servers (potentially clients too)
* Testing
 * mockito
 * split out client tests, allow running against mocks as well as various 
versions of server
 * improve C client testing
* test container support
* split out zab? (might be useful as a library for third parties)


Feel free to respond to this email with any thoughts, suggestions, etc...

Regards,

Patrick

Re: ZooKeeper Roadmap - 3.4.0 and beyond

[Patrick Hunt]

Gustavo Niemeyer wrote:

Hey Patrick,


* ZOOKEEPER-22
* Netty - allow encryption btw servers (potentially clients too)


Yes! Yes! Both server and client, please!  As we discussed, we want to
use ZooKeeper as a coordination service within a network which
shouldn't be *entirely* trusted, and this is a critical feature for
this reason.


I believe Ben's worked out most of the server-server comm changes, but 
there are still many rough edges I planned to take up and work out. We'd 
need ppl to test it. If it's something you'd want to use it would be in 
your best interest to do so prior to the release. Client-Server support 
is a different matter, we'd have to see how the server works out first 
re release timing.



* Testing
 * mockito
 * split out client tests, allow running against mocks as well as various
versions of server
 * improve C client testing
* test container support
* split out zab? (might be useful as a library for third parties)


What is zab?


zab is the atomic broadcast used by the servers to communicate changes.


Feel free to respond to this email with any thoughts, suggestions, etc...


Do you think there's any chance that the ephemeral parents that were
mentioned in another thread could see some attention within this
release?



Depends if someone is interested enough to work on it. ;-) It's an 
interesting project that's not too difficult however there hasn't been 
much deep thinking on it yet to uncover all the potential difficulties.


Patrick

Re: ZooKeeper Roadmap - 3.4.0 and beyond

[Patrick Hunt]

Gustavo Niemeyer wrote:

* Netty - allow encryption btw servers (potentially clients too)

Yes! Yes! Both server and client, please!  As we discussed, we want to
use ZooKeeper as a coordination service within a network which
shouldn't be *entirely* trusted, and this is a critical feature for
this reason.

I believe Ben's worked out most of the server-server comm changes, but there
are still many rough edges I planned to take up and work out. We'd need ppl
to test it. If it's something you'd want to use it would be in your best
interest to do so prior to the release. Client-Server support is a different
matter, we'd have to see how the server works out first re release timing.


Ok, understood.  Do you think the implementation of server-client
support would be relatively straightforward once server-server is in
place?


Ben just mentioned to me that he's done the client side, not the server, 
and that the server should be easier than the client. We'll see. ;-) I'm 
trying to get him to submit a patch to JIRA now.


Patrick


Depends if someone is interested enough to work on it. ;-) It's an
interesting project that's not too difficult however there hasn't been much
deep thinking on it yet to uncover all the potential difficulties.


I am interested, and depending on how things go, I do think this
feature would help the design of the system.  I'm pretty sure I won't
be able to put my own time on it, though.  My best bet is for someone
else from the community to take it over, or for us to find someone
which would be able to contribute more directly.

Re: QA effort to support windows as a production platform

[Patrick Hunt]

Vishal K wrote:

We will be using zookeeper quite extensively for clustering. Windows is one
of the platforms that we may need to support. Since Win32 is not supported
as a production platform I was wondering to what extent is zookeeper tested
on windows. We are also interested on using Zookeeper on Win64 platform. Is
Win64 supported? Are there any plans to support Win32/Win64 for production?
If not, what would one need to do support windows and what would be the
estimated QA effort?


My use of ZK is exclusively 32/64bit linux, however I can tell you that 
given that the client/server are implemented in java it should work. 
Problems you might encounter would be things like NIO issues with the 
JVM implementation on windows.


Testing on windows? Pretty much 0 afaik. We do support development on 
cygwin, so provides some basic exercising of the codepaths with the 
windows jvm, however it's not likely production level qa.


This question (zk on win) has come up once or twice before, I haven't 
seen any followup from the users who asked about it previously though.


3.3.0 has batch files for running the server in windows, give those a 
try. Probably what you'd want to do is run "ant test-core-java" or 
similar in the top of the ZK release directory. This will run all the 
java tests and give you some insight into status. I'd be happy to work 
with you to land patches that address issues with ZK on windows. 
Depending on the interest level and support from win users we could 
support win as a dev/prod platform at some point in the future - having 
ongoing support for this would be important though (people interested in 
testing/fixing under win I mean). Try exercising under windows and 
create some JIRAs based on what you find.


Regards,

Patrick

Re: Blog ECF Discovery: How many zookeepers does it take to screw in a lightbulb

[Patrick Hunt]

Smart lightbulbs and ZK, whoda thunk it. :-) Great post, thanks.

Patrick

Wim Jongman wrote:

Hi Guys,

Ahmed Aadel who works at Remain Software created a Zookeeper based Discovery
provider which can be used to publish remote services in an OSGi container.

I have created a blog posting here: http://tiny.cc/zookeepers

Regards,

Wim Jongman
MP @ Remain Software

Re: QA effort to support windows as a production platform

[Patrick Hunt]

Are you using 3.3.0? 3.3.0 included a number of fixes for cygwin and 
includes windows specific batch files. If you are planning to deploy to 
production on windows I'd encourage you to develop under windows 
directly as well.


If you find issues, bugs, etc... be sure to enter JIRAs. Don't worry, 
you won't hurt our feelings, on the contrary we'll be happy if you 
find/fix issues on windows and make things better for everyone. (just 
make sure you are using the latest release).


Regards,

Patrick


On 04/02/2010 07:05 AM, Vishal K wrote:

Hi,

I was able to start zookeeper on windows using cygwin. I had to do minor
changes to the shell scripts to use cygpath wherever needed.
I will run a few tests and post the progress. I greped through the zookeeper
sources just to check if ZK is using any native code.
I didn't find any, but just to confirm - is ZK using native code.

I have talked to a few guys around and they said it is fair to assume that
the programs are portable (to Windows) if they don't have native code. Just
wanted to check. Thanks.
On Thu, Apr 1, 2010 at 10:09 AM, Vishal K  wrote:


Hi Patrick,

Thanks for your response. I start running ZK on windows and let you know if
I run into issues.


On Wed, Mar 31, 2010 at 11:32 AM, Patrick Hunt  wrote:



Vishal K wrote:


We will be using zookeeper quite extensively for clustering. Windows is
one
of the platforms that we may need to support. Since Win32 is not
supported
as a production platform I was wondering to what extent is zookeeper
tested
on windows. We are also interested on using Zookeeper on Win64 platform.
Is
Win64 supported? Are there any plans to support Win32/Win64 for
production?
If not, what would one need to do support windows and what would be the
estimated QA effort?



My use of ZK is exclusively 32/64bit linux, however I can tell you that
given that the client/server are implemented in java it should work.
Problems you might encounter would be things like NIO issues with the JVM
implementation on windows.

Testing on windows? Pretty much 0 afaik. We do support development on
cygwin, so provides some basic exercising of the codepaths with the windows
jvm, however it's not likely production level qa.

This question (zk on win) has come up once or twice before, I haven't seen
any followup from the users who asked about it previously though.

3.3.0 has batch files for running the server in windows, give those a try.
Probably what you'd want to do is run "ant test-core-java" or similar in the
top of the ZK release directory. This will run all the java tests and give
you some insight into status. I'd be happy to work with you to land patches
that address issues with ZK on windows. Depending on the interest level and
support from win users we could support win as a dev/prod platform at some
point in the future - having ongoing support for this would be important
though (people interested in testing/fixing under win I mean). Try
exercising under windows and create some JIRAs based on what you find.

Regards,

Patrick

Re: Zookeeper Contributor Workshop

[Patrick Hunt]

I was thinking that we could have a few presentations on some 
interesting internal implementation detail that would be educational for 
contributors. The guts of the server in particular which can be 
difficult to grok w/o support from experienced developers.


After that we could have some Q/A or general discussion about where 
things are going with the project.


If you have any specific ideas, things you'd like to see or something 
you'd like to present let me know. Keep in mind the audience for all 
this is ZK contributors.


Patrick

On 04/01/2010 10:13 AM, Owen O'Malley wrote:

Hi all,
Patrick Hunt will be organizing the Zookeeper Contributor Workshop on
6/30. Please send email to him if you'd like propose a topic.

-- Owen

Re: QA effort to support windows as a production platform

[Patrick Hunt]

Pretty seamless, just do a rolling upgrade (see the faq) of the servers. 
Then upgrade your clients. Code APIs on the client are all b/w compat.


Patrick

On 04/02/2010 10:41 AM, Vishal K wrote:

Hi Patrick,

We have not upgraded to 3.3.0 yet. We are using 3.2.2. I did notice the
windows specific batch files in 3.3.0. How seemless is upgrade from 3.2.2 -

3.3.0?


I will let you know if I run into any windows related problems. Thanks.

On Fri, Apr 2, 2010 at 11:40 AM, Patrick Hunt  wrote:


Are you using 3.3.0? 3.3.0 included a number of fixes for cygwin and
includes windows specific batch files. If you are planning to deploy to
production on windows I'd encourage you to develop under windows directly as
well.

If you find issues, bugs, etc... be sure to enter JIRAs. Don't worry, you
won't hurt our feelings, on the contrary we'll be happy if you find/fix
issues on windows and make things better for everyone. (just make sure you
are using the latest release).

Regards,

Patrick



On 04/02/2010 07:05 AM, Vishal K wrote:


Hi,

I was able to start zookeeper on windows using cygwin. I had to do minor
changes to the shell scripts to use cygpath wherever needed.
I will run a few tests and post the progress. I greped through the
zookeeper
sources just to check if ZK is using any native code.
I didn't find any, but just to confirm - is ZK using native code.

I have talked to a few guys around and they said it is fair to assume that
the programs are portable (to Windows) if they don't have native code.
Just
wanted to check. Thanks.
On Thu, Apr 1, 2010 at 10:09 AM, Vishal K   wrote:

Hi Patrick,


Thanks for your response. I start running ZK on windows and let you know
if
I run into issues.


On Wed, Mar 31, 2010 at 11:32 AM, Patrick Hunt   wrote:



Vishal K wrote:

We will be using zookeeper quite extensively for clustering. Windows is

one
of the platforms that we may need to support. Since Win32 is not
supported
as a production platform I was wondering to what extent is zookeeper
tested
on windows. We are also interested on using Zookeeper on Win64
platform.
Is
Win64 supported? Are there any plans to support Win32/Win64 for
production?
If not, what would one need to do support windows and what would be the
estimated QA effort?



My use of ZK is exclusively 32/64bit linux, however I can tell you that
given that the client/server are implemented in java it should work.
Problems you might encounter would be things like NIO issues with the
JVM
implementation on windows.

Testing on windows? Pretty much 0 afaik. We do support development on
cygwin, so provides some basic exercising of the codepaths with the
windows
jvm, however it's not likely production level qa.

This question (zk on win) has come up once or twice before, I haven't
seen
any followup from the users who asked about it previously though.

3.3.0 has batch files for running the server in windows, give those a
try.
Probably what you'd want to do is run "ant test-core-java" or similar in
the
top of the ZK release directory. This will run all the java tests and
give
you some insight into status. I'd be happy to work with you to land
patches
that address issues with ZK on windows. Depending on the interest level
and
support from win users we could support win as a dev/prod platform at
some
point in the future - having ongoing support for this would be important
though (people interested in testing/fixing under win I mean). Try
exercising under windows and create some JIRAs based on what you find.

Regards,

Patrick

Re: Zookeeper Contributor Workshop

[Patrick Hunt]

I've never done one so I don't know. I don't see why we couldn't setup a 
phone conference though. Might be hard to see the whiteboard, etc... . 
;-) I've noted it down to try and make this happen.


Thanks!

Patrick

On 04/02/2010 11:11 AM, Vishal K wrote:

Hi Patrick,

This will be certainly very interesting. Since I haven't been to these
events in the past. I assume one can participate (at least listen) from
remote locations.

-Vishal

On Fri, Apr 2, 2010 at 11:45 AM, Patrick Hunt  wrote:


I was thinking that we could have a few presentations on some interesting
internal implementation detail that would be educational for contributors.
The guts of the server in particular which can be difficult to grok w/o
support from experienced developers.

After that we could have some Q/A or general discussion about where things
are going with the project.

If you have any specific ideas, things you'd like to see or something you'd
like to present let me know. Keep in mind the audience for all this is ZK
contributors.

Patrick


On 04/01/2010 10:13 AM, Owen O'Malley wrote:


Hi all,
Patrick Hunt will be organizing the Zookeeper Contributor Workshop on
6/30. Please send email to him if you'd like propose a topic.

-- Owen

FYI, HBase jira for adding ZK auth.

[Patrick Hunt]

You might want to watch this one:

https://issues.apache.org/jira/browse/HBASE-2418

Patrick

Re: QA effort to support windows as a production platform

[Patrick Hunt]

On 04/07/2010 01:51 PM, Vishal K wrote:

Hi Mahadev,

Thanks for your response. Currently I am running ZK without cygwin on
windows. I will give it a try on cygwin. I am not quite familiar with
cppunit. Why will cppinit give me more confidence in native windows
libraries?

I have few more questions relevant to testing:
1. How much code coverage do we get with "test-core-java"? I see 68.8%
coverage on hudson
http://hudson.zones.apache.org/hudson/job/ZooKeeper-trunk/clover/.  Does
this measure the coverage from junit tests run from "test-core-java"?



Right, afaik that's test-core-java.

I think if you plan to run in production under windows you would want to 
run in win, not cygwin. If that's the case you'd have to port the c 
client and c client tests to some windows specific compiler.



2. What would be a good and reliable set of tests that will help me verify
that the cluster holding up fine on windows. I tried to run systest and
generateload. But I am having issues with the tests (and also understanding
the output of the tests since I am not familiar with the source). The
systest did exit with output something ilke Test OK (1). I presume this is a
good sign :-) generateLoad crashed and I will look into it later. Please let
me know if you have any suggestions. Last few lines are shown below:



Well the best thing would be to run one's own tests for their specific 
use cases. Short of that you've done the core set of tests that we have 
available to us. You might try the smoke/latency tests:

http://github.com/phunt/zk-smoketest
in particular you can run the latency tests from a number of clients in 
parallel and see the results.

http://wiki.apache.org/hadoop/ZooKeeper/ServiceLatencyOverview

Patrick



2010-04-07 14:44:20,055 - WARN  [QuorumPeer:/0.0.0.0:3155:quorump...@662] -
QuorumPeer main thread exited
Got rc = -4
Got rc = -4
[many such messages as above]
WatchedEvent state:Disconnected type:None path:null
java.lang.InterruptedException: sleep interrupted
 at java.lang.Thread.sleep(Native Method)
 at
org.apache.zookeeper.test.system.GenerateLoad$GeneratorInstance$SenderThread.run(GenerateLoad.java:425)
java.lang.InterruptedException
 at java.lang.Object.wait(Native Method)
 at java.lang.Object.wait(Object.java:485)
 at
org.apache.zookeeper.test.system.GenerateLoad$GeneratorInstance$ZooKeeperThread.incOutstanding(GenerateLoad.java:305)
 at
org.apache.zookeeper.test.system.GenerateLoad$GeneratorInstance$ZooKeeperThread.run(GenerateLoad.java:353)

2010-04-07 14:44:20,711 - INFO
[Thread-42-SendThread(vkher-devd:3155):clientcnxn$sendthr...@1000] - Opening
socket connection to server/:48214
Got rc = -4
2010-04-07 14:44:22,008 - INFO  [Thread-42:zookee...@538] -
Session:0x127d97600cc closed

Thanks.

Regards,
-Vishal


On Wed, Apr 7, 2010 at 3:55 PM, Mahadev Konar  wrote:


HI Vishal,
  I would be a good think to actually get cppunit working on windows (rather
than dropping it) since it would make you more confident on being able to
use the native libraries for windows.

Though there is already an open jira to try and compile librarires without
CPPIUNIT being installed on the machines.

http://issues.apache.org/jira/browse/ZOOKEEPER-316


Would you want to try and take a shot at fixing the cppunit tests?

Would be great to have cppunit tests working on cygwin!


Thanks
mahadev


On 4/6/10 4:15 PM, "Vishal K"  wrote:


Hi,

I had few minor problems (
https://issues.apache.org/jira/browse/ZOOKEEPER-734) after which all

junits

passed (I ran ant test-java-core).

But the build failed later in create-cppunit-configure:

-
test-core-java:
call-test-cppunit:
init:
check-cppunit-makefile:
create-cppunit-makefile:
init:
check-cppunit-configure:
create-cppunit-configure:
 [mkdir] Created dir:
C:\zookeeper\zookeeper-3.3.0\build\test\test-cppunit
BUILD FAILED
C:\zookeeper\zookeeper-3.3.0\build.xml:907: The following error occurred
while executing this line:
C:\zookeeper\zookeeper-3.3.0\build.xml:865: The following error occurred
while executing this line:
C:\zookeeper\zookeeper-3.3.0\build.xml:857: Execute failed:
java.io.IOException: Cannot run program
"C:\zookeeper\zookeeper-3.3.0\src\c\configure" (in directory
"C:\zookeeper\zookeeper-3.3.0\build\test\test-cppunit"): CreateProcess
error=193, %1 is not a valid Win32 application
  at java.lang.ProcessBuilder.start(ProcessBuilder.java:459)
  at java.lang.Runtime.exec(Runtime.java:593
---
cppunit tests should be probably dropped for windows.

Also, one point to note the dataDir in zoo.cfg should have UNIX path (or
double backslash instead of single backslash). I suppose all Java

developers

might be already aware of that.

Regards,
-Vishal

On Fri, Apr 2, 2010 at 1:46 PM, Patrick Hunt  wrote:


Pretty seamless, just do a rolling upgrade (see the faq) of the servers.
Then upgrade your clients. Code APIs on the c

Re: QA effort to support windows as a production platform

[Patrick Hunt]

I can't tell for certain. Looking a bit closer the code that starts the 
server for the c tests does reference the clover jar, so my guess would 
be that it does, but the messages from clover and the build, etc.. don't 
shed any light. I didn't setup the zk clover stuff for hudson, cc'ing 
Giri who might have insight.


Patrick

On 04/07/2010 02:34 PM, Vishal K wrote:

Hi Patrick,
We are not using C clients so we are not worried about porting them to
Windows. Just out of curiosity, is there any way to confirm if the code
coverage results are a result of test-core-java. If not, I will run the
coverage tools locally.
I had tried the smoketests earlier, but I ran into some issues there. I
forgot to note down the problem. I will revisit and upload the results.
Thanks for your help.

On Wed, Apr 7, 2010 at 5:11 PM, Patrick Hunt mailto:ph...@apache.org>> wrote:


On 04/07/2010 01:51 PM, Vishal K wrote:

Hi Mahadev,

Thanks for your response. Currently I am running ZK without
cygwin on
windows. I will give it a try on cygwin. I am not quite familiar
with
cppunit. Why will cppinit give me more confidence in native windows
libraries?

I have few more questions relevant to testing:
1. How much code coverage do we get with "test-core-java"? I see
68.8%
coverage on hudson
http://hudson.zones.apache.org/hudson/job/ZooKeeper-trunk/clover/.
  Does
this measure the coverage from junit tests run from
"test-core-java"?


Right, afaik that's test-core-java.

I think if you plan to run in production under windows you would
want to run in win, not cygwin. If that's the case you'd have to
port the c client and c client tests to some windows specific compiler.


2. What would be a good and reliable set of tests that will help
me verify
that the cluster holding up fine on windows. I tried to run
systest and
generateload. But I am having issues with the tests (and also
understanding
the output of the tests since I am not familiar with the
source). The
systest did exit with output something ilke Test OK (1). I
presume this is a
good sign :-) generateLoad crashed and I will look into it
later. Please let
me know if you have any suggestions. Last few lines are shown below:


Well the best thing would be to run one's own tests for their
specific use cases. Short of that you've done the core set of tests
that we have available to us. You might try the smoke/latency tests:
http://github.com/phunt/zk-smoketest
in particular you can run the latency tests from a number of clients
in parallel and see the results.
http://wiki.apache.org/hadoop/ZooKeeper/ServiceLatencyOverview

Patrick



2010-04-07 14:44:20,055 - WARN
  [QuorumPeer:/0.0.0.0:3155:quorump...@662] -
QuorumPeer main thread exited
Got rc = -4
Got rc = -4
[many such messages as above]
WatchedEvent state:Disconnected type:None path:null
java.lang.InterruptedException: sleep interrupted
 at java.lang.Thread.sleep(Native Method)
 at

org.apache.zookeeper.test.system.GenerateLoad$GeneratorInstance$SenderThread.run(GenerateLoad.java:425)
java.lang.InterruptedException
 at java.lang.Object.wait(Native Method)
 at java.lang.Object.wait(Object.java:485)
 at

org.apache.zookeeper.test.system.GenerateLoad$GeneratorInstance$ZooKeeperThread.incOutstanding(GenerateLoad.java:305)
 at

org.apache.zookeeper.test.system.GenerateLoad$GeneratorInstance$ZooKeeperThread.run(GenerateLoad.java:353)

2010-04-07 14:44:20,711 - INFO
[Thread-42-SendThread(vkher-devd:3155):clientcnxn$sendthr...@1000]
- Opening
socket connection to server/:48214
Got rc = -4
2010-04-07 14:44:22,008 - INFO  [Thread-42:zookee...@538] -
Session:0x127d97600cc closed

Thanks.

Regards,
-Vishal


On Wed, Apr 7, 2010 at 3:55 PM, Mahadev
Konarmailto:maha...@yahoo-inc.com>>  wrote:

HI Vishal,
  I would be a good think to actually get cppunit working on
windows (rather
than dropping it) since it would make you more confident on
being able to
use the native libraries for windows.

Though there is already an open jira to try and compile
librarires without
CPPIUNIT being installed on the machines.

http://issues.apache.org/jira/browse/ZOOKEEPER-316


Would you want to try and take a shot at fixing the cppunit
tests?

Would be great to h

Re: QA effort to support windows as a production platform

[Patrick Hunt]

There's probably some timing difference in the jvm or networking that's 
causing this to show up in windows vs unix. We shutdown the servers as 
part of the tests, as each server (quorum test) shuts down connectivity 
btw the servers will be lost, that's what this is showing.


Patrick

On 04/08/2010 03:21 PM, Vishal K wrote:

Hi Patrick, Mahadev,
I was going through the logs of junit tests (all of the tests passed). I
noticed the logs for windows are different from the logs on linux.
On windows, the logs show following "Failed to send last message" error
message several times. None of the logs on Linux have them.
Are these messages normal? I have also attached the log file for the
AsyncHammerTest.
2010-04-08 18:07:35,750 - WARN
[Thread-80:quorumcnxmanager$sendwor...@586] - Exception when using
channel: 4
java.nio.channels.ClosedChannelException
  at
sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126)
  at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
  at
org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.send(QuorumCnxManager.java:548)
  at
org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:578)
2010-04-08 18:07:35,750 - WARN
[Thread-80:quorumcnxmanager$sendwor...@589] - Send worker leaving thread
2010-04-08 18:07:35,750 - WARN
[Thread-95:quorumcnxmanager$recvwor...@658] - Connection broken:
java.io.IOException: Channel eof
  at
org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630)
2010-04-08 18:07:35,750 - ERROR
[Thread-92:quorumcnxmanager$sendwor...@559] - Failed to send last
message. Shutting down thread.
java.nio.channels.ClosedChannelException
  at
sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126)
  at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
  at
org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.send(QuorumCnxManager.java:548)
  at
org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:557)
2010-04-08 18:07:35,750 - WARN
[Thread-92:quorumcnxmanager$sendwor...@589] - Send worker leaving threa
Thanks.
Regards,
-Vishal
On Wed, Apr 7, 2010 at 5:53 PM, Patrick Hunt mailto:ph...@apache.org>> wrote:

I can't tell for certain. Looking a bit closer the code that starts
the server for the c tests does reference the clover jar, so my
guess would be that it does, but the messages from clover and the
build, etc.. don't shed any light. I didn't setup the zk clover
stuff for hudson, cc'ing Giri who might have insight.

Patrick


On 04/07/2010 02:34 PM, Vishal K wrote:

Hi Patrick,
We are not using C clients so we are not worried about porting
them to
Windows. Just out of curiosity, is there any way to confirm if
the code
coverage results are a result of test-core-java. If not, I will
run the
coverage tools locally.
I had tried the smoketests earlier, but I ran into some issues
there. I
forgot to note down the problem. I will revisit and upload the
results.
Thanks for your help.

    On Wed, Apr 7, 2010 at 5:11 PM, Patrick Hunt mailto:ph...@apache.org>
<mailto:ph...@apache.org <mailto:ph...@apache.org>>> wrote:


On 04/07/2010 01:51 PM, Vishal K wrote:

Hi Mahadev,

Thanks for your response. Currently I am running ZK without
cygwin on
windows. I will give it a try on cygwin. I am not quite
familiar
with
cppunit. Why will cppinit give me more confidence in
native windows
libraries?

I have few more questions relevant to testing:
1. How much code coverage do we get with
"test-core-java"? I see
68.8%
coverage on hudson
http://hudson.zones.apache.org/hudson/job/ZooKeeper-trunk/clover/.
  Does
this measure the coverage from junit tests run from
"test-core-java"?


Right, afaik that's test-core-java.

I think if you plan to run in production under windows you would
want to run in win, not cygwin. If that's the case you'd have to
port the c client and c client tests to some windows
specific compiler.


2. What would be a good and reliable set of tests that
will help
me verify
that the cluster holding up fine on windows. I tried to run
systest and
generateload. But I am having issues with the tests (and
also
understanding
the output of the tests since I am not familiar with the
source). The

Re: QA effort to support windows as a production platform

[Patrick Hunt]

Vishal, in general how are things going with your windows evaluation? 
Can you give a short review of what you've tested, what's working and 
what you still would like to look at? You are testing under windows 
directly, correct? (not cygwin I mean). Any chance you will be able to 
test the c, python and/or perl bindings? (although this would mean 
porting to windows compiler if i understand correctly).


We should consider updating the documentation to include detail about 
windows support if you feel comfortable. Enter a JIRA for this if you 
agree. Perhaps a wiki page detailing status and any issues running on 
that platform might also be useful? It would be great to offer (with 
confidence) ZK to users of the windows platform.


Thanks for taking on this effort, regards,

Patrick

On 04/08/2010 03:21 PM, Vishal K wrote:

Hi Patrick, Mahadev,
I was going through the logs of junit tests (all of the tests passed). I
noticed the logs for windows are different from the logs on linux.
On windows, the logs show following "Failed to send last message" error
message several times. None of the logs on Linux have them.
Are these messages normal? I have also attached the log file for the
AsyncHammerTest.
2010-04-08 18:07:35,750 - WARN
[Thread-80:quorumcnxmanager$sendwor...@586] - Exception when using
channel: 4
java.nio.channels.ClosedChannelException
  at
sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126)
  at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
  at
org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.send(QuorumCnxManager.java:548)
  at
org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:578)
2010-04-08 18:07:35,750 - WARN
[Thread-80:quorumcnxmanager$sendwor...@589] - Send worker leaving thread
2010-04-08 18:07:35,750 - WARN
[Thread-95:quorumcnxmanager$recvwor...@658] - Connection broken:
java.io.IOException: Channel eof
  at
org.apache.zookeeper.server.quorum.QuorumCnxManager$RecvWorker.run(QuorumCnxManager.java:630)
2010-04-08 18:07:35,750 - ERROR
[Thread-92:quorumcnxmanager$sendwor...@559] - Failed to send last
message. Shutting down thread.
java.nio.channels.ClosedChannelException
  at
sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126)
  at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
  at
org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.send(QuorumCnxManager.java:548)
  at
org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:557)
2010-04-08 18:07:35,750 - WARN
[Thread-92:quorumcnxmanager$sendwor...@589] - Send worker leaving threa
Thanks.
Regards,
-Vishal
On Wed, Apr 7, 2010 at 5:53 PM, Patrick Hunt mailto:ph...@apache.org>> wrote:

I can't tell for certain. Looking a bit closer the code that starts
the server for the c tests does reference the clover jar, so my
guess would be that it does, but the messages from clover and the
build, etc.. don't shed any light. I didn't setup the zk clover
stuff for hudson, cc'ing Giri who might have insight.

Patrick


On 04/07/2010 02:34 PM, Vishal K wrote:

Hi Patrick,
We are not using C clients so we are not worried about porting
them to
Windows. Just out of curiosity, is there any way to confirm if
the code
coverage results are a result of test-core-java. If not, I will
run the
coverage tools locally.
I had tried the smoketests earlier, but I ran into some issues
there. I
forgot to note down the problem. I will revisit and upload the
results.
Thanks for your help.

    On Wed, Apr 7, 2010 at 5:11 PM, Patrick Hunt mailto:ph...@apache.org>
<mailto:ph...@apache.org <mailto:ph...@apache.org>>> wrote:


On 04/07/2010 01:51 PM, Vishal K wrote:

Hi Mahadev,

Thanks for your response. Currently I am running ZK without
cygwin on
windows. I will give it a try on cygwin. I am not quite
familiar
with
cppunit. Why will cppinit give me more confidence in
native windows
libraries?

I have few more questions relevant to testing:
1. How much code coverage do we get with
"test-core-java"? I see
68.8%
coverage on hudson
http://hudson.zones.apache.org/hudson/job/ZooKeeper-trunk/clover/.
  Does
this measure the coverage from junit tests run from
"test-core-java"?


Right, afaik that's test-core-java.

I think if you plan to run in production under windows you would
want to run in win, not cygwin. If that's the case you'd have to
port the c client and c client test

Re: QA effort to support windows as a production platform

[Patrick Hunt]

On 04/09/2010 07:01 AM, Vishal K wrote:

Hi Patrick,

So far I have done the following:

1. Ran junit tests on both windows and Linux. I ran into few issues with the
tests: https://issues.apache.org/jira/browse/ZOOKEEPER-734. Also, the path
to dataDir in zoo.cfg should be either UNIX-style path or should contain
double backslashes instead of one. I think the path compatiblity issue is
the only bug found so far. I was a big skeptical earlier about the junit
test results since the logs of the junit tests on Windows differ from that
on linux. But as you mentioned in your earlier reply it is probably
difference in networking libraries (I am using the same java version on
both).



Ok, great. I've slated 734 for 3.3.1/3.4.0. If you'd like to contribute 
a patch please feel free (you need to submit as an attachment rather 
than inlining in the desc, see this link for details):

http://wiki.apache.org/hadoop/ZooKeeper/HowToContribute


2. Created a 1 Windows + 2 Linux cluster. Did some hand testing and things
seem to work fine.



3 distinct clusters you mean or 3 servers; 1 on windows and 2 on linux? 
(that would be a interesting test btw :-) )



3. I tried to run smoketest on the cluster, but I had some issues with the
test. I will be trying to run this test again soon.



Ben has been out of the office, once he gets back he's the best person 
to shed light. If you have specific questions someone else might be able 
to shed light in the mean time.



4. I tried systest and generateLoad test. systest passed but generateLoad
test started spewing a lot of connection/session closed exceptions.



You might be underprovisioning the server(s)? Try monitoring the servers 
while running the test. You can use something like

http://github.com/phunt/zktop
esp the "min/avg/max latency" information.

My experience on windows is that the IO is not as good as unix (esp 
going through java). You should def. test under some load (using the 
latency tester as I mentioned earlier, simliar to my tests on the wiki) 
to ensure that the system will function in this environment. Ensure that 
you have enough heap, you might need to tune the GC to turn on 
cms/incremental GC (that could also be the issue you saw earlier), and 
also monitor the disk activity.


Btw, have you tested the 4 letter words? stat and such I mean. Would be 
good to verify as well as they are critical for monitoring the system 
once in production.



I hope to run systest, generateloadtest, smoktest and latencytest. If we
decide to go with the ZK on windows, then we will also write our own tests
(and do more rigorous QA). I will be happy to add these to the ZK test suite
as well.


Great, happy to have them. See the how to contribute link above.


I believe at the end of this exercise we will have fair amount of confidence
in ZK on windows. I will keep you posted.


That will be great. Having an actual user who's running in production 
under that configuration would bring even more weight.


At some point we should look into the C client side of things as well. 
That will be harder, but if we can do it it would mean that 
python/perl/etc... would also be available for the platform.


Thanks!

Patrick



On Fri, Apr 9, 2010 at 1:01 AM, Patrick Hunt  wrote:


Vishal, in general how are things going with your windows evaluation? Can
you give a short review of what you've tested, what's working and what you
still would like to look at? You are testing under windows directly,
correct? (not cygwin I mean). Any chance you will be able to test the c,
python and/or perl bindings? (although this would mean porting to windows
compiler if i understand correctly).

We should consider updating the documentation to include detail about
windows support if you feel comfortable. Enter a JIRA for this if you agree.
Perhaps a wiki page detailing status and any issues running on that platform
might also be useful? It would be great to offer (with confidence) ZK to
users of the windows platform.

Thanks for taking on this effort, regards,

Patrick


On 04/08/2010 03:21 PM, Vishal K wrote:


  Hi Patrick, Mahadev,
I was going through the logs of junit tests (all of the tests passed). I
noticed the logs for windows are different from the logs on linux.
On windows, the logs show following "Failed to send last message" error
message several times. None of the logs on Linux have them.
Are these messages normal? I have also attached the log file for the
AsyncHammerTest.
2010-04-08 18:07:35,750 - WARN
[Thread-80:quorumcnxmanager$sendwor...@586] - Exception when using
channel: 4
java.nio.channels.ClosedChannelException
  at
sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:126)
  at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:324)
  at

org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.send(QuorumCnxManager.java:548)
  at

org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(

4letter word (monitor) issue in 3.3.0 with nc (netcat)

[Patrick Hunt]

I discovered a problem using nc with 3.3.0 such as:

echo stat | nc localhost 2181

full details here: http://bit.ly/ahkHdb

The failure is intermittent for me (timing issue) but I wanted to let 
everyone know about it given that nc is frequently used for monitoring. 
Given the nature of the problem you may be seeing results inconsistent 
with the status of the server (server up but stat not returning data).


This does not effect the functioning of the server, just the command 
itself is impaired.


We'll probably do a 3.3.1 relatively soon to correct this (and a few 
other issues that are slated for 3.3.1).


Patrick

Re: Zookeeper behind load balancer

[Patrick Hunt]

Two main issues I can think of:

1) unnecessary -- clients already randomize their connect string to 
balance the sessions across servers
2) unless you turn on some sort of session tracking in the ballancer 
client's won't be able to use the service. ZK sessions use long lived 
TCP connections to maintain a session with a particular server.


See session docs for more detail. http://bit.ly/db90Y5

Regards,

Patrick

On 04/14/2010 06:48 AM, Toader, Sebastian wrote:

Hi ZooKeeper Dev Team,

We are thinking of putting our ZooKeeper server behind load balancer. The 
client applications would connect to assemble through the load balancer.

Can you let us know if there are any drawbacks in putting the assemble behind a 
load balancer?

Thank you,
Seb

--
NOTICE: If received in error, please destroy, and notify sender. Sender does 
not intend to waive confidentiality or privilege. Use of this email is 
prohibited when received in error. We may monitor and store emails to the 
extent permitted by applicable law.

Re: Zookeeper behind load balancer

[Patrick Hunt]

On 04/14/2010 09:33 AM, Mekaraj, Prashant wrote:

Many thanks for the quick reply.


1) unnecessary -- clients already randomize their connect string
to balance the sessions across servers


The problems we are trying to solve are 1. updating the ensemble to
add new servers implies finding an updating every client who is
connecting to it(which implies we know who the clients are)


That's true. However the only real reason to change the ensemble size is
to increase reliability. We typically suggest 5 servers for on-line
production serving environments. This provides high reliability and also
allows for things like bringing down a server for maint, but still being
able to handle the case where another server may fail unexpectedly (but
the service will still stay up). Over 5 servers there is no significant
benefit to reliability, only the negative performance impact of having a
larger quorum.


2. if we deploy zookeeper in a region and put observers in other
regions, load balancers make it easy to preferably connect to a local
regional ip and failover to a remote one if the local ip is
unavailable


Yes, this is a good point. Also if you change the name/ip of a server
(say one fails and you need to replace)


3. load balancers offer more sophisticated balancing algo's - load
based for example.


2) unless you turn on some sort of session tracking in the
ballancer client's won't be able to use the service. ZK sessions
use long lived TCP connections to maintain a session with a
particular server.


The load balancer tracks sessions and makes sure that a client will
"stick" to a particular backend zookeeper server.


This is a trick question. ;-) One additional thing you would need to
ensure is that if the zk server fails the load balancer will take it out
of rotation.

I don't have any experience using LBs for this. It should be easy enough 
for you to test in your environment/lbs though. I'd be interested to 
know you results - if possible please post your findings!


Regards,

Patrick


Flavio,

Hi Seb, I was wondering how you're thinking about performing
session management interposing a load balancer between clients>
and zookeeper servers. I'm probably missing some important piece of
your design.


Load balancer stickiness should handle it if I understand your point
correctly.


-prashant





See session docs for more detail. http://bit.ly/db90Y5

Regards,

Patrick

On 04/14/2010 06:48 AM, Toader, Sebastian wrote:

Hi ZooKeeper Dev Team,

We are thinking of putting our ZooKeeper server behind load
balancer.

The client applications would connect to assemble through the load
balancer.


Can you let us know if there are any drawbacks in putting the

assemble behind a load balancer?


Thank you, Seb

-





-

NOTICE: If received in error, please destroy, and notify sender.

Sender does not intend to waive confidentiality or privilege. Use
of this email is prohibited when received in error. We may monitor
and store emails to the extent permitted by applicable law.




--


NOTICE: If received in error, please destroy, and notify sender. Sender 
does not intend to waive confidentiality or privilege. Use of this email 
is prohibited when received in error. We may monitor and store emails to 
the extent permitted by applicable law.

Re: Zookeeper behind load balancer

[Patrick Hunt]

On 04/14/2010 10:09 AM, Mekaraj, Prashant wrote:

We did test this and it seems to work fine, but ymmv depending on the
load balancer product I suppose.


Yes, that's def true. Please highlight this in your results writeup. If 
you can provide the name, config, etc.. of the lb you used that would be 
helpful.



We've done some tests and can post results. Do we just file a jira with the 
text of the findings ?


Awesome. If you don't have any issues (bugs/features) I'd suggest you 
put it on the wiki directly, something like:


http://wiki.apache.org/hadoop/ZooKeeper/LoadBalancer

Feel free to put that on the ZooKeeper main wiki page.

You might also consider doing a blog post about this. It would be 
interesting for our users!


Regards,

Patrick



-prashant



Regards,

Patrick


Flavio,

Hi Seb, I was wondering how you're thinking about performing
session management interposing a load balancer between clients>
and zookeeper servers. I'm probably missing some important piece of
your design.


Load balancer stickiness should handle it if I understand your point
correctly.


-prashant





See session docs for more detail. http://bit.ly/db90Y5

Regards,

Patrick

On 04/14/2010 06:48 AM, Toader, Sebastian wrote:

Hi ZooKeeper Dev Team,

We are thinking of putting our ZooKeeper server behind load
balancer.

The client applications would connect to assemble through the load
balancer.


Can you let us know if there are any drawbacks in putting the

assemble behind a load balancer?


Thank you, Seb

---

--





-

NOTICE: If received in error, please destroy, and notify sender.

Sender does not intend to waive confidentiality or privilege. Use
of this email is prohibited when received in error. We may monitor
and store emails to the extent permitted by applicable law.




-

-




NOTICE: If received in error, please destroy, and notify sender. Sender
does not intend to waive confidentiality or privilege. Use of this
email
is prohibited when received in error. We may monitor and store emails
to
the extent permitted by applicable law.


--
NOTICE: If received in error, please destroy, and notify sender. Sender does 
not intend to waive confidentiality or privilege. Use of this email is 
prohibited when received in error. We may monitor and store emails to the 
extent permitted by applicable law.

Re: .NET client

[Patrick Hunt]

Hi Eric, this sounds great! Please consider submitting the client 
binding back, you could start it as a contrib package (src/contrib) 
until things settle and you get some usage. We recently have been seeing 
interest in running ZK on windows servers, so having .net would be awesome.

http://wiki.apache.org/hadoop/ZooKeeper/HowToContribute

It would be nice to see the Jute support, btw this is a port of Hadoop's 
"recordio" package from a few years ago. We are actually looking to move 
to Avro at some point, I'm sure they would love to have .net support as 
well.


Regards,

Patrick

On 04/15/2010 05:55 PM, Eric Hauser wrote:

Hi,

I've ported the Java ZK client to .NET.  It's available for forking at
http://github.com/ewhauser/zookeeper/.  It is still in very early stages,
but all of the client tests that I have ported complete successfully - both
normal calls and watches are working successfully.  Some things to keep in
mind:

- It's not actively being used at the moment
- I have not had any chance to do any performance testing
- The API will definitely be changing.  I've ported watches over as is in
order to get everything working, but I'll be changing that to use events
instead to make it more .NET-like.
- There are not any asynchronous methods on the ZK client API -- async
method calls are easy to do .NET.  May add a wrapper for it at some point.

If there is any interest in having any part of my work go back into trunk,
I'm happy to sign the ASF release.  In particular, it would be nice if the
C# support I added to Jute made it back into the core code base.

Status on 3.3.1 release

[Patrick Hunt]

We're working towards a 3.3.1 which should have a release candidate up 
for vote early/mid next week. There are currently two blockers being 
investigated. Henry, Mahadev and I reviewed the 3.3.1 jira list today 
and reassigned some issues. We're hoping to address the 3.3.1 jira list 
for the fix release, but once the blockers are fixed we'll push any 
non-blockers into a subsequent release.


Patrick

Re: .NET client

[Patrick Hunt]

Eric, this is great. I've slated the change for 3.4.0, if you could 
address the RAT issue (add license header), resubmit and someone will 
review and work with you to get committed.


Thanks! And getting the client bindings would be great too.

Patrick

On 04/21/2010 08:32 PM, Eric Hauser wrote:

I submitted the Jute changes back as ZOOKEEPER-747.

Regarding contrib, I'd be happy to submit the client bindings back once
I do a bit more cleanup.

On Mon, Apr 19, 2010 at 1:11 PM, Patrick Hunt mailto:ph...@apache.org>> wrote:

Hi Eric, this sounds great! Please consider submitting the client
binding back, you could start it as a contrib package (src/contrib)
until things settle and you get some usage. We recently have been
seeing interest in running ZK on windows servers, so having .net
would be awesome.
http://wiki.apache.org/hadoop/ZooKeeper/HowToContribute

It would be nice to see the Jute support, btw this is a port of
Hadoop's "recordio" package from a few years ago. We are actually
looking to move to Avro at some point, I'm sure they would love to
have .net support as well.

Regards,

Patrick


On 04/15/2010 05:55 PM, Eric Hauser wrote:

Hi,

I've ported the Java ZK client to .NET.  It's available for
forking at
http://github.com/ewhauser/zookeeper/.  It is still in very
early stages,
but all of the client tests that I have ported complete
successfully - both
normal calls and watches are working successfully.  Some things
to keep in
mind:

- It's not actively being used at the moment
- I have not had any chance to do any performance testing
- The API will definitely be changing.  I've ported watches over
as is in
order to get everything working, but I'll be changing that to
use events
instead to make it more .NET-like.
- There are not any asynchronous methods on the ZK client API --
async
method calls are easy to do .NET.  May add a wrapper for it at
some point.

If there is any interest in having any part of my work go back
into trunk,
I'm happy to sign the ASF release.  In particular, it would be
nice if the
C# support I added to Jute made it back into the core code base.

Re: ZooKeeper gets three Google Summer of Code students

[Patrick Hunt]

3 GSOC projects approved, that's great! Thank you students for applying for
a project, should be some interesting new functionality/features developed
over the summer. Thanks to all the mentors for stepping up and volunteering
their time and Henry for shepherding the gsoc process (as well as being a
mentor).

I encourage our user base to engage with the students if they have interest
in a particular project. If you have ideas, problems, or suggestions for
features please discuss with the student (on the list is great). Exposing
the mentees to how ZK based software is used in real life production
environments will allow them to build a better product.

Regards,

Patrick

On Mon, Apr 26, 2010 at 4:55 PM, Henry Robinson  wrote:

> Hi -
>
> Just wanted to announce to the community that we are lucky to have three
> talented students working on Google's Summer of Code projects directly
> related to ZooKeeper.
>
> Andrei Savu  will be working with Patrick Hunt on a Web-based
> Administrative
> Interface, extending and improving Patrick's Django-based front end.
> Abmar Barros will be working with Flavio Junqueira on improving ZooKeeper's
> failure detector module - making the code cleaner and easier to try out new
> implementations, as well as implementing a few failure detection algorithms
> himself!
> Finally, Sergey Doroshenko will be working with me on a Read-Only Mode for
> ZooKeeper, which will help bolster ZK's availability in certain
> circumstances when a network partition is detected, as well as potentially
> optimising the read-path.
>
> (The full list of 450 GSoC students is here:
> http://socghop.appspot.com/gsoc/program/list_projects/google/gsoc2010)
>
> Congratulations to all three - we look forward to seeing what you produce
> over the summer. Thanks to everyone who applied, suggested projects and
> offered to mentor students; this program will have a big effect on
> ZooKeeper's visibility and community, as well as hopefully producing some
> great code!
>
> cheers,
> Henry
>
> --
> Henry Robinson
> Software Engineer
> Cloudera
> 415-994-6679
>

FYI Apache Avro considering .NET support.

[Patrick Hunt]

For those windows based ZooKeeper developers, Avro is looking at .NET 
support http://bit.ly/9nThxD seems like it would be a great project (and 
we are planning to add Avro marshaling support at some point).


Patrick

Interesting Avro/Thrift JIRA re utf8 performance

[Patrick Hunt]

We also suffer from Java's poor string conversion performance (ser/deser 
of znode paths in the marshalling code), we should take advantage of 
this as well:


https://issues.apache.org/jira/browse/AVRO-532

"I did perform a micro-benchmark of string encoding and decoding, which 
showed the new methods take about half as long"


https://issues.apache.org/jira/browse/ZOOKEEPER-760

Patrick

Re: Dynamic adding/removing ZK servers on client

[Patrick Hunt]

On 05/03/2010 07:03 AM, Dave Wright wrote:

I've got a situation where I essentially need dynamic cluster
membership, which has been talked about in ZOOKEEPER-107 but doesn't
look like it's going to happen any time soon.



Could you provide some insight into why you need this? Just so we have 
addl background, I'm interested to know the use case.



For now, I'm planning on working around this by having a simple
coordinator service on the server nodes that will re-write the configs
and bounce the servers when membership changes. Clients will may get
an error or two and need to reconnect, but that should be handled by
the normal error logic.



Are you expecting all of the servers to change each time, or just 
incremental changes (add/remove a single server, vs say move the entire 
cluster from 3 hosts a/b/c to x/y/z)



On the client side, I'd really like to dynamically update the server
list w/o having to re-create the entire Zookeeper object. Looking at
the code, it seems like it would be pretty trivial to add
"RemoveServer()/AddServer()" functions for Zookeeper that calls down
to ClientCnxn, where they are just maintained in a list. Of course if
the server being removed is the one currently connected, we'd need to
disconnect, but a simple call to disconnect() seems like it would
resolve that and trigger the automatic re-connection logic.



You would hook this (add/remove) into JMX? That seems like a good option 
to provide.


Any chance you could use DNS for this? ie change the mapping for the 
hostname from a -> x ip? Since the server a will go down anyway, this 
would cause the client to reconnect to b/c (eventually when dns ttl 
expires the client would also potentially connect to x).


If this is an option be sure to see (a bit of work to do):
https://issues.apache.org/jira/browse/ZOOKEEPER-328
https://issues.apache.org/jira/browse/ZOOKEEPER-338

You might also look at this patch, we never committed it but it might be 
interesting to you:

https://issues.apache.org/jira/browse/ZOOKEEPER-146

The benefit is that you'd only have one place to make the change, esp 
given that clients might be down/unreachable when this change occurs. 
Clients would have to poll this service whenever they get disconnected 
from the ensemble. One drawback of this approach is that the HTTP now 
becomes a potential SPOF. (although I guess you could always fall back 
to something, or potentially have a list of HTTP hosts to do the lookup, 
etc...).



Does anyone see an issue with that approach?
Were I to create the patch, do you think it would be interesting
enough to merge? It seems like that functionality will eventually be
needed for whatever full dynamic server support is eventually
implemented.


It does sound interesting, however once we add something like this it's 
hard to change given that we try very hard to maintain b/w 
compatibility. If you did the testing and were able to verify I don't 
see why we couldn't add it - as it's "optional" in the sense that it 
would only be called in the use case you describe. I would feel more 
confident if we had more concrete detail on how we intend to do 107 (a 
basic functional/design doc that at least reviews all the issues), and 
how this would fit in. But I don't see that should necessarily be a 
blocker (although others might feel differently).


(fyi it's good to discuss this sort of thing on zookeeper-dev, please 
move responses to that list)


Sounds like an useful project, I'm interested to her what others think 
about it. Regards,


Patrick

Re: Dynamic adding/removing ZK servers on client

[Patrick Hunt]

Another benefit of ZOOKEEPER-146 - we could use this for some sort of 
load balancing amongst the ensemble members. The first version could 
return a static list, however I can see where the HTTPD might be updated 
to monitor the load on the servers/ensemble and prioritize the list for 
each client request...


Patrick

On 05/03/2010 09:34 AM, Patrick Hunt wrote:


On 05/03/2010 07:03 AM, Dave Wright wrote:

I've got a situation where I essentially need dynamic cluster
membership, which has been talked about in ZOOKEEPER-107 but doesn't
look like it's going to happen any time soon.



Could you provide some insight into why you need this? Just so we have
addl background, I'm interested to know the use case.


For now, I'm planning on working around this by having a simple
coordinator service on the server nodes that will re-write the configs
and bounce the servers when membership changes. Clients will may get
an error or two and need to reconnect, but that should be handled by
the normal error logic.



Are you expecting all of the servers to change each time, or just
incremental changes (add/remove a single server, vs say move the entire
cluster from 3 hosts a/b/c to x/y/z)


On the client side, I'd really like to dynamically update the server
list w/o having to re-create the entire Zookeeper object. Looking at
the code, it seems like it would be pretty trivial to add
"RemoveServer()/AddServer()" functions for Zookeeper that calls down
to ClientCnxn, where they are just maintained in a list. Of course if
the server being removed is the one currently connected, we'd need to
disconnect, but a simple call to disconnect() seems like it would
resolve that and trigger the automatic re-connection logic.



You would hook this (add/remove) into JMX? That seems like a good option
to provide.

Any chance you could use DNS for this? ie change the mapping for the
hostname from a -> x ip? Since the server a will go down anyway, this
would cause the client to reconnect to b/c (eventually when dns ttl
expires the client would also potentially connect to x).

If this is an option be sure to see (a bit of work to do):
https://issues.apache.org/jira/browse/ZOOKEEPER-328
https://issues.apache.org/jira/browse/ZOOKEEPER-338

You might also look at this patch, we never committed it but it might be
interesting to you:
https://issues.apache.org/jira/browse/ZOOKEEPER-146

The benefit is that you'd only have one place to make the change, esp
given that clients might be down/unreachable when this change occurs.
Clients would have to poll this service whenever they get disconnected
from the ensemble. One drawback of this approach is that the HTTP now
becomes a potential SPOF. (although I guess you could always fall back
to something, or potentially have a list of HTTP hosts to do the lookup,
etc...).


Does anyone see an issue with that approach?
Were I to create the patch, do you think it would be interesting
enough to merge? It seems like that functionality will eventually be
needed for whatever full dynamic server support is eventually
implemented.


It does sound interesting, however once we add something like this it's
hard to change given that we try very hard to maintain b/w
compatibility. If you did the testing and were able to verify I don't
see why we couldn't add it - as it's "optional" in the sense that it
would only be called in the use case you describe. I would feel more
confident if we had more concrete detail on how we intend to do 107 (a
basic functional/design doc that at least reviews all the issues), and
how this would fit in. But I don't see that should necessarily be a
blocker (although others might feel differently).

(fyi it's good to discuss this sort of thing on zookeeper-dev, please
move responses to that list)

Sounds like an useful project, I'm interested to her what others think
about it. Regards,

Patrick

Re: Dynamic adding/removing ZK servers on client

[Patrick Hunt]

On 05/03/2010 10:28 AM, Dave Wright wrote:

You might also look at this patch, we never committed it but it might be
interesting to you:
https://issues.apache.org/jira/browse/ZOOKEEPER-146

The benefit is that you'd only have one place to make the change, esp given
that clients might be down/unreachable when this change occurs. Clients
would have to poll this service whenever they get disconnected from the
ensemble. One drawback of this approach is that the HTTP now becomes a
potential SPOF. (although I guess you could always fall back to something,
or potentially have a list of HTTP hosts to do the lookup, etc...).


Well, that just handles distribution of the list (which isn't really
our problem), it doesn't help with restarting the ZK client when the
list changes - it only pulls the list once, so you still have to
completely shutdown and restart the ZK client.



Well the old server is being shutdown right? If the client were 
connected to that server this would force the client to reconnect to 
another server, what I was suggesting is that the client would ping the 
"server lookup" service as part of this. (so lookup on every disconnect say)





It does sound interesting, however once we add something like this it's hard
to change given that we try very hard to maintain b/w compatibility. If you
did the testing and were able to verify I don't see why we couldn't add it -
as it's "optional" in the sense that it would only be called in the use case
you describe. I would feel more confident if we had more concrete detail on
how we intend to do 107 (a basic functional/design doc that at least reviews
all the issues), and how this would fit in. But I don't see that should
necessarily be a blocker (although others might feel differently).


Have you ever considered adding features like this via a protected
interface (i.e. the are useful but aren't fully standardized, so if a
client wants to use it they can sub-class ZK and make them public)?



If you take a look at the end of ZooKeeper you'll see that we have some 
like that already, but IIRC they were for testing purposes. But yes, 
that would be a great way to expose it. We could also mark the initial 
version as "experimental" or perhaps "unstable" letting potential users 
know that we're still working on it.



The ability to dynamically modify the server list on the client side
seems like it would be required no matter what approach were taken to
dynamic clusters.


Hasn't come up before, but yes I agree it's a useful feature.

Patrick

Re: Dynamic adding/removing ZK servers on client

[Patrick Hunt]

On 05/03/2010 11:29 AM, Dave Wright wrote:

Well, that just handles distribution of the list (which isn't really
our problem), it doesn't help with restarting the ZK client when the
list changes - it only pulls the list once, so you still have to
completely shutdown and restart the ZK client.



Well the old server is being shutdown right? If the client were connected to
that server this would force the client to reconnect to another server, what
I was suggesting is that the client would ping the "server lookup" service
as part of this. (so lookup on every disconnect say)


Perhaps we should clarify what you mean by "client" (..would ping..).
If you mean the ZK client library, then that would make sense - rather
than use a static list of servers, each time it was disconnected it
would refresh it's list and pick one.
I took it to mean the client application (using the ZK library). The
issue is that the client application has no way to tell the ZK client
lib to use a different list of servers, other than a complete teardown
of the ZK object&  session, which I'm trying to avoid.



Yes, that's what I meant - we could update the ZK client lib to do this. 
It would be invisible to the client application (your code) itself.




Hasn't come up before, but yes I agree it's a useful feature.


Ok, thanks. We don't have a specific ETA to implement it, I just
wanted to explore the option a bit before we finalized some aspects of
our design. Should we do the work I'll submit matches for the Java and
C client.


That would be great.

Patrick

Re: Dynamic adding/removing ZK servers on client

[Patrick Hunt]

On 05/03/2010 12:07 PM, Dave Wright wrote:


Yes, that's what I meant - we could update the ZK client lib to do this. It
would be invisible to the client application (your code) itself.


I don't think that's a bad idea, and the general approach in ZK-146 of
using an interface that gets called to retrieve the list of hosts
seems good (so that you aren't tied to a specific implementation of
hosts lists, be it HTTP or DNS). That said, I don't think the actual
implementation of ZK-146 is a good solution, since it only resolves
the host list once. An implementation that resolved it on each
disconnection would be better but require deeper changes to the
ClientCnxn.


You could update 146 as appropriate, handling changes to the ensemble 
members wasn't an original goal. Notice there was some discussion on how 
to do this in a way that would be as flexible as possible going forward, 
and so that we don't end up with all kinds of constructors (etc...) on 
top of ZK client for the different schemes. That is still a concern, 
something that we should come to agreement on before implementation is 
started I mean.


Patrick