[jira] Commented: (ZOOKEEPER-368) Observers: core functionality

2009-11-16 Thread Mahadev konar (JIRA) 

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12778704#action_12778704
 ] 

Mahadev konar commented on ZOOKEEPER-368:
-

henry, 
  did you get a chance to test observers with old (meaning without hte patch) 
zookeeper servers? repeating my comments here:

 -   looking at the patch it seems like it would work with servers prior to 
including this patch. Did you try some testing with current servers (killing 
one at a time and brining them up in a round robin fashion) just to make sure 
it works all fine with the current servers (not including the patch)?
 - what happens if a server configured as follower is suddenly brought down and 
is made an observer and the other way around as well? Just checking to see if 
we have these scenarios covered because such mistakes are easy to make when 
setting up servers

> Observers: core functionality 
> --
>
> Key: ZOOKEEPER-368
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-368
> Project: Zookeeper
>  Issue Type: New Feature
>  Components: quorum
>Reporter: Flavio Paiva Junqueira
>Assignee: Henry Robinson
> Fix For: 3.3.0
>
> Attachments: obs-refactor.patch, observer-refactor.patch, observers 
> sync benchmark.png, observers.patch, ZOOKEEPER-368.patch, 
> ZOOKEEPER-368.patch, ZOOKEEPER-368.patch, ZOOKEEPER-368.patch, 
> ZOOKEEPER-368.patch, ZOOKEEPER-368.patch, ZOOKEEPER-368.patch, 
> ZOOKEEPER-368.patch, ZOOKEEPER-368.patch, ZOOKEEPER-368.patch
>
>
> Edit (Henry Robinson/henryr) 12/11/09:
> This JIRA specifically concerns the implementation of non-voting peers called 
> Observers, their documentation and their tests. 
> Explicit goals are 1. not breaking any current ZK functionality, 2. enabling 
> at least one deployment scenario involving Observers, 3. documentation 
> describing how to use the feature and 4. tests validating the correct 
> behaviour of 2. 
> Non goals of this JIRA are 1. performance optimizations specific to 
> Observers, 2. compatibility with every feature of ZooKeeper (in particular 
> all leader election protocols), which are both to be addressed in future 
> JIRAs. 
> See http://wiki.apache.org/hadoop/ZooKeeper/Observers for more detail of use 
> cases, proposed design and usage.
> See http://wiki.apache.org/hadoop/ZooKeeper/Observers/ReviewGuide for a brief 
> commentary on the current patch. 
> -
> Currently, all servers of an ensemble participate actively in reaching 
> agreement on the order of ZooKeeper transactions. That is, all followers 
> receive proposals, acknowledge them, and receive commit messages from the 
> leader. A leader issues commit messages once it receives acknowledgments from 
> a quorum of followers. For cross-colo operation, it would be useful to have a 
> third role: observer. Using Paxos terminology, observers are similar to 
> learners. An observer does not participate actively in the agreement step of 
> the atomic broadcast protocol. Instead, it only commits proposals that have 
> been accepted by some quorum of followers.
> One simple solution to implement observers is to have the leader forwarding 
> commit messages not only to followers but also to observers, and have 
> observers applying transactions according to the order followers agreed upon. 
> In the current implementation of the protocol, however, commit messages do 
> not carry their corresponding transaction payload because all servers 
> different from the leader are followers and followers receive such a payload 
> first through a proposal message. Just forwarding commit messages as they 
> currently are to an observer consequently is not sufficient. We have a couple 
> of options:
> 1- Include the transaction payload along in commit messages to observers;
> 2- Send proposals to observers as well.
> Number 2 is simpler to implement because it doesn't require changing the 
> protocol implementation, but it increases traffic slightly. The performance 
> impact due to such an increase might be insignificant, though.
> For scalability purposes, we may consider having followers also forwarding 
> commit messages to observers. With this option, observers can connect to 
> followers, and receive messages from followers. This choice is important to 
> avoid increasing the load on the leader with the number of observers. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (ZOOKEEPER-547) Sanity check in QuorumCnxn Manager and quorum communication port.

2009-11-16 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12778663#action_12778663
 ] 

Hadoop QA commented on ZOOKEEPER-547:
-

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12425168/ZOOKEEPER-547.patch
  against trunk revision 835618.

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 3 new or modified tests.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

+1 findbugs.  The patch does not introduce any new Findbugs warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

+1 core tests.  The patch passed core unit tests.

+1 contrib tests.  The patch passed contrib unit tests.

Test results: 
http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/66/testReport/
Findbugs warnings: 
http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/66/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: 
http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/66/console

This message is automatically generated.

> Sanity check in QuorumCnxn Manager and quorum communication port.
> -
>
> Key: ZOOKEEPER-547
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-547
> Project: Zookeeper
>  Issue Type: Bug
>  Components: leaderElection, server
>Affects Versions: 3.2.0, 3.2.1
>Reporter: Mahadev konar
>Assignee: Mahadev konar
> Fix For: 3.2.2, 3.3.0
>
> Attachments: ZOOKEEPER-547.patch, ZOOKEEPER-547.patch, 
> ZOOKEEPER-547.patch, ZOOKEEPER-547.patch
>
>
> We need to put some sanity checks in QuorumCnxnManager and the other quorum 
> port for rogue clients. Sometimes a clients might get misconfigured and they 
> might send random characters on such ports. We need to make sure that such 
> rogue clients do not bring down the clients and need to put in some sanity 
> checks with respect to packet lengths and deserialization.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (ZOOKEEPER-507) Improve error handling of BookKeeper client

2009-11-16 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12778650#action_12778650
 ] 

Hadoop QA commented on ZOOKEEPER-507:
-

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12422953/ZOOKEEPER-507.patch
  against trunk revision 835618.

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 36 new or modified tests.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

+1 findbugs.  The patch does not introduce any new Findbugs warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

+1 core tests.  The patch passed core unit tests.

+1 contrib tests.  The patch passed contrib unit tests.

Test results: 
http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/65/testReport/
Findbugs warnings: 
http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/65/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: 
http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/65/console

This message is automatically generated.

> Improve error handling of BookKeeper client
> ---
>
> Key: ZOOKEEPER-507
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-507
> Project: Zookeeper
>  Issue Type: Improvement
>  Components: contrib-bookkeeper
>Reporter: Flavio Paiva Junqueira
>Assignee: Utkarsh Srivastava
> Attachments: ZOOKEEPER-507.patch
>
>
> Error handling is far from ideal currently in the BookKeeper client.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (ZOOKEEPER-547) Sanity check in QuorumCnxn Manager and quorum communication port.

2009-11-16 Thread Mahadev konar (JIRA) 

 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mahadev konar updated ZOOKEEPER-547:


Attachment: ZOOKEEPER-547.patch

this patch fixes the test case to pass.

> Sanity check in QuorumCnxn Manager and quorum communication port.
> -
>
> Key: ZOOKEEPER-547
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-547
> Project: Zookeeper
>  Issue Type: Bug
>  Components: leaderElection, server
>Affects Versions: 3.2.0, 3.2.1
>Reporter: Mahadev konar
>Assignee: Mahadev konar
> Fix For: 3.2.2, 3.3.0
>
> Attachments: ZOOKEEPER-547.patch, ZOOKEEPER-547.patch, 
> ZOOKEEPER-547.patch, ZOOKEEPER-547.patch
>
>
> We need to put some sanity checks in QuorumCnxnManager and the other quorum 
> port for rogue clients. Sometimes a clients might get misconfigured and they 
> might send random characters on such ports. We need to make sure that such 
> rogue clients do not bring down the clients and need to put in some sanity 
> checks with respect to packet lengths and deserialization.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (ZOOKEEPER-547) Sanity check in QuorumCnxn Manager and quorum communication port.

2009-11-16 Thread Mahadev konar (JIRA) 

 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mahadev konar updated ZOOKEEPER-547:


Status: Patch Available  (was: Open)

> Sanity check in QuorumCnxn Manager and quorum communication port.
> -
>
> Key: ZOOKEEPER-547
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-547
> Project: Zookeeper
>  Issue Type: Bug
>  Components: leaderElection, server
>Affects Versions: 3.2.1, 3.2.0
>Reporter: Mahadev konar
>Assignee: Mahadev konar
> Fix For: 3.2.2, 3.3.0
>
> Attachments: ZOOKEEPER-547.patch, ZOOKEEPER-547.patch, 
> ZOOKEEPER-547.patch, ZOOKEEPER-547.patch
>
>
> We need to put some sanity checks in QuorumCnxnManager and the other quorum 
> port for rogue clients. Sometimes a clients might get misconfigured and they 
> might send random characters on such ports. We need to make sure that such 
> rogue clients do not bring down the clients and need to put in some sanity 
> checks with respect to packet lengths and deserialization.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (ZOOKEEPER-547) Sanity check in QuorumCnxn Manager and quorum communication port.

2009-11-16 Thread Mahadev konar (JIRA) 

 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mahadev konar updated ZOOKEEPER-547:


Status: Open  (was: Patch Available)

> Sanity check in QuorumCnxn Manager and quorum communication port.
> -
>
> Key: ZOOKEEPER-547
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-547
> Project: Zookeeper
>  Issue Type: Bug
>  Components: leaderElection, server
>Affects Versions: 3.2.1, 3.2.0
>Reporter: Mahadev konar
>Assignee: Mahadev konar
> Fix For: 3.2.2, 3.3.0
>
> Attachments: ZOOKEEPER-547.patch, ZOOKEEPER-547.patch, 
> ZOOKEEPER-547.patch
>
>
> We need to put some sanity checks in QuorumCnxnManager and the other quorum 
> port for rogue clients. Sometimes a clients might get misconfigured and they 
> might send random characters on such ports. We need to make sure that such 
> rogue clients do not bring down the clients and need to put in some sanity 
> checks with respect to packet lengths and deserialization.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (ZOOKEEPER-368) Observers: core functionality

2009-11-16 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12778612#action_12778612
 ] 

Hadoop QA commented on ZOOKEEPER-368:
-

+1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12425104/ZOOKEEPER-368.patch
  against trunk revision 835618.

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 25 new or modified tests.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

+1 findbugs.  The patch does not introduce any new Findbugs warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

+1 core tests.  The patch passed core unit tests.

+1 contrib tests.  The patch passed contrib unit tests.

Test results: 
http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/64/testReport/
Findbugs warnings: 
http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/64/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: 
http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/64/console

This message is automatically generated.

> Observers: core functionality 
> --
>
> Key: ZOOKEEPER-368
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-368
> Project: Zookeeper
>  Issue Type: New Feature
>  Components: quorum
>Reporter: Flavio Paiva Junqueira
>Assignee: Henry Robinson
> Fix For: 3.3.0
>
> Attachments: obs-refactor.patch, observer-refactor.patch, observers 
> sync benchmark.png, observers.patch, ZOOKEEPER-368.patch, 
> ZOOKEEPER-368.patch, ZOOKEEPER-368.patch, ZOOKEEPER-368.patch, 
> ZOOKEEPER-368.patch, ZOOKEEPER-368.patch, ZOOKEEPER-368.patch, 
> ZOOKEEPER-368.patch, ZOOKEEPER-368.patch, ZOOKEEPER-368.patch
>
>
> Edit (Henry Robinson/henryr) 12/11/09:
> This JIRA specifically concerns the implementation of non-voting peers called 
> Observers, their documentation and their tests. 
> Explicit goals are 1. not breaking any current ZK functionality, 2. enabling 
> at least one deployment scenario involving Observers, 3. documentation 
> describing how to use the feature and 4. tests validating the correct 
> behaviour of 2. 
> Non goals of this JIRA are 1. performance optimizations specific to 
> Observers, 2. compatibility with every feature of ZooKeeper (in particular 
> all leader election protocols), which are both to be addressed in future 
> JIRAs. 
> See http://wiki.apache.org/hadoop/ZooKeeper/Observers for more detail of use 
> cases, proposed design and usage.
> See http://wiki.apache.org/hadoop/ZooKeeper/Observers/ReviewGuide for a brief 
> commentary on the current patch. 
> -
> Currently, all servers of an ensemble participate actively in reaching 
> agreement on the order of ZooKeeper transactions. That is, all followers 
> receive proposals, acknowledge them, and receive commit messages from the 
> leader. A leader issues commit messages once it receives acknowledgments from 
> a quorum of followers. For cross-colo operation, it would be useful to have a 
> third role: observer. Using Paxos terminology, observers are similar to 
> learners. An observer does not participate actively in the agreement step of 
> the atomic broadcast protocol. Instead, it only commits proposals that have 
> been accepted by some quorum of followers.
> One simple solution to implement observers is to have the leader forwarding 
> commit messages not only to followers but also to observers, and have 
> observers applying transactions according to the order followers agreed upon. 
> In the current implementation of the protocol, however, commit messages do 
> not carry their corresponding transaction payload because all servers 
> different from the leader are followers and followers receive such a payload 
> first through a proposal message. Just forwarding commit messages as they 
> currently are to an observer consequently is not sufficient. We have a couple 
> of options:
> 1- Include the transaction payload along in commit messages to observers;
> 2- Send proposals to observers as well.
> Number 2 is simpler to implement because it doesn't require changing the 
> protocol implementation, but it increases traffic slightly. The performance 
> impact due to such an increase might be insignificant, though.
> For scalability purposes, we may consider having followers also forwarding 
> commit messages to observers. With this option, observers can connect to 
> followers, and receive messages from followers. This choice is important to 
> avoid incre

[jira] Updated: (ZOOKEEPER-507) Improve error handling of BookKeeper client

2009-11-16 Thread Patrick Hunt (JIRA) 

 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patrick Hunt updated ZOOKEEPER-507:
---

Status: Open  (was: Patch Available)

> Improve error handling of BookKeeper client
> ---
>
> Key: ZOOKEEPER-507
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-507
> Project: Zookeeper
>  Issue Type: Improvement
>  Components: contrib-bookkeeper
>Reporter: Flavio Paiva Junqueira
>Assignee: Utkarsh Srivastava
> Attachments: ZOOKEEPER-507.patch
>
>
> Error handling is far from ideal currently in the BookKeeper client.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (ZOOKEEPER-507) Improve error handling of BookKeeper client

2009-11-16 Thread Patrick Hunt (JIRA) 

 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patrick Hunt updated ZOOKEEPER-507:
---

Status: Patch Available  (was: Open)

> Improve error handling of BookKeeper client
> ---
>
> Key: ZOOKEEPER-507
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-507
> Project: Zookeeper
>  Issue Type: Improvement
>  Components: contrib-bookkeeper
>Reporter: Flavio Paiva Junqueira
>Assignee: Utkarsh Srivastava
> Attachments: ZOOKEEPER-507.patch
>
>
> Error handling is far from ideal currently in the BookKeeper client.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (ZOOKEEPER-368) Observers: core functionality

2009-11-16 Thread Patrick Hunt (JIRA) 

 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patrick Hunt updated ZOOKEEPER-368:
---

Fix Version/s: 3.3.0
   Status: Patch Available  (was: Open)

> Observers: core functionality 
> --
>
> Key: ZOOKEEPER-368
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-368
> Project: Zookeeper
>  Issue Type: New Feature
>  Components: quorum
>Reporter: Flavio Paiva Junqueira
>Assignee: Henry Robinson
> Fix For: 3.3.0
>
> Attachments: obs-refactor.patch, observer-refactor.patch, observers 
> sync benchmark.png, observers.patch, ZOOKEEPER-368.patch, 
> ZOOKEEPER-368.patch, ZOOKEEPER-368.patch, ZOOKEEPER-368.patch, 
> ZOOKEEPER-368.patch, ZOOKEEPER-368.patch, ZOOKEEPER-368.patch, 
> ZOOKEEPER-368.patch, ZOOKEEPER-368.patch, ZOOKEEPER-368.patch
>
>
> Edit (Henry Robinson/henryr) 12/11/09:
> This JIRA specifically concerns the implementation of non-voting peers called 
> Observers, their documentation and their tests. 
> Explicit goals are 1. not breaking any current ZK functionality, 2. enabling 
> at least one deployment scenario involving Observers, 3. documentation 
> describing how to use the feature and 4. tests validating the correct 
> behaviour of 2. 
> Non goals of this JIRA are 1. performance optimizations specific to 
> Observers, 2. compatibility with every feature of ZooKeeper (in particular 
> all leader election protocols), which are both to be addressed in future 
> JIRAs. 
> See http://wiki.apache.org/hadoop/ZooKeeper/Observers for more detail of use 
> cases, proposed design and usage.
> See http://wiki.apache.org/hadoop/ZooKeeper/Observers/ReviewGuide for a brief 
> commentary on the current patch. 
> -
> Currently, all servers of an ensemble participate actively in reaching 
> agreement on the order of ZooKeeper transactions. That is, all followers 
> receive proposals, acknowledge them, and receive commit messages from the 
> leader. A leader issues commit messages once it receives acknowledgments from 
> a quorum of followers. For cross-colo operation, it would be useful to have a 
> third role: observer. Using Paxos terminology, observers are similar to 
> learners. An observer does not participate actively in the agreement step of 
> the atomic broadcast protocol. Instead, it only commits proposals that have 
> been accepted by some quorum of followers.
> One simple solution to implement observers is to have the leader forwarding 
> commit messages not only to followers but also to observers, and have 
> observers applying transactions according to the order followers agreed upon. 
> In the current implementation of the protocol, however, commit messages do 
> not carry their corresponding transaction payload because all servers 
> different from the leader are followers and followers receive such a payload 
> first through a proposal message. Just forwarding commit messages as they 
> currently are to an observer consequently is not sufficient. We have a couple 
> of options:
> 1- Include the transaction payload along in commit messages to observers;
> 2- Send proposals to observers as well.
> Number 2 is simpler to implement because it doesn't require changing the 
> protocol implementation, but it increases traffic slightly. The performance 
> impact due to such an increase might be insignificant, though.
> For scalability purposes, we may consider having followers also forwarding 
> commit messages to observers. With this option, observers can connect to 
> followers, and receive messages from followers. This choice is important to 
> avoid increasing the load on the leader with the number of observers. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (ZOOKEEPER-368) Observers: core functionality

2009-11-16 Thread Patrick Hunt (JIRA) 

 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patrick Hunt updated ZOOKEEPER-368:
---

Status: Open  (was: Patch Available)

> Observers: core functionality 
> --
>
> Key: ZOOKEEPER-368
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-368
> Project: Zookeeper
>  Issue Type: New Feature
>  Components: quorum
>Reporter: Flavio Paiva Junqueira
>Assignee: Henry Robinson
> Attachments: obs-refactor.patch, observer-refactor.patch, observers 
> sync benchmark.png, observers.patch, ZOOKEEPER-368.patch, 
> ZOOKEEPER-368.patch, ZOOKEEPER-368.patch, ZOOKEEPER-368.patch, 
> ZOOKEEPER-368.patch, ZOOKEEPER-368.patch, ZOOKEEPER-368.patch, 
> ZOOKEEPER-368.patch, ZOOKEEPER-368.patch, ZOOKEEPER-368.patch
>
>
> Edit (Henry Robinson/henryr) 12/11/09:
> This JIRA specifically concerns the implementation of non-voting peers called 
> Observers, their documentation and their tests. 
> Explicit goals are 1. not breaking any current ZK functionality, 2. enabling 
> at least one deployment scenario involving Observers, 3. documentation 
> describing how to use the feature and 4. tests validating the correct 
> behaviour of 2. 
> Non goals of this JIRA are 1. performance optimizations specific to 
> Observers, 2. compatibility with every feature of ZooKeeper (in particular 
> all leader election protocols), which are both to be addressed in future 
> JIRAs. 
> See http://wiki.apache.org/hadoop/ZooKeeper/Observers for more detail of use 
> cases, proposed design and usage.
> See http://wiki.apache.org/hadoop/ZooKeeper/Observers/ReviewGuide for a brief 
> commentary on the current patch. 
> -
> Currently, all servers of an ensemble participate actively in reaching 
> agreement on the order of ZooKeeper transactions. That is, all followers 
> receive proposals, acknowledge them, and receive commit messages from the 
> leader. A leader issues commit messages once it receives acknowledgments from 
> a quorum of followers. For cross-colo operation, it would be useful to have a 
> third role: observer. Using Paxos terminology, observers are similar to 
> learners. An observer does not participate actively in the agreement step of 
> the atomic broadcast protocol. Instead, it only commits proposals that have 
> been accepted by some quorum of followers.
> One simple solution to implement observers is to have the leader forwarding 
> commit messages not only to followers but also to observers, and have 
> observers applying transactions according to the order followers agreed upon. 
> In the current implementation of the protocol, however, commit messages do 
> not carry their corresponding transaction payload because all servers 
> different from the leader are followers and followers receive such a payload 
> first through a proposal message. Just forwarding commit messages as they 
> currently are to an observer consequently is not sufficient. We have a couple 
> of options:
> 1- Include the transaction payload along in commit messages to observers;
> 2- Send proposals to observers as well.
> Number 2 is simpler to implement because it doesn't require changing the 
> protocol implementation, but it increases traffic slightly. The performance 
> impact due to such an increase might be insignificant, though.
> For scalability purposes, we may consider having followers also forwarding 
> commit messages to observers. With this option, observers can connect to 
> followers, and receive messages from followers. This choice is important to 
> avoid increasing the load on the leader with the number of observers. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (ZOOKEEPER-3) syncLimit has slightly different comments in the class header, and > inline with the variable.

2009-11-16 Thread Hadoop QA (JIRA) 

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-3?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12778588#action_12778588
 ] 

Hadoop QA commented on ZOOKEEPER-3:
---

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12425142/ZOOKEEPER-3.patch
  against trunk revision 835618.

+1 @author.  The patch does not contain any @author tags.

-1 tests included.  The patch doesn't appear to include any new or modified 
tests.
Please justify why no tests are needed for this patch.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

+1 findbugs.  The patch does not introduce any new Findbugs warnings.

+1 release audit.  The applied patch does not increase the total number of 
release audit warnings.

+1 core tests.  The patch passed core unit tests.

+1 contrib tests.  The patch passed contrib unit tests.

Test results: 
http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/63/testReport/
Findbugs warnings: 
http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/63/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: 
http://hudson.zones.apache.org/hudson/job/Zookeeper-Patch-h8.grid.sp2.yahoo.net/63/console

This message is automatically generated.

>  syncLimit has slightly different comments in the class header, and > inline 
> with the variable.
> ---
>
> Key: ZOOKEEPER-3
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3
> Project: Zookeeper
>  Issue Type: Bug
>  Components: quorum
>Reporter: Benjamin Reed
>Assignee: Mahadev konar
>Priority: Trivial
> Fix For: 3.3.0
>
> Attachments: ZOOKEEPER-3.patch
>
>
> syncLimit as documented in QuorumPeer is documented twice with two different 
> aspects of in each instance. It should be better documented and unified. 
> (Probably remove the second instance.)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Hudson build is back to normal: ZooKeeper-trunk #541

2009-11-16 Thread Apache Hudson Server 
See

Build failed in Hudson: ZooKeeper-trunk #540

2009-11-16 Thread Apache Hudson Server 
See 

--
[...truncated 80891 lines...]
[junit] 2009-11-16 21:19:09,462 - INFO  
[NIOServerCxn.Factory:11225:nioservercnxn$fact...@214] - Accepted socket 
connection from /127.0.0.1:52979
[junit] 2009-11-16 21:19:09,462 - INFO  
[main-SendThread(localhost:11225):clientcnxn$sendthr...@837] - Socket 
connection established to localhost/127.0.0.1:11225, initiating session
[junit] 2009-11-16 21:19:09,463 - INFO  
[NIOServerCxn.Factory:11225:nioserverc...@688] - Client attempting to renew 
session 0x124fedcff7f at /127.0.0.1:52979
[junit] 2009-11-16 21:19:09,463 - INFO  
[NIOServerCxn.Factory:11225:nioserverc...@1131] - Established session 
0x124fedcff7f for client /127.0.0.1:52979
[junit] 2009-11-16 21:19:09,463 - INFO  
[main-SendThread(localhost:11225):clientcnxn$sendthr...@640] - Session 
establishment complete, sessionid = 0x124fedcff7f
[junit] 2009-11-16 21:19:09,471 - INFO  [main:clientb...@376] - STOPPING 
server
[junit] 2009-11-16 21:19:09,472 - INFO  [main:nioserverc...@974] - Closed 
socket connection for client /127.0.0.1:52979 which had sessionid 
0x124fedcff7f
[junit] 2009-11-16 21:19:09,472 - INFO  
[main-SendThread(localhost:11225):clientcnxn$sendthr...@1047] - Unable to read 
additional data from server sessionid 0x124fedcff7f, likely server has 
closed socket, closing socket connection and attempting reconnect
[junit] 2009-11-16 21:19:09,472 - INFO  
[NIOServerCxn.Factory:11225:nioservercnxn$fact...@241] - NIOServerCnxn factory 
exited run method
[junit] 2009-11-16 21:19:09,473 - INFO  [main:finalrequestproces...@365] - 
shutdown of request processor complete
[junit] 2009-11-16 21:19:09,473 - INFO  
[SyncThread:0:syncrequestproces...@151] - SyncRequestProcessor exited!
[junit] 2009-11-16 21:19:09,473 - INFO  
[ProcessThread:-1:preprequestproces...@119] - PrepRequestProcessor exited loop!
[junit] ensureOnly:[]
[junit] 2009-11-16 21:19:09,572 - INFO  [main:clientb...@369] - STARTING 
server
[junit] 2009-11-16 21:19:09,573 - INFO  [main:zookeeperser...@160] - 
Created server
[junit] 2009-11-16 21:19:09,573 - INFO  [main:nioservercnxn$fact...@123] - 
binding to port 11225
[junit] 2009-11-16 21:19:09,575 - INFO  [main:files...@81] - Reading 
snapshot 

[junit] 2009-11-16 21:19:09,577 - INFO  [main:filetxnsnap...@208] - 
Snapshotting: 6
[junit] 2009-11-16 21:19:09,579 - INFO  
[NIOServerCxn.Factory:11225:nioservercnxn$fact...@214] - Accepted socket 
connection from /127.0.0.1:52981
[junit] 2009-11-16 21:19:09,579 - INFO  
[NIOServerCxn.Factory:11225:nioserverc...@782] - Processing stat command from 
/127.0.0.1:52981
[junit] 2009-11-16 21:19:09,580 - INFO  
[NIOServerCxn.Factory:11225:nioserverc...@974] - Closed socket connection for 
client /127.0.0.1:52981 (no session established for client)
[junit] ensureOnly:[InMemoryDataTree, StandaloneServer_port]
[junit] expect:InMemoryDataTree
[junit] found:InMemoryDataTree 
org.apache.ZooKeeperService:name0=StandaloneServer_port-1,name1=InMemoryDataTree
[junit] expect:StandaloneServer_port
[junit] found:StandaloneServer_port 
org.apache.ZooKeeperService:name0=StandaloneServer_port-1
[junit] 2009-11-16 21:19:10,747 - INFO  
[main-SendThread(localhost:11225):clientcnxn$sendthr...@929] - Opening socket 
connection to server localhost/127.0.0.1:11225
[junit] 2009-11-16 21:19:10,747 - INFO  
[main-SendThread(localhost:11225):clientcnxn$sendthr...@837] - Socket 
connection established to localhost/127.0.0.1:11225, initiating session
[junit] 2009-11-16 21:19:10,747 - INFO  
[NIOServerCxn.Factory:11225:nioservercnxn$fact...@214] - Accepted socket 
connection from /127.0.0.1:52982
[junit] 2009-11-16 21:19:10,748 - INFO  
[NIOServerCxn.Factory:11225:nioserverc...@688] - Client attempting to renew 
session 0x124fedcff7f at /127.0.0.1:52982
[junit] 2009-11-16 21:19:10,748 - INFO  
[NIOServerCxn.Factory:11225:nioserverc...@1131] - Established session 
0x124fedcff7f for client /127.0.0.1:52982
[junit] 2009-11-16 21:19:10,748 - INFO  
[main-SendThread(localhost:11225):clientcnxn$sendthr...@640] - Session 
establishment complete, sessionid = 0x124fedcff7f
[junit] 2009-11-16 21:19:11,756 - INFO  
[ProcessThread:-1:preprequestproces...@385] - Processed session termination for 
sessionid: 0x124fedcff7f
[junit] 2009-11-16 21:19:11,757 - INFO  [main:zookee...@524] - Session: 
0x124fedcff7f closed
[junit] 2009-11-16 21:19:11,758 - INFO  [main:clientb...@399] - tearDown 
starting
[junit] 2009-11-16 21:19:11,758 - INFO  
[NIOServerCxn.Factory:11225:nioserverc...@974] - Closed socket connection for 
client /127.0.0.1:52982 which had sessionid 0x124fedcff7f
[ju

[jira] Updated: (ZOOKEEPER-3) syncLimit has slightly different comments in the class header, and > inline with the variable.

2009-11-16 Thread Mahadev konar (JIRA) 

 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-3?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mahadev konar updated ZOOKEEPER-3:
--

Attachment: ZOOKEEPER-3.patch

this patch removes redundant javadocs. Very trivial patch. no tests.

>  syncLimit has slightly different comments in the class header, and > inline 
> with the variable.
> ---
>
> Key: ZOOKEEPER-3
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3
> Project: Zookeeper
>  Issue Type: Bug
>  Components: quorum
>Reporter: Benjamin Reed
>Priority: Trivial
> Fix For: 3.3.0
>
> Attachments: ZOOKEEPER-3.patch
>
>
> syncLimit as documented in QuorumPeer is documented twice with two different 
> aspects of in each instance. It should be better documented and unified. 
> (Probably remove the second instance.)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Assigned: (ZOOKEEPER-3) syncLimit has slightly different comments in the class header, and > inline with the variable.

2009-11-16 Thread Mahadev konar (JIRA) 

 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-3?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mahadev konar reassigned ZOOKEEPER-3:
-

Assignee: Mahadev konar

>  syncLimit has slightly different comments in the class header, and > inline 
> with the variable.
> ---
>
> Key: ZOOKEEPER-3
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3
> Project: Zookeeper
>  Issue Type: Bug
>  Components: quorum
>Reporter: Benjamin Reed
>Assignee: Mahadev konar
>Priority: Trivial
> Fix For: 3.3.0
>
> Attachments: ZOOKEEPER-3.patch
>
>
> syncLimit as documented in QuorumPeer is documented twice with two different 
> aspects of in each instance. It should be better documented and unified. 
> (Probably remove the second instance.)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (ZOOKEEPER-3) syncLimit has slightly different comments in the class header, and > inline with the variable.

2009-11-16 Thread Mahadev konar (JIRA) 

 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-3?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mahadev konar updated ZOOKEEPER-3:
--

Status: Patch Available  (was: Open)

>  syncLimit has slightly different comments in the class header, and > inline 
> with the variable.
> ---
>
> Key: ZOOKEEPER-3
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-3
> Project: Zookeeper
>  Issue Type: Bug
>  Components: quorum
>Reporter: Benjamin Reed
>Assignee: Mahadev konar
>Priority: Trivial
> Fix For: 3.3.0
>
> Attachments: ZOOKEEPER-3.patch
>
>
> syncLimit as documented in QuorumPeer is documented twice with two different 
> aspects of in each instance. It should be better documented and unified. 
> (Probably remove the second instance.)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Resolved: (ZOOKEEPER-158) Leader and followers increase cpu utilization upon loss of a follower

2009-11-16 Thread Mahadev konar (JIRA) 

 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mahadev konar resolved ZOOKEEPER-158.
-

Resolution: Fixed

this issue was same as ZOOKEEPER-427. closing it as resolved.

> Leader and followers increase cpu utilization upon loss of a follower
> -
>
> Key: ZOOKEEPER-158
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-158
> Project: Zookeeper
>  Issue Type: Improvement
>  Components: quorum
>Reporter: Flavio Paiva Junqueira
>Assignee: Mahadev konar
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: dead-follower.tar.gz
>
>
> In a set of ZooKeeper servers, when there is a leader operation and supported 
> by a quorum of servers, we have observed that cpu utilization increases 
> substantially once a follower fails or disconnects. Stu Hood provided logs 
> showing this behavior.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (ZOOKEEPER-368) Observers: core functionality

2009-11-16 Thread Mahadev konar (JIRA) 

[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-368?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12778447#action_12778447
 ] 

Mahadev konar commented on ZOOKEEPER-368:
-

henry,
 you dont have to upload the diff the built docs (pdf/html). THe person 
committing the patch will run ant docs and commit the updated docs in src/docs.



> Observers: core functionality 
> --
>
> Key: ZOOKEEPER-368
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-368
> Project: Zookeeper
>  Issue Type: New Feature
>  Components: quorum
>Reporter: Flavio Paiva Junqueira
>Assignee: Henry Robinson
> Attachments: obs-refactor.patch, observer-refactor.patch, observers 
> sync benchmark.png, observers.patch, ZOOKEEPER-368.patch, 
> ZOOKEEPER-368.patch, ZOOKEEPER-368.patch, ZOOKEEPER-368.patch, 
> ZOOKEEPER-368.patch, ZOOKEEPER-368.patch, ZOOKEEPER-368.patch, 
> ZOOKEEPER-368.patch, ZOOKEEPER-368.patch, ZOOKEEPER-368.patch
>
>
> Edit (Henry Robinson/henryr) 12/11/09:
> This JIRA specifically concerns the implementation of non-voting peers called 
> Observers, their documentation and their tests. 
> Explicit goals are 1. not breaking any current ZK functionality, 2. enabling 
> at least one deployment scenario involving Observers, 3. documentation 
> describing how to use the feature and 4. tests validating the correct 
> behaviour of 2. 
> Non goals of this JIRA are 1. performance optimizations specific to 
> Observers, 2. compatibility with every feature of ZooKeeper (in particular 
> all leader election protocols), which are both to be addressed in future 
> JIRAs. 
> See http://wiki.apache.org/hadoop/ZooKeeper/Observers for more detail of use 
> cases, proposed design and usage.
> See http://wiki.apache.org/hadoop/ZooKeeper/Observers/ReviewGuide for a brief 
> commentary on the current patch. 
> -
> Currently, all servers of an ensemble participate actively in reaching 
> agreement on the order of ZooKeeper transactions. That is, all followers 
> receive proposals, acknowledge them, and receive commit messages from the 
> leader. A leader issues commit messages once it receives acknowledgments from 
> a quorum of followers. For cross-colo operation, it would be useful to have a 
> third role: observer. Using Paxos terminology, observers are similar to 
> learners. An observer does not participate actively in the agreement step of 
> the atomic broadcast protocol. Instead, it only commits proposals that have 
> been accepted by some quorum of followers.
> One simple solution to implement observers is to have the leader forwarding 
> commit messages not only to followers but also to observers, and have 
> observers applying transactions according to the order followers agreed upon. 
> In the current implementation of the protocol, however, commit messages do 
> not carry their corresponding transaction payload because all servers 
> different from the leader are followers and followers receive such a payload 
> first through a proposal message. Just forwarding commit messages as they 
> currently are to an observer consequently is not sufficient. We have a couple 
> of options:
> 1- Include the transaction payload along in commit messages to observers;
> 2- Send proposals to observers as well.
> Number 2 is simpler to implement because it doesn't require changing the 
> protocol implementation, but it increases traffic slightly. The performance 
> impact due to such an increase might be insignificant, though.
> For scalability purposes, we may consider having followers also forwarding 
> commit messages to observers. With this option, observers can connect to 
> followers, and receive messages from followers. This choice is important to 
> avoid increasing the load on the leader with the number of observers. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Build failed in Hudson: ZooKeeper-trunk #539

2009-11-16 Thread Apache Hudson Server 
See 

--
[...truncated 82279 lines...]
[junit] 2009-11-16 18:15:40,426 - INFO  [main:finalrequestproces...@365] - 
shutdown of request processor complete
[junit] 2009-11-16 18:15:40,426 - INFO  
[ProcessThread:-1:preprequestproces...@119] - PrepRequestProcessor exited loop!
[junit] 2009-11-16 18:15:40,426 - INFO  
[SyncThread:0:syncrequestproces...@151] - SyncRequestProcessor exited!
[junit] ensureOnly:[]
[junit] 2009-11-16 18:15:40,525 - INFO  [main:clientb...@369] - STARTING 
server
[junit] 2009-11-16 18:15:40,526 - INFO  [main:zookeeperser...@160] - 
Created server
[junit] 2009-11-16 18:15:40,526 - INFO  [main:nioservercnxn$fact...@123] - 
binding to port 11225
[junit] 2009-11-16 18:15:40,528 - INFO  [main:files...@81] - Reading 
snapshot 

[junit] 2009-11-16 18:15:40,530 - INFO  [main:filetxnsnap...@208] - 
Snapshotting: 6
[junit] 2009-11-16 18:15:40,532 - INFO  
[NIOServerCxn.Factory:11225:nioservercnxn$fact...@214] - Accepted socket 
connection from /127.0.0.1:37425
[junit] 2009-11-16 18:15:40,532 - INFO  
[NIOServerCxn.Factory:11225:nioserverc...@782] - Processing stat command from 
/127.0.0.1:37425
[junit] 2009-11-16 18:15:40,533 - INFO  
[NIOServerCxn.Factory:11225:nioserverc...@974] - Closed socket connection for 
client /127.0.0.1:37425 (no session established for client)
[junit] ensureOnly:[InMemoryDataTree, StandaloneServer_port]
[junit] expect:InMemoryDataTree
[junit] found:InMemoryDataTree 
org.apache.ZooKeeperService:name0=StandaloneServer_port-1,name1=InMemoryDataTree
[junit] expect:StandaloneServer_port
[junit] found:StandaloneServer_port 
org.apache.ZooKeeperService:name0=StandaloneServer_port-1
[junit] 2009-11-16 18:15:42,000 - INFO  
[SessionTracker:sessiontrackeri...@145] - SessionTrackerImpl exited loop!
[junit] 2009-11-16 18:15:42,024 - INFO  
[main-SendThread(localhost:11225):clientcnxn$sendthr...@929] - Opening socket 
connection to server localhost/127.0.0.1:11225
[junit] 2009-11-16 18:15:42,024 - INFO  
[main-SendThread(localhost:11225):clientcnxn$sendthr...@837] - Socket 
connection established to localhost/127.0.0.1:11225, initiating session
[junit] 2009-11-16 18:15:42,024 - INFO  
[NIOServerCxn.Factory:11225:nioservercnxn$fact...@214] - Accepted socket 
connection from /127.0.0.1:37426
[junit] 2009-11-16 18:15:42,024 - INFO  
[NIOServerCxn.Factory:11225:nioserverc...@688] - Client attempting to renew 
session 0x124fe3500a1 at /127.0.0.1:37426
[junit] 2009-11-16 18:15:42,025 - INFO  
[NIOServerCxn.Factory:11225:nioserverc...@1131] - Established session 
0x124fe3500a1 for client /127.0.0.1:37426
[junit] 2009-11-16 18:15:42,025 - INFO  
[main-SendThread(localhost:11225):clientcnxn$sendthr...@640] - Session 
establishment complete, sessionid = 0x124fe3500a1
[junit] 2009-11-16 18:15:43,033 - INFO  
[ProcessThread:-1:preprequestproces...@385] - Processed session termination for 
sessionid: 0x124fe3500a1
[junit] 2009-11-16 18:15:43,034 - INFO  [main:zookee...@524] - Session: 
0x124fe3500a1 closed
[junit] 2009-11-16 18:15:43,034 - INFO  
[NIOServerCxn.Factory:11225:nioserverc...@974] - Closed socket connection for 
client /127.0.0.1:37426 which had sessionid 0x124fe3500a1
[junit] 2009-11-16 18:15:43,035 - INFO  [main:clientb...@399] - tearDown 
starting
[junit] 2009-11-16 18:15:43,036 - INFO  [main:clientb...@411] - fdcount 
after test is: 46
[junit] 2009-11-16 18:15:43,036 - INFO  [main:clientb...@376] - STOPPING 
server
[junit] 2009-11-16 18:15:43,037 - INFO  
[NIOServerCxn.Factory:11225:nioservercnxn$fact...@241] - NIOServerCnxn factory 
exited run method
[junit] 2009-11-16 18:15:43,037 - INFO  [main:finalrequestproces...@365] - 
shutdown of request processor complete
[junit] 2009-11-16 18:15:43,037 - INFO  
[SyncThread:0:syncrequestproces...@151] - SyncRequestProcessor exited!
[junit] 2009-11-16 18:15:43,037 - INFO  
[ProcessThread:-1:preprequestproces...@119] - PrepRequestProcessor exited loop!
[junit] ensureOnly:[]
[junit] 2009-11-16 18:15:43,040 - INFO  [main:clientb...@430] - FINISHED 
testWatcherAutoResetDisabledWithGlobal
[junit] 2009-11-16 18:15:43,041 - INFO  [main:clientb...@340] - STARTING 
testWatcherAutoResetDisabledWithLocal
[junit] 2009-11-16 18:15:43,047 - INFO  [main:clientb...@369] - STARTING 
server
[junit] 2009-11-16 18:15:43,048 - INFO  [main:zookeeperser...@160] - 
Created server
[junit] 2009-11-16 18:15:43,048 - INFO  [main:nioservercnxn$fact...@123] - 
binding to port 11226
[junit] 2009-11-16 18:15:43,049 - INFO  [main:filetxnsnap...@208] - 
Snapshotting: 0
[junit] 2009-11-16 18:15:43,051 - INFO  
[NIOServerCxn.Factory:11226:nioser

[jira] Updated: (ZOOKEEPER-368) Observers: core functionality

2009-11-16 Thread Henry Robinson (JIRA) 

 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henry Robinson updated ZOOKEEPER-368:
-

Attachment: ZOOKEEPER-368.patch

Docs update: added explicit mention of need to set electionAlg=0. 

> Observers: core functionality 
> --
>
> Key: ZOOKEEPER-368
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-368
> Project: Zookeeper
>  Issue Type: New Feature
>  Components: quorum
>Reporter: Flavio Paiva Junqueira
>Assignee: Henry Robinson
> Attachments: obs-refactor.patch, observer-refactor.patch, observers 
> sync benchmark.png, observers.patch, ZOOKEEPER-368.patch, 
> ZOOKEEPER-368.patch, ZOOKEEPER-368.patch, ZOOKEEPER-368.patch, 
> ZOOKEEPER-368.patch, ZOOKEEPER-368.patch, ZOOKEEPER-368.patch, 
> ZOOKEEPER-368.patch, ZOOKEEPER-368.patch, ZOOKEEPER-368.patch
>
>
> Edit (Henry Robinson/henryr) 12/11/09:
> This JIRA specifically concerns the implementation of non-voting peers called 
> Observers, their documentation and their tests. 
> Explicit goals are 1. not breaking any current ZK functionality, 2. enabling 
> at least one deployment scenario involving Observers, 3. documentation 
> describing how to use the feature and 4. tests validating the correct 
> behaviour of 2. 
> Non goals of this JIRA are 1. performance optimizations specific to 
> Observers, 2. compatibility with every feature of ZooKeeper (in particular 
> all leader election protocols), which are both to be addressed in future 
> JIRAs. 
> See http://wiki.apache.org/hadoop/ZooKeeper/Observers for more detail of use 
> cases, proposed design and usage.
> See http://wiki.apache.org/hadoop/ZooKeeper/Observers/ReviewGuide for a brief 
> commentary on the current patch. 
> -
> Currently, all servers of an ensemble participate actively in reaching 
> agreement on the order of ZooKeeper transactions. That is, all followers 
> receive proposals, acknowledge them, and receive commit messages from the 
> leader. A leader issues commit messages once it receives acknowledgments from 
> a quorum of followers. For cross-colo operation, it would be useful to have a 
> third role: observer. Using Paxos terminology, observers are similar to 
> learners. An observer does not participate actively in the agreement step of 
> the atomic broadcast protocol. Instead, it only commits proposals that have 
> been accepted by some quorum of followers.
> One simple solution to implement observers is to have the leader forwarding 
> commit messages not only to followers but also to observers, and have 
> observers applying transactions according to the order followers agreed upon. 
> In the current implementation of the protocol, however, commit messages do 
> not carry their corresponding transaction payload because all servers 
> different from the leader are followers and followers receive such a payload 
> first through a proposal message. Just forwarding commit messages as they 
> currently are to an observer consequently is not sufficient. We have a couple 
> of options:
> 1- Include the transaction payload along in commit messages to observers;
> 2- Send proposals to observers as well.
> Number 2 is simpler to implement because it doesn't require changing the 
> protocol implementation, but it increases traffic slightly. The performance 
> impact due to such an increase might be insignificant, though.
> For scalability purposes, we may consider having followers also forwarding 
> commit messages to observers. With this option, observers can connect to 
> followers, and receive messages from followers. This choice is important to 
> avoid increasing the load on the leader with the number of observers. 

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Build failed in Hudson: ZooKeeper-trunk #538

2009-11-16 Thread Apache Hudson Server 
See 

--
[...truncated 80759 lines...]
[junit] 2009-11-16 10:50:23,583 - INFO  
[main-SendThread(localhost:11225):clientcnxn$sendthr...@837] - Socket 
connection established to localhost/127.0.0.1:11225, initiating session
[junit] 2009-11-16 10:50:23,583 - INFO  
[NIOServerCxn.Factory:11225:nioservercnxn$fact...@214] - Accepted socket 
connection from /127.0.0.1:49551
[junit] 2009-11-16 10:50:23,584 - INFO  
[NIOServerCxn.Factory:11225:nioserverc...@688] - Client attempting to renew 
session 0x124fc9d55fc at /127.0.0.1:49551
[junit] 2009-11-16 10:50:23,584 - INFO  
[NIOServerCxn.Factory:11225:nioserverc...@1131] - Established session 
0x124fc9d55fc for client /127.0.0.1:49551
[junit] 2009-11-16 10:50:23,584 - INFO  
[main-SendThread(localhost:11225):clientcnxn$sendthr...@640] - Session 
establishment complete, sessionid = 0x124fc9d55fc
[junit] 2009-11-16 10:50:23,595 - INFO  [main:clientb...@376] - STOPPING 
server
[junit] 2009-11-16 10:50:23,596 - INFO  [main:nioserverc...@974] - Closed 
socket connection for client /127.0.0.1:49551 which had sessionid 
0x124fc9d55fc
[junit] 2009-11-16 10:50:23,596 - INFO  
[main-SendThread(localhost:11225):clientcnxn$sendthr...@1047] - Unable to read 
additional data from server sessionid 0x124fc9d55fc, likely server has 
closed socket, closing socket connection and attempting reconnect
[junit] 2009-11-16 10:50:23,597 - INFO  
[NIOServerCxn.Factory:11225:nioservercnxn$fact...@241] - NIOServerCnxn factory 
exited run method
[junit] 2009-11-16 10:50:23,597 - INFO  [main:finalrequestproces...@365] - 
shutdown of request processor complete
[junit] 2009-11-16 10:50:23,597 - INFO  
[SyncThread:0:syncrequestproces...@151] - SyncRequestProcessor exited!
[junit] 2009-11-16 10:50:23,597 - INFO  
[ProcessThread:-1:preprequestproces...@119] - PrepRequestProcessor exited loop!
[junit] ensureOnly:[]
[junit] 2009-11-16 10:50:23,697 - INFO  [main:clientb...@369] - STARTING 
server
[junit] 2009-11-16 10:50:23,697 - INFO  [main:zookeeperser...@160] - 
Created server
[junit] 2009-11-16 10:50:23,698 - INFO  [main:nioservercnxn$fact...@123] - 
binding to port 11225
[junit] 2009-11-16 10:50:23,699 - INFO  [main:files...@81] - Reading 
snapshot 

[junit] 2009-11-16 10:50:23,702 - INFO  [main:filetxnsnap...@208] - 
Snapshotting: 6
[junit] 2009-11-16 10:50:23,703 - INFO  
[NIOServerCxn.Factory:11225:nioservercnxn$fact...@214] - Accepted socket 
connection from /127.0.0.1:49553
[junit] 2009-11-16 10:50:23,704 - INFO  
[NIOServerCxn.Factory:11225:nioserverc...@782] - Processing stat command from 
/127.0.0.1:49553
[junit] 2009-11-16 10:50:23,704 - INFO  
[NIOServerCxn.Factory:11225:nioserverc...@974] - Closed socket connection for 
client /127.0.0.1:49553 (no session established for client)
[junit] ensureOnly:[InMemoryDataTree, StandaloneServer_port]
[junit] expect:InMemoryDataTree
[junit] found:InMemoryDataTree 
org.apache.ZooKeeperService:name0=StandaloneServer_port-1,name1=InMemoryDataTree
[junit] expect:StandaloneServer_port
[junit] found:StandaloneServer_port 
org.apache.ZooKeeperService:name0=StandaloneServer_port-1
[junit] 2009-11-16 10:50:24,000 - INFO  
[SessionTracker:sessiontrackeri...@145] - SessionTrackerImpl exited loop!
[junit] 2009-11-16 10:50:24,001 - INFO  
[SessionTracker:sessiontrackeri...@145] - SessionTrackerImpl exited loop!
[junit] 2009-11-16 10:50:25,438 - INFO  
[main-SendThread(localhost:11225):clientcnxn$sendthr...@929] - Opening socket 
connection to server localhost/127.0.0.1:11225
[junit] 2009-11-16 10:50:25,438 - INFO  
[main-SendThread(localhost:11225):clientcnxn$sendthr...@837] - Socket 
connection established to localhost/127.0.0.1:11225, initiating session
[junit] 2009-11-16 10:50:25,438 - INFO  
[NIOServerCxn.Factory:11225:nioservercnxn$fact...@214] - Accepted socket 
connection from /127.0.0.1:49554
[junit] 2009-11-16 10:50:25,439 - INFO  
[NIOServerCxn.Factory:11225:nioserverc...@688] - Client attempting to renew 
session 0x124fc9d55fc at /127.0.0.1:49554
[junit] 2009-11-16 10:50:25,439 - INFO  
[NIOServerCxn.Factory:11225:nioserverc...@1131] - Established session 
0x124fc9d55fc for client /127.0.0.1:49554
[junit] 2009-11-16 10:50:25,440 - INFO  
[main-SendThread(localhost:11225):clientcnxn$sendthr...@640] - Session 
establishment complete, sessionid = 0x124fc9d55fc
[junit] 2009-11-16 10:50:26,453 - INFO  
[ProcessThread:-1:preprequestproces...@385] - Processed session termination for 
sessionid: 0x124fc9d55fc
[junit] 2009-11-16 10:50:26,454 - INFO  [main:zookee...@524] - Session: 
0x124fc9d55fc closed
[junit] 2009-11-16 10:50:26,454 - IN