from:"Patrick Hunt"


 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patrick Hunt reassigned ZOOKEEPER-880:
--

Assignee: Vishal K

 QuorumCnxManager$SendWorker grows without bounds
 

 Key: ZOOKEEPER-880
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-880
 Project: Zookeeper
  Issue Type: Bug
Affects Versions: 3.2.2
Reporter: Jean-Daniel Cryans
Assignee: Vishal K
Priority: Critical
 Attachments: hbase-hadoop-zookeeper-sv4borg12.log.gz, 
 hbase-hadoop-zookeeper-sv4borg9.log.gz, jstack, 
 TRACE-hbase-hadoop-zookeeper-sv4borg9.log.gz, ZOOKEEPER-880.patch


 We're seeing an issue where one server in the ensemble has a steady growing 
 number of QuorumCnxManager$SendWorker threads up to a point where the OS runs 
 out of native threads, and at the same time we see a lot of exceptions in the 
 logs.  This is on 3.2.2 and our config looks like:
 {noformat}
 tickTime=3000
 dataDir=/somewhere_thats_not_tmp
 clientPort=2181
 initLimit=10
 syncLimit=5
 server.0=sv4borg9:2888:3888
 server.1=sv4borg10:2888:3888
 server.2=sv4borg11:2888:3888
 server.3=sv4borg12:2888:3888
 server.4=sv4borg13:2888:3888
 {noformat}
 The issue is on the first server. I'm going to attach threads dumps and logs 
 in moment.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (ZOOKEEPER-880) QuorumCnxManager$SendWorker grows without bounds


 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-880?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patrick Hunt updated ZOOKEEPER-880:
---

Fix Version/s: 3.4.0
   3.3.3
   Status: Open  (was: Patch Available)

We really should have a test for this case. Vishal can you add it? (the more 
the better)

 QuorumCnxManager$SendWorker grows without bounds
 

 Key: ZOOKEEPER-880
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-880
 Project: Zookeeper
  Issue Type: Bug
Affects Versions: 3.2.2
Reporter: Jean-Daniel Cryans
Assignee: Vishal K
Priority: Critical
 Fix For: 3.3.3, 3.4.0

 Attachments: hbase-hadoop-zookeeper-sv4borg12.log.gz, 
 hbase-hadoop-zookeeper-sv4borg9.log.gz, jstack, 
 TRACE-hbase-hadoop-zookeeper-sv4borg9.log.gz, ZOOKEEPER-880.patch


 We're seeing an issue where one server in the ensemble has a steady growing 
 number of QuorumCnxManager$SendWorker threads up to a point where the OS runs 
 out of native threads, and at the same time we see a lot of exceptions in the 
 logs.  This is on 3.2.2 and our config looks like:
 {noformat}
 tickTime=3000
 dataDir=/somewhere_thats_not_tmp
 clientPort=2181
 initLimit=10
 syncLimit=5
 server.0=sv4borg9:2888:3888
 server.1=sv4borg10:2888:3888
 server.2=sv4borg11:2888:3888
 server.3=sv4borg12:2888:3888
 server.4=sv4borg13:2888:3888
 {noformat}
 The issue is on the first server. I'm going to attach threads dumps and logs 
 in moment.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

ZooKeeper approved by Apache Board as TLP!

2010-11-22 Thread Patrick Hunt

We are now officially an Apache TLP! http://bit.ly/9czN2x

As part of the process for moving out from under Hadoop and into full
TLP status we need to work through the following:
http://incubator.apache.org/guides/graduation.html#new-project-hand-over
If you are involved with the project, esp on the dev side, please
review these sections. Notice that a number of things will be
changing; mailinglist, source repo, wiki, etc...

I'll be sending out updates as we work through these, regards and
Congratulations everyone!

Patrick

[jira] Updated: (ZOOKEEPER-925) Consider maven site generation to replace our forrest site and documentation generation

[
https://issues.apache.org/jira/browse/ZOOKEEPER-925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Patrick Hunt updated ZOOKEEPER-925:
---

Attachment: ZOOKEEPER-925.patch

This updated patch has a db2confluence.py script which attempts to convert our
docs from simpledocbook to confluence format. I converted the admin guide as an
example.

Try running mvn site then open the generated html file for admin (in target
directory)

Note:

1) doxia confluence doesn't support toc generation, so we'd need to maintain
this for the time being until they implement
2) the note sections are not supported. We'd have to reformat this a bit
ourselves to make it work (probably by specifying a css class that provides
similar style)
3) xrefs in sdocbook will pull in the text of the referenced section, the
conversion tool does not do this. We'd have to add this as part of the
conversion.

Take a look ant let me know. Anyone could run the conversion on the other docs
to see how it works there (I only ran on admin). Use db2confluence.py (redirect
stdout) to test on the other files.

The python script (included in patch) is pretty simple to tweak if we notice
other elements that are not being converted (correctly).

Consider maven site generation to replace our forrest site and documentation
generation
---

Key: ZOOKEEPER-925
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-925
Project: Zookeeper
Issue Type: Wish
Components: documentation
Reporter: Patrick Hunt
Assignee: Patrick Hunt
Attachments: ZOOKEEPER-925.patch, ZOOKEEPER-925.patch

See WHIRR-19 for some background.
In whirr we looked at a number of site/doc generation facilities. In the end
Maven site generation plugin turned out to be by far the best option. You can
see our nascent site here (no attempt at styling,etc so far):
http://incubator.apache.org/whirr/
In particular take a look at the quick start:
http://incubator.apache.org/whirr/quick-start-guide.html
which was generated from
http://svn.apache.org/repos/asf/incubator/whirr/trunk/src/site/confluence/quick-start-guide.confluence
notice this was standard wiki markup (confluence wiki markup, same as
available from apache)
You can read more about mvn site plugin here:
http://maven.apache.org/guides/mini/guide-site.html
Notice that other formats are available, not just confluence markup, also
note that you can use different markup formats if you like in the same site
(although probably not a great idea, but in some cases might be handy, for
example whirr uses the confluence wiki, so we can pretty much copy/paste
source docs from wiki to our site (svn) if we like)
Re maven vs our current ant based build. It's probably a good idea for us to
move the build to maven at some point. We could initially move just the doc
generation, and then incrementally move functionality from build.xml to mvn
over a longer time period.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (ZOOKEEPER-756) some cleanup and improvements for zooinspector

[
https://issues.apache.org/jira/browse/ZOOKEEPER-756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Patrick Hunt updated ZOOKEEPER-756:
---

Status: Open (was: Patch Available)

Colin, looks like you are having some weird line ending problem... patch is not
applying.

some cleanup and improvements for zooinspector
--

Key: ZOOKEEPER-756
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-756
Project: Zookeeper
Issue Type: Improvement
Components: contrib
Affects Versions: 3.3.0
Reporter: Thomas Koch
Assignee: Colin Goodheart-Smithe
Fix For: 3.4.0

Attachments: Zooinspector-patch.patch, zooInspectorChanges.patch,
zooInspectorChanges.patch, ZOOKEEPER-756.patch

Copied from the already closed ZOOKEEPER-678:
* specify the exact URL, where the icons are from. It's best to include the
link also in the NOTICE.txt file.
It seems, that zooinspector finds it's icons only if the icons folder is in
the current path. But when I install zooinspector as part of the Zookeeper
Debian package, I want to be able to call it regardless of the current path.
Could you use getRessources or something so that I can point to the icons
location from the wrapper shell script?
Can I place the zooinspector config files in /etc/zookeeper/zooinspector/ ?
Could I give zooinspector a property to point to the config file location?
There are several places, where viewers is missspelled as Veiwers. Please
do a case insensitive search for veiw to correct these. Even the config
file defaultNodeVeiwers.cfg is missspelled like this. This has the
potential to confuse the hell out of people when debugging something!

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

FYI: ZooKeeper's move to TLP is in progress

2010-11-22 Thread Patrick Hunt

The following jira has been created to get the process started:

https://issues.apache.org/jira/browse/INFRA-3228

Patrick

[jira] Commented: (ZOOKEEPER-880) QuorumCnxManager$SendWorker grows without bounds

2010-11-19 Thread Patrick Hunt (JIRA)


[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12933891#action_12933891
 ] 

Patrick Hunt commented on ZOOKEEPER-880:


Flavio (and others) we should update the docs to include details on which ports 
can/should be monitored, and which ports should NOT be monitored (or if 
monitoring is supported any conditions).

Can we update the docs as part of any patch/fix? Thanks.

 QuorumCnxManager$SendWorker grows without bounds
 

 Key: ZOOKEEPER-880
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-880
 Project: Zookeeper
  Issue Type: Bug
Affects Versions: 3.2.2
Reporter: Jean-Daniel Cryans
Priority: Critical
 Attachments: hbase-hadoop-zookeeper-sv4borg12.log.gz, 
 hbase-hadoop-zookeeper-sv4borg9.log.gz, jstack, 
 TRACE-hbase-hadoop-zookeeper-sv4borg9.log.gz


 We're seeing an issue where one server in the ensemble has a steady growing 
 number of QuorumCnxManager$SendWorker threads up to a point where the OS runs 
 out of native threads, and at the same time we see a lot of exceptions in the 
 logs.  This is on 3.2.2 and our config looks like:
 {noformat}
 tickTime=3000
 dataDir=/somewhere_thats_not_tmp
 clientPort=2181
 initLimit=10
 syncLimit=5
 server.0=sv4borg9:2888:3888
 server.1=sv4borg10:2888:3888
 server.2=sv4borg11:2888:3888
 server.3=sv4borg12:2888:3888
 server.4=sv4borg13:2888:3888
 {noformat}
 The issue is on the first server. I'm going to attach threads dumps and logs 
 in moment.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (ZOOKEEPER-936) zkpython is leaking ACL_vector

2010-11-18 Thread Patrick Hunt (JIRA)


 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patrick Hunt updated ZOOKEEPER-936:
---

 Priority: Critical  (was: Major)
Fix Version/s: 3.4.0
   3.3.3
 Assignee: Gustavo Niemeyer

 zkpython is leaking ACL_vector
 --

 Key: ZOOKEEPER-936
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-936
 Project: Zookeeper
  Issue Type: Bug
  Components: contrib-bindings
Reporter: Gustavo Niemeyer
Assignee: Gustavo Niemeyer
Priority: Critical
 Fix For: 3.3.3, 3.4.0


 It looks like there are no calls to deallocate_ACL_vector() within 
 zookeeper.c in the zkpython binding, which means that (at least) the result 
 of zoo_get_acl() must be leaking.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (ZOOKEEPER-935) Concurrent primitives library - shared lock

2010-11-18 Thread Patrick Hunt (JIRA)


 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patrick Hunt updated ZOOKEEPER-935:
---

Fix Version/s: 3.4.0
 Assignee: ChiaHung Lin

Thanks for the patch! Slating for 3.4.0.

 Concurrent primitives library - shared lock
 ---

 Key: ZOOKEEPER-935
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-935
 Project: Zookeeper
  Issue Type: Improvement
  Components: recipes
 Environment: Debian squeeze 
 JDK 1.6.x
 zookeeper trunk
Reporter: ChiaHung Lin
Assignee: ChiaHung Lin
Priority: Minor
 Fix For: 3.4.0

 Attachments: ZOOKEEPER-935.patch


 I create this jira to add sharedock function. The function follows recipes at 
 http://hadoop.apache.org/zookeeper/docs/r3.1.2/recipes.html#Shared+Locks

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Re: bug in ZooKeeperSever.java

2010-11-18 Thread Patrick Hunt

Sorry for the slow response but this went into my spam folder and I
only just noticed it.

Please do enter a JIRA for this!
https://issues.apache.org/jira/browse/ZOOKEEPER

Thanks,

Patrick

On Thu, Nov 4, 2010 at 3:29 PM, Ling Liu l...@linkedin.com wrote:
 I found a bug in  this file ( version 3.1.1).

 public ZooKeeperServer(File snapDir, File logDir, int tickTime)
            throws IOException {
        this( new FileTxnSnapLog(snapDir, logDir),
                tickTime, new BasicDataTreeBuilder());
    }


 the FileTxnSnapLog constructor need logDir as the first parameter and the 
 snapDir as the second parameter.  Here the ZooKeeperServer  misplace the two 
 parameters.

 Ling

Fwd: Problem with Zookeeper cluster configuration

2010-11-18 Thread Patrick Hunt

I'm afraid this went into my spam folder and I only just noticed it.
Is this still and issue or did you work past it?

-- Forwarded message --
From: siddhartha banik siddhartha.ba...@gmail.com
To: zookeeper-user-subscr...@hadoop.apache.org, zookeeper-dev@hadoop.apache.org
Date: Wed, 27 Oct 2010 18:40:27 +0530
Subject: Problem with Zookeeper cluster configuration

Hi,

I am trying to configure zookeeper cluster ... with 2 server
instances. zookeeper version : 3.2.2

Config files are :

Server 1. zoo.cfg

tickTime=2000
initLimit=10
syncLimit=5
dataDir=/home/xuser/zookeeper1/zookeeper-3.2.2/data/
clientPort=5181
server.1=3.7.192.142:5181:5888
server.2=3.7.192.145:5181:5888

Server 2. zoo.cfg

tickTime=2000
initLimit=10
syncLimit=5
dataDir=/home/xuser/zookeeper2/zookeeper-3.2.2/data/
clientPort=5181
server.1=3.7.192.142:5181:5888
server.2=3.7.192.145:5181:5888

I have also created myid files in respective data folders. Below are
the exception I am seeing :

Server 1

2010-10-27 07:43:43,411 - INFO
[QuorumPeer:/0.0.0.0:5181:quorump...@514] - LOOKING
2010-10-27 07:43:43,418 - INFO
[QuorumPeer:/0.0.0.0:5181:fastleaderelect...@579] - New election: -1
2010-10-27 07:43:43,419 - INFO
[QuorumPeer:/0.0.0.0:5181:fastleaderelect...@618] - Notification: 1,
-1, 382, 1, LOOKING, LOOKING, 1
2010-10-27 07:43:43,420 - INFO
[QuorumPeer:/0.0.0.0:5181:fastleaderelect...@642] - Adding vote
2010-10-27 07:43:43,436 - INFO
[QuorumPeer:/0.0.0.0:5181:fastleaderelect...@618] - Notification: 2,
0, 383, 1, LOOKING, LOOKING, 2
2010-10-27 07:43:43,442 - INFO
[QuorumPeer:/0.0.0.0:5181:fastleaderelect...@642] - Adding vote
2010-10-27 07:43:43,443 - INFO
[QuorumPeer:/0.0.0.0:5181:fastleaderelect...@618] - Notification: 2,
0, 383, 1, LOOKING, LOOKING, 1
2010-10-27 07:43:43,443 - INFO
[QuorumPeer:/0.0.0.0:5181:fastleaderelect...@642] - Adding vote
2010-10-27 07:43:43,444 - INFO
[QuorumPeer:/0.0.0.0:5181:quorump...@523] - FOLLOWING
2010-10-27 07:43:43,445 - INFO
[QuorumPeer:/0.0.0.0:5181:zookeeperser...@160] - Created server
2010-10-27 07:43:43,447 - INFO
[QuorumPeer:/0.0.0.0:5181:follo...@147] - Following /3.7.192.145:5181
2010-10-27 07:43:43,461 - INFO  [WorkerReceiver
Thread:fastleaderelection$messenger$workerrecei...@254] - Sending new
notification.
2010-10-27 07:43:43,462 - WARN
[QuorumPeer:/0.0.0.0:5181:follo...@318] - Exception when following the
leader
java.io.EOFException
    at java.io.DataInputStream.readInt(DataInputStream.java:375)
    at 
org.apache.jute.BinaryInputArchive.readInt(BinaryInputArchive.java:63)
    at 
org.apache.zookeeper.server.quorum.QuorumPacket.deserialize(QuorumPacket.java:66)
    at 
org.apache.jute.BinaryInputArchive.readRecord(BinaryInputArchive.java:108)
    at 
org.apache.zookeeper.server.quorum.Follower.readPacket(Follower.java:114)
    at 
org.apache.zookeeper.server.quorum.Follower.followLeader(Follower.java:193)
    at 
org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:525)
2010-10-27 07:43:43,464 - INFO
[QuorumPeer:/0.0.0.0:5181:follo...@436] - shutdown called
java.lang.Exception: shutdown Follower
    at 
org.apache.zookeeper.server.quorum.Follower.shutdown(Follower.java:436)
    at 
org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:529)

Server 2

010-10-27 07:59:22,387 - INFO
[QuorumPeer:/0.0.0.0:5181:quorump...@535] - LEADING
2010-10-27 07:59:22,388 - INFO
[QuorumPeer:/0.0.0.0:5181:zookeeperser...@160] - Created server
2010-10-27 07:59:22,390 - ERROR [QuorumPeer:/0.0.0.0:5181:lea...@127]
- Couldn't bind to port 5181
java.net.BindException: Address already in use
    at java.net.PlainSocketImpl.socketBind(Native Method)
    at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:359)
    at java.net.ServerSocket.bind(ServerSocket.java:319)
    at java.net.ServerSocket.init(ServerSocket.java:185)
    at java.net.ServerSocket.init(ServerSocket.java:97)
    at org.apache.zookeeper.server.quorum.Leader.init(Leader.java:125)
    at 
org.apache.zookeeper.server.quorum.QuorumPeer.makeLeader(QuorumPeer.java:417)
    at 
org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:537)
2010-10-27 07:59:22,392 - WARN
[QuorumPeer:/0.0.0.0:5181:quorump...@541] - Unexpected exception
java.net.BindException: Address already in use
    at java.net.PlainSocketImpl.socketBind(Native Method)
    at java.net.PlainSocketImpl.bind(PlainSocketImpl.java:359)
    at java.net.ServerSocket.bind(ServerSocket.java:319)
    at java.net.ServerSocket.init(ServerSocket.java:185)
    at java.net.ServerSocket.init(ServerSocket.java:97)
    at org.apache.zookeeper.server.quorum.Leader.init(Leader.java:125)
    at 
org.apache.zookeeper.server.quorum.QuorumPeer.makeLeader(QuorumPeer.java:417)
    at 
org.apache.zookeeper.server.quorum.QuorumPeer.run(QuorumPeer.java:537)
2010-10-27 07:59:22,393 - INFO  [WorkerReceiver
Thread:fastleaderelection$messenger$workerrecei...@254] - Sending new

[jira] Commented: (ZOOKEEPER-925) Consider maven site generation to replace our forrest site and documentation generation


[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12933018#action_12933018
 ] 

Patrick Hunt commented on ZOOKEEPER-925:


@Alex looks good to me. We're new with mvn based site gen, what's the 
implications on our side of adding a site.vm file? Is that not something we can 
specify with site.xml?

 Consider maven site generation to replace our forrest site and documentation 
 generation
 ---

 Key: ZOOKEEPER-925
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-925
 Project: Zookeeper
  Issue Type: Wish
  Components: documentation
Reporter: Patrick Hunt
Assignee: Patrick Hunt
 Attachments: ZOOKEEPER-925.patch


 See WHIRR-19 for some background.
 In whirr we looked at a number of site/doc generation facilities. In the end 
 Maven site generation plugin turned out to be by far the best option. You can 
 see our nascent site here (no attempt at styling,etc so far):
 http://incubator.apache.org/whirr/
 In particular take a look at the quick start:
 http://incubator.apache.org/whirr/quick-start-guide.html
 which was generated from
 http://svn.apache.org/repos/asf/incubator/whirr/trunk/src/site/confluence/quick-start-guide.confluence
 notice this was standard wiki markup (confluence wiki markup, same as 
 available from apache)
 You can read more about mvn site plugin here:
 http://maven.apache.org/guides/mini/guide-site.html
 Notice that other formats are available, not just confluence markup, also 
 note that you can use different markup formats if you like in the same site 
 (although probably not a great idea, but in some cases might be handy, for 
 example whirr uses the confluence wiki, so we can pretty much copy/paste 
 source docs from wiki to our site (svn) if we like)
 Re maven vs our current ant based build. It's probably a good idea for us to 
 move the build to maven at some point. We could initially move just the doc 
 generation, and then incrementally move functionality from build.xml to mvn 
 over a longer time period.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (ZOOKEEPER-900) FLE implementation should be improved to use non-blocking sockets

[
https://issues.apache.org/jira/browse/ZOOKEEPER-900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12933024#action_12933024
]

Patrick Hunt commented on ZOOKEEPER-900:

I see some great information about how the code/algos operate being detailed in
these jiras. I highly encourage you guys to document this stuff in either the
code or in a separate document available on the wiki/forrest (now mvn site,
whatever). It's critical
that we provide more details like this to our devs.

See ZOOKEEPER-918 as a great example of what I'm talking about. (although
adding more comments to the code is fine too).

Basically, if you find yourself describing something in a jira that's not
documented already, consider documenting it. Thanks.

FLE implementation should be improved to use non-blocking sockets
-

Key: ZOOKEEPER-900
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-900
Project: Zookeeper
Issue Type: Bug
Reporter: Vishal K
Assignee: Vishal K
Priority: Critical
Fix For: 3.4.0

Attachments: ZOOKEEPER-900.patch, ZOOKEEPER-900.patch1,
ZOOKEEPER-900.patch2

From earlier email exchanges:
1. Blocking connects and accepts:
a) The first problem is in manager.toSend(). This invokes connectOne(), which
does a blocking connect. While testing, I changed the code so that
connectOne() starts a new thread called AsyncConnct(). AsyncConnect.run()
does a socketChannel.connect(). After starting AsyncConnect, connectOne
starts a timer. connectOne continues with normal operations if the connection
is established before the timer expires, otherwise, when the timer expires it
interrupts AsyncConnect() thread and returns. In this way, I can have an
upper bound on the amount of time we need to wait for connect to succeed. Of
course, this was a quick fix for my testing. Ideally, we should use Selector
to do non-blocking connects/accepts. I am planning to do that later once we
at least have a quick fix for the problem and consensus from others for the
real fix (this problem is big blocker for us). Note that it is OK to do
blocking IO in SenderWorker and RecvWorker threads since they block IO to the
respective !
peer.
b) The blocking IO problem is not just restricted to connectOne(), but also
in receiveConnection(). The Listener thread calls receiveConnection() for
each incoming connection request. receiveConnection does blocking IO to get
peer's info (s.read(msgBuffer)). Worse, it invokes connectOne() back to the
peer that had sent the connection request. All of this is happening from the
Listener. In short, if a peer fails after initiating a connection, the
Listener thread won't be able to accept connections from other peers, because
it would be stuck in read() or connetOne(). Also the code has an inherent
cycle. initiateConnection() and receiveConnection() will have to be very
carefully synchronized otherwise, we could run into deadlocks. This code is
going to be difficult to maintain/modify.
Also see: https://issues.apache.org/jira/browse/ZOOKEEPER-822

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (ZOOKEEPER-902) Fix findbug issue in trunk Malicious code vulnerability


[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12933020#action_12933020
 ] 

Patrick Hunt commented on ZOOKEEPER-902:


Sounds good. Let's clear out 900 then we can adjust the OK setting back to 0 
(as part of this jira) once 900 is committed.

 Fix findbug issue in trunk Malicious code vulnerability
 -

 Key: ZOOKEEPER-902
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-902
 Project: Zookeeper
  Issue Type: Bug
  Components: quorum, server
Affects Versions: 3.4.0
Reporter: Patrick Hunt
Priority: Minor
 Fix For: 3.4.0


 https://hudson.apache.org/hudson/view/ZooKeeper/job/ZooKeeper-trunk/970/artifact/trunk/findbugs/zookeeper-findbugs-report.html#Warnings_MALICIOUS_CODE
 Malicious code vulnerability Warnings
 Code  Warning
 MSorg.apache.zookeeper.server.quorum.LeaderElection.epochGen isn't final 
 but should be

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (ZOOKEEPER-925) Consider maven site generation to replace our forrest site and documentation generation


[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12933074#action_12933074
 ] 

Patrick Hunt commented on ZOOKEEPER-925:


TANSTAAFL ;-) You'd have to modify the db2rst a bit to get it to output 
confluence tags, iirc that was pretty easy to do (plus it's in python). Or find 
a rst to confluence converter (I looked a bit last night but didn't find, would 
doxia converter work?)

 Consider maven site generation to replace our forrest site and documentation 
 generation
 ---

 Key: ZOOKEEPER-925
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-925
 Project: Zookeeper
  Issue Type: Wish
  Components: documentation
Reporter: Patrick Hunt
Assignee: Patrick Hunt
 Attachments: ZOOKEEPER-925.patch


 See WHIRR-19 for some background.
 In whirr we looked at a number of site/doc generation facilities. In the end 
 Maven site generation plugin turned out to be by far the best option. You can 
 see our nascent site here (no attempt at styling,etc so far):
 http://incubator.apache.org/whirr/
 In particular take a look at the quick start:
 http://incubator.apache.org/whirr/quick-start-guide.html
 which was generated from
 http://svn.apache.org/repos/asf/incubator/whirr/trunk/src/site/confluence/quick-start-guide.confluence
 notice this was standard wiki markup (confluence wiki markup, same as 
 available from apache)
 You can read more about mvn site plugin here:
 http://maven.apache.org/guides/mini/guide-site.html
 Notice that other formats are available, not just confluence markup, also 
 note that you can use different markup formats if you like in the same site 
 (although probably not a great idea, but in some cases might be handy, for 
 example whirr uses the confluence wiki, so we can pretty much copy/paste 
 source docs from wiki to our site (svn) if we like)
 Re maven vs our current ant based build. It's probably a good idea for us to 
 move the build to maven at some point. We could initially move just the doc 
 generation, and then incrementally move functionality from build.xml to mvn 
 over a longer time period.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (ZOOKEEPER-925) Consider maven site generation to replace our forrest site and documentation generation


[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12932590#action_12932590
 ] 

Patrick Hunt commented on ZOOKEEPER-925:


At this point to move fwd we need to work on 2 main areas:

1) site gen
2) doc conversion

item 1) is looking pretty good, but some work yet to be done, icons and 
look/feel mainly
item 2) just requires us to decide which if any format we want to standardize 
on and then try moving some docs to that format.

I would highly suggest that our standard/preferred format be confluence 
format -- we can move our wiki to there at some point, which will match up 
nicely.


 Consider maven site generation to replace our forrest site and documentation 
 generation
 ---

 Key: ZOOKEEPER-925
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-925
 Project: Zookeeper
  Issue Type: Wish
  Components: documentation
Reporter: Patrick Hunt
Assignee: Patrick Hunt
 Attachments: ZOOKEEPER-925.patch


 See WHIRR-19 for some background.
 In whirr we looked at a number of site/doc generation facilities. In the end 
 Maven site generation plugin turned out to be by far the best option. You can 
 see our nascent site here (no attempt at styling,etc so far):
 http://incubator.apache.org/whirr/
 In particular take a look at the quick start:
 http://incubator.apache.org/whirr/quick-start-guide.html
 which was generated from
 http://svn.apache.org/repos/asf/incubator/whirr/trunk/src/site/confluence/quick-start-guide.confluence
 notice this was standard wiki markup (confluence wiki markup, same as 
 available from apache)
 You can read more about mvn site plugin here:
 http://maven.apache.org/guides/mini/guide-site.html
 Notice that other formats are available, not just confluence markup, also 
 note that you can use different markup formats if you like in the same site 
 (although probably not a great idea, but in some cases might be handy, for 
 example whirr uses the confluence wiki, so we can pretty much copy/paste 
 source docs from wiki to our site (svn) if we like)
 Re maven vs our current ant based build. It's probably a good idea for us to 
 move the build to maven at some point. We could initially move just the doc 
 generation, and then incrementally move functionality from build.xml to mvn 
 over a longer time period.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (ZOOKEEPER-896) Improve C client to support dynamic authentication schemes

[
https://issues.apache.org/jira/browse/ZOOKEEPER-896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12932607#action_12932607
]

Patrick Hunt commented on ZOOKEEPER-896:

The security docs (both client side and server (plugin arch is totally
undoc'd)) is sorely in need of improvement. If you are in the area and would
like to help out adding docs would be huge. Thanks.

Improve C client to support dynamic authentication schemes
--

Key: ZOOKEEPER-896
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-896
Project: Zookeeper
Issue Type: Improvement
Components: c client
Affects Versions: 3.3.1
Reporter: Botond Hejj
Assignee: Botond Hejj
Fix For: 3.4.0

Attachments: ZOOKEEPER-896.patch

When we started exploring zookeeper for our requirements we found the
authentication mechanism is not flexible enough.
We want to use kerberos for authentication but using the current API we ran
into a few problems. The idea is that we get a kerberos token on the client
side and than send that token to the server with a kerberos scheme. A server
side authentication plugin can use that token to authenticate the client and
also use the token for authorization.
We ran into two problems with this approach:
1. A different kerberos token is needed for each different server that client
can connect to since kerberos uses mutual authentication. That means when the
client acquires this kerberos token it has to know which server it connects
to and generate the token according to that. The client currently can't
generate a token for a specific server. The token stored in the auth_info is
used for all the servers.
2. The kerberos token might have an expiry time so if the client loses the
connection to the server and than it tries to reconnect it should acquire a
new token. That is not possible currently since the token is stored in
auth_info and reused for every connection.
The problem can be solved if we allow the client to register a callback for
authentication instead a static token. This can be a callback with an
argument which passes the current host string. The zookeeper client code
could call this callback before it sends the authentication info to the
server to get a fresh server specific token.
This would solve our problem with the kerberos authentication and also could
be used for other more dynamic authentication schemes.
The solution could be generalization also for the java client as well.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (ZOOKEEPER-366) Session timeout detection can go wrong if the leader system time changes

[
https://issues.apache.org/jira/browse/ZOOKEEPER-366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12932708#action_12932708
]

Patrick Hunt commented on ZOOKEEPER-366:

FYI, this came up again today on hbase list:

14:59 _hp_ man this system time update on a bunch of machines causing
zookeeper session timeouts causing hr's to die is really taking its toll, count
on a table now hangs, i disabled and enabled the table, tried count again, and
it hangs at the same place still. Arg.

Ben any progress on this? Should we try to get it into 3.3.3?

Session timeout detection can go wrong if the leader system time changes

Key: ZOOKEEPER-366
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-366
Project: Zookeeper
Issue Type: Bug
Reporter: Benjamin Reed
Assignee: Benjamin Reed
Attachments: ZOOKEEPER-366.patch

the leader tracks session expirations by calculating when a session will
timeout and then periodically checking to see what needs to be timed out
based on the current time. this works great as long as the leaders clock
progresses at a steady pace. the problem comes when there are big (session
size) changes in clock, by ntp for example. if time gets adjusted forward,
all the sessions could timeout immediately. if time goes backward sessions
that should timeout may take a lot longer to actually expire.
this is really just a leader issue. the easiest way to deal with this is to
have the leader relinquish leadership if it detects a big jump forward in
time. when a new leader gets elected, it will recalculate timeouts of active
sessions.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (ZOOKEEPER-366) Session timeout detection can go wrong if the leader system time changes


 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patrick Hunt updated ZOOKEEPER-366:
---

  Component/s: server
   quorum
Fix Version/s: 3.4.0
   3.3.3

 Session timeout detection can go wrong if the leader system time changes
 

 Key: ZOOKEEPER-366
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-366
 Project: Zookeeper
  Issue Type: Bug
  Components: quorum, server
Reporter: Benjamin Reed
Assignee: Benjamin Reed
 Fix For: 3.3.3, 3.4.0

 Attachments: ZOOKEEPER-366.patch


 the leader tracks session expirations by calculating when a session will 
 timeout and then periodically checking to see what needs to be timed out 
 based on the current time. this works great as long as the leaders clock 
 progresses at a steady pace. the problem comes when there are big (session 
 size) changes in clock, by ntp for example. if time gets adjusted forward, 
 all the sessions could timeout immediately. if time goes backward sessions 
 that should timeout may take a lot longer to actually expire.
 this is really just a leader issue. the easiest way to deal with this is to 
 have the leader relinquish leadership if it detects a big jump forward in 
 time. when a new leader gets elected, it will recalculate timeouts of active 
 sessions.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (ZOOKEEPER-756) some cleanup and improvements for zooinspector

[
https://issues.apache.org/jira/browse/ZOOKEEPER-756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Patrick Hunt updated ZOOKEEPER-756:
---

Status: Open (was: Patch Available)

some cleanup and improvements for zooinspector
--

Attachments: zooInspectorChanges.patch, ZOOKEEPER-756.patch

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (ZOOKEEPER-756) some cleanup and improvements for zooinspector

[
https://issues.apache.org/jira/browse/ZOOKEEPER-756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Patrick Hunt updated ZOOKEEPER-756:
---

Status: Patch Available (was: Open)

some cleanup and improvements for zooinspector
--

Attachments: zooInspectorChanges.patch, ZOOKEEPER-756.patch

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (ZOOKEEPER-756) some cleanup and improvements for zooinspector

[
https://issues.apache.org/jira/browse/ZOOKEEPER-756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Patrick Hunt updated ZOOKEEPER-756:
---

Attachment: ZOOKEEPER-756.patch

Looks like line endings are the problem. The current source in svn has some
lines with ^M line endings and this seems to be messing up the patch
application (at least under unix).

I tried it myself and had similar problems (I'm on unix). I then used dos2unix
to convert all the zooinspector files to unix line endings, then reapplied
Colin's patch and it patched fine.

I created a new patch file based on this, which I'm now attaching.

Please review this updated patch and make sure I didn't miss anything

some cleanup and improvements for zooinspector
--

Attachments: zooInspectorChanges.patch, ZOOKEEPER-756.patch

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (ZOOKEEPER-756) some cleanup and improvements for zooinspector

[
https://issues.apache.org/jira/browse/ZOOKEEPER-756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Patrick Hunt updated ZOOKEEPER-756:
---

Status: Open (was: Patch Available)

cancelling patch - please address the @author issues (remove them), the javadoc
warning seems to be some issue with the link being accessed (not with the
change itself).

some cleanup and improvements for zooinspector
--

Attachments: zooInspectorChanges.patch, ZOOKEEPER-756.patch

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (ZOOKEEPER-900) FLE implementation should be improved to use non-blocking sockets

2010-11-15 Thread Patrick Hunt (JIRA)


[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12932118#action_12932118
 ] 

Patrick Hunt commented on ZOOKEEPER-900:


I'd appreciate if you could fix the findbugs, that would be great. See also 
ZOOKEEPER-902 -- as part of the fix set the findbugs acceptable back to 0. 
Thanks!

 FLE implementation should be improved to use non-blocking sockets
 -

 Key: ZOOKEEPER-900
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-900
 Project: Zookeeper
  Issue Type: Bug
Reporter: Vishal K
Assignee: Vishal K
Priority: Critical
 Fix For: 3.4.0

 Attachments: ZOOKEEPER-900.patch1, ZOOKEEPER-900.patch2


 From earlier email exchanges:
 1. Blocking connects and accepts:
 a) The first problem is in manager.toSend(). This invokes connectOne(), which 
 does a blocking connect. While testing, I changed the code so that 
 connectOne() starts a new thread called AsyncConnct(). AsyncConnect.run() 
 does a socketChannel.connect(). After starting AsyncConnect, connectOne 
 starts a timer. connectOne continues with normal operations if the connection 
 is established before the timer expires, otherwise, when the timer expires it 
 interrupts AsyncConnect() thread and returns. In this way, I can have an 
 upper bound on the amount of time we need to wait for connect to succeed. Of 
 course, this was a quick fix for my testing. Ideally, we should use Selector 
 to do non-blocking connects/accepts. I am planning to do that later once we 
 at least have a quick fix for the problem and consensus from others for the 
 real fix (this problem is big blocker for us). Note that it is OK to do 
 blocking IO in SenderWorker and RecvWorker threads since they block IO to the 
 respective !
 peer.
 b) The blocking IO problem is not just restricted to connectOne(), but also 
 in receiveConnection(). The Listener thread calls receiveConnection() for 
 each incoming connection request. receiveConnection does blocking IO to get 
 peer's info (s.read(msgBuffer)). Worse, it invokes connectOne() back to the 
 peer that had sent the connection request. All of this is happening from the 
 Listener. In short, if a peer fails after initiating a connection, the 
 Listener thread won't be able to accept connections from other peers, because 
 it would be stuck in read() or connetOne(). Also the code has an inherent 
 cycle. initiateConnection() and receiveConnection() will have to be very 
 carefully synchronized otherwise, we could run into deadlocks. This code is 
 going to be difficult to maintain/modify.
 Also see: https://issues.apache.org/jira/browse/ZOOKEEPER-822

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Re: Watcher examples

2010-11-15 Thread Patrick Hunt

It would be great to have more examples as part of the release
artifact.  Would you mind creating a JIRA/patch for this?
http://wiki.apache.org/hadoop/ZooKeeper/HowToContribute

I'm thinking that we could have a src/contrib/examples or src/examples
... what do you guys think? (mahadev?)

Patrick


On Thu, Nov 11, 2010 at 12:46 PM, Robert Crocombe rcroc...@gmail.com wrote:
 On Tue, Nov 9, 2010 at 12:34 PM, Jeremy Hanna 
 jeremy.hanna1...@gmail.comwrote:

 Anyone know of a good blog post or docs anywhere that gives a simple
 example of Watchers in action?  I saw the one on:

 http://hadoop.apache.org/zookeeper/docs/current/javaExample.html#ch_Introduction

 but it seems kind of overly complicated for an intro to Watchers.  I
 appreciate the example but wondered if there were other examples out there.


 Appended is a Java example of using a Watcher simply to wait for the client
 to actually be connected to a server.  I used it when I was confirming to my
 satisfaction that there was a bug in the ZooKeeper recipe for WriteLock
 awhile ago.  I think this use is slightly unusual in that it is more
 interested in KeeperState than the event type.  A more conventional Watcher
 might be like the following sketch (uhm, this is Groovy), though really
 you'd have to look at both:

 @Override
 public void process(WatchedEvent event) {
 switch (event?.getType()) {
 case EventType.NodeDeleted:
 // TODO: what should we do if the node being watched is itself
 // deleted?
 LOG.error(The node being watched ' + event.getPath + ' has been deleted:
 that's not good)
 break
 case EventType.NodeChildrenChanged:
 childrenChanged(event)
 break
 default:
 LOG.debug(Ignoring event type ' + event?.getType() + ')
 break
 }
 }

 --
 Robert Crocombe

 package derp;

 import java.io.IOException;
 import java.util.concurrent.atomic.AtomicBoolean;
 import java.util.concurrent.locks.Condition;
 import java.util.concurrent.locks.Lock;
 import java.util.concurrent.locks.ReentrantLock;

 import org.apache.commons.logging.Log;
 import org.apache.commons.logging.LogFactory;
 import org.apache.zookeeper.KeeperException;
 import org.apache.zookeeper.WatchedEvent;
 import org.apache.zookeeper.Watcher;
 import org.apache.zookeeper.ZooKeeper;
 import org.apache.zookeeper.Watcher.Event.KeeperState;
 import org.apache.zookeeper.recipes.lock.WriteLock;

 public class Test {
 private static final Log LOG = LogFactory.getLog(Test.class);

 private static final String ZOO_CONFIG = 10.2.1.54:2181/test;
 private static final String LOCK_DIR = /locking-test;
 private static final int TIMEOUT_MILLIS = 1;

 private static class ConnectWatcher implements Watcher {

 private final Lock connectedLock = new ReentrantLock();
 private final Condition connectedCondition = connectedLock.newCondition();

 private final AtomicBoolean connected = new AtomicBoolean(false);

 @Override
 public void process(WatchedEvent event) {
 LOG.debug(Event:  + event);

 KeeperState keeperState = event.getState();
 switch (keeperState) {
 case SyncConnected:
 if (!connected.get()) {
 connected.set(true);
 signal();
 }
 break;
 case Expired:
 case Disconnected:
 if (connected.get()) {
 connected.set(false);
 signal();
 }
 }
 }

 public void waitForConnection() throws InterruptedException {
 connectedLock.lock();
 try {
 while (!connected.get()) {
 LOG.debug(Waiting for condition to be signalled);
 connectedCondition.await();
 LOG.debug(Woken up on condition signalled);
 }
 } finally {
 connectedLock.unlock();
 }
 LOG.debug(After signalling, we are connected);
 }

 @Override
 public String toString() {
 StringBuilder b = new StringBuilder([);
 b.append(connectedLock:).append(connectedLock);
 b.append(,connectedCondition:).append(connectedCondition);
 b.append(,connected:).append(connected);
 b.append(]);
 return b.toString();
 }

 private void signal() {
 LOG.debug(Signaling after event);
 connectedLock.lock();
 try {
 connectedCondition.signal();
 } finally {
 connectedLock.unlock();
 }
 }
 }

 private static final void fine(ZooKeeper lowerId, ZooKeeper higherId) throws
 KeeperException,
 InterruptedException {
 WriteLock lower = new WriteLock(lowerId, LOCK_DIR, null);
 WriteLock higher = new WriteLock(higherId, LOCK_DIR, null);

 boolean lowerAcquired = lower.lock();
 assert lowerAcquired;

 LOG.debug(Lower acquired lock successfully, so higher should fail);

 boolean higherAcquired = higher.lock();
 assert !higherAcquired;

 LOG.debug(Correct: higher session fails to acquire lock);

 lower.unlock();
 // Now that lower has unlocked, higher will acquire. Really should use
 // the version of WriteLock with the LockListener, but a short sleep
 // should do.
 Thread.sleep(2000);
 higher.unlock();
 // make sure we let go.
 assert !higher.isOwner();
 }

 /*
  * Using recipes from ZooKeeper 3.2.1.
  *
  * This bug occurs because the sort in ZooKeeperLockOperation.execute
  * (beginning @ line 221) orders the paths, but the paths contain the
  * session ID (lines 206-207), so that sorting

[jira] Commented: (ZOOKEEPER-900) FLE implementation should be improved to use non-blocking sockets


[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12931444#action_12931444
 ] 

Patrick Hunt commented on ZOOKEEPER-900:


Flavio, I'd be worried that different tcp stacks might (inter)operate 
differently in practice vs theory.


In general it's pretty tough to get this right - look at all the problems we've 
been having with netcat behavior
https://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=truequery=netcatsummary=truedescription=truebody=truepid=12310801

Ubuntu recently moved from traditional to the newish bsd flavor (supports 
ipv6 natively) of nc and we are back to having issues after having made 
significant changes in 3.3 to fix this (incl a number of tests that simulated 
the nc behavior as closely as we could understand it).

 FLE implementation should be improved to use non-blocking sockets
 -

 Key: ZOOKEEPER-900
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-900
 Project: Zookeeper
  Issue Type: Bug
Reporter: Vishal K
Assignee: Vishal K
Priority: Critical
 Fix For: 3.4.0


 From earlier email exchanges:
 1. Blocking connects and accepts:
 a) The first problem is in manager.toSend(). This invokes connectOne(), which 
 does a blocking connect. While testing, I changed the code so that 
 connectOne() starts a new thread called AsyncConnct(). AsyncConnect.run() 
 does a socketChannel.connect(). After starting AsyncConnect, connectOne 
 starts a timer. connectOne continues with normal operations if the connection 
 is established before the timer expires, otherwise, when the timer expires it 
 interrupts AsyncConnect() thread and returns. In this way, I can have an 
 upper bound on the amount of time we need to wait for connect to succeed. Of 
 course, this was a quick fix for my testing. Ideally, we should use Selector 
 to do non-blocking connects/accepts. I am planning to do that later once we 
 at least have a quick fix for the problem and consensus from others for the 
 real fix (this problem is big blocker for us). Note that it is OK to do 
 blocking IO in SenderWorker and RecvWorker threads since they block IO to the 
 respective !
 peer.
 b) The blocking IO problem is not just restricted to connectOne(), but also 
 in receiveConnection(). The Listener thread calls receiveConnection() for 
 each incoming connection request. receiveConnection does blocking IO to get 
 peer's info (s.read(msgBuffer)). Worse, it invokes connectOne() back to the 
 peer that had sent the connection request. All of this is happening from the 
 Listener. In short, if a peer fails after initiating a connection, the 
 Listener thread won't be able to accept connections from other peers, because 
 it would be stuck in read() or connetOne(). Also the code has an inherent 
 cycle. initiateConnection() and receiveConnection() will have to be very 
 carefully synchronized otherwise, we could run into deadlocks. This code is 
 going to be difficult to maintain/modify.
 Also see: https://issues.apache.org/jira/browse/ZOOKEEPER-822

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (ZOOKEEPER-860) Add alternative search-provider to ZK site


[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12931462#action_12931462
 ] 

Patrick Hunt commented on ZOOKEEPER-860:


Hi Alex, Otis. Take a look at ZOOKEEPER-925. I think this is a good time (new 
site gen and new site once/if we get approved as TLP) to introduce this change. 
Perhaps you could update the sitegen to include this? It would give ppl a 
change to try it out. Regards.

 Add alternative search-provider to ZK site
 --

 Key: ZOOKEEPER-860
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-860
 Project: Zookeeper
  Issue Type: Improvement
  Components: documentation
Reporter: Alex Baranau
Assignee: Alex Baranau
Priority: Minor
 Fix For: 3.4.0

 Attachments: ZOOKEEPER-860.patch


 Use search-hadoop.com service to make available search in ZK sources, MLs, 
 wiki, etc.
 This was initially proposed on user mailing list 
 (http://search-hadoop.com/m/sTZ4Y1BVKWg1). The search service was already 
 added in site's skin (common for all Hadoop related projects) before (as a 
 part of [AVRO-626|https://issues.apache.org/jira/browse/AVRO-626]) so this 
 issue is about enabling it for ZK. The ultimate goal is to use it at all 
 Hadoop's sub-projects' sites.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (ZOOKEEPER-900) FLE implementation should be improved to use non-blocking sockets


[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12931466#action_12931466
 ] 

Patrick Hunt commented on ZOOKEEPER-900:


I don't know for this specific case, but the corners I've looked at (tearing 
down a connection) there have been issues. Perhaps they are issues on our side, 
I'm not certain, but I do know that we fail with this version of nc (default in 
ubuntu maverick) even after significant work was done to address the original 
problem:
OpenBSD netcat (Debian patchlevel 1.89-3ubuntu2)

Let's assume what you say is correct -- we'd want to test this carefully to 
assure ourselves.


 FLE implementation should be improved to use non-blocking sockets
 -

 Key: ZOOKEEPER-900
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-900
 Project: Zookeeper
  Issue Type: Bug
Reporter: Vishal K
Assignee: Vishal K
Priority: Critical
 Fix For: 3.4.0


 From earlier email exchanges:
 1. Blocking connects and accepts:
 a) The first problem is in manager.toSend(). This invokes connectOne(), which 
 does a blocking connect. While testing, I changed the code so that 
 connectOne() starts a new thread called AsyncConnct(). AsyncConnect.run() 
 does a socketChannel.connect(). After starting AsyncConnect, connectOne 
 starts a timer. connectOne continues with normal operations if the connection 
 is established before the timer expires, otherwise, when the timer expires it 
 interrupts AsyncConnect() thread and returns. In this way, I can have an 
 upper bound on the amount of time we need to wait for connect to succeed. Of 
 course, this was a quick fix for my testing. Ideally, we should use Selector 
 to do non-blocking connects/accepts. I am planning to do that later once we 
 at least have a quick fix for the problem and consensus from others for the 
 real fix (this problem is big blocker for us). Note that it is OK to do 
 blocking IO in SenderWorker and RecvWorker threads since they block IO to the 
 respective !
 peer.
 b) The blocking IO problem is not just restricted to connectOne(), but also 
 in receiveConnection(). The Listener thread calls receiveConnection() for 
 each incoming connection request. receiveConnection does blocking IO to get 
 peer's info (s.read(msgBuffer)). Worse, it invokes connectOne() back to the 
 peer that had sent the connection request. All of this is happening from the 
 Listener. In short, if a peer fails after initiating a connection, the 
 Listener thread won't be able to accept connections from other peers, because 
 it would be stuck in read() or connetOne(). Also the code has an inherent 
 cycle. initiateConnection() and receiveConnection() will have to be very 
 carefully synchronized otherwise, we could run into deadlocks. This code is 
 going to be difficult to maintain/modify.
 Also see: https://issues.apache.org/jira/browse/ZOOKEEPER-822

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[ANNOUNCE] Apache ZooKeeper 3.3.2

2010-11-12 Thread Patrick Hunt

The Apache ZooKeeper team is proud to announce Apache ZooKeeper version 3.3.2

ZooKeeper is a high-performance coordination service for distributed
applications. It exposes common services - such as naming,
configuration management, synchronization, and group services - in a
simple interface so you don't have to write them from scratch. You can
use it off-the-shelf to implement consensus, group management, leader
election, and presence protocols. And you can build on it for your
own, specific needs.

For ZooKeeper release details and downloads, visit:
http://hadoop.apache.org/zookeeper/releases.html

ZooKeeper 3.3.2 Release Notes are at:
http://hadoop.apache.org/zookeeper/docs/r3.3.2/releasenotes.html

Regards,

The ZooKeeper Team

FYI: Netty is forking

2010-11-12 Thread Patrick Hunt

http://www.jboss.org/netty/community.html#nabble-td5730963

Patrick

[jira] Commented: (ZOOKEEPER-900) FLE implementation should be improved to use non-blocking sockets


[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12931487#action_12931487
 ] 

Patrick Hunt commented on ZOOKEEPER-900:


please try to keep the reformatting changes to a minimum unless it's code 
directly being worked on. otw it makes it harder to review (svn -x -w diff does 
help, but still) and blame detail is lost.

 FLE implementation should be improved to use non-blocking sockets
 -

 Key: ZOOKEEPER-900
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-900
 Project: Zookeeper
  Issue Type: Bug
Reporter: Vishal K
Assignee: Vishal K
Priority: Critical
 Fix For: 3.4.0

 Attachments: ZOOKEEPER-900.patch1


 From earlier email exchanges:
 1. Blocking connects and accepts:
 a) The first problem is in manager.toSend(). This invokes connectOne(), which 
 does a blocking connect. While testing, I changed the code so that 
 connectOne() starts a new thread called AsyncConnct(). AsyncConnect.run() 
 does a socketChannel.connect(). After starting AsyncConnect, connectOne 
 starts a timer. connectOne continues with normal operations if the connection 
 is established before the timer expires, otherwise, when the timer expires it 
 interrupts AsyncConnect() thread and returns. In this way, I can have an 
 upper bound on the amount of time we need to wait for connect to succeed. Of 
 course, this was a quick fix for my testing. Ideally, we should use Selector 
 to do non-blocking connects/accepts. I am planning to do that later once we 
 at least have a quick fix for the problem and consensus from others for the 
 real fix (this problem is big blocker for us). Note that it is OK to do 
 blocking IO in SenderWorker and RecvWorker threads since they block IO to the 
 respective !
 peer.
 b) The blocking IO problem is not just restricted to connectOne(), but also 
 in receiveConnection(). The Listener thread calls receiveConnection() for 
 each incoming connection request. receiveConnection does blocking IO to get 
 peer's info (s.read(msgBuffer)). Worse, it invokes connectOne() back to the 
 peer that had sent the connection request. All of this is happening from the 
 Listener. In short, if a peer fails after initiating a connection, the 
 Listener thread won't be able to accept connections from other peers, because 
 it would be stuck in read() or connetOne(). Also the code has an inherent 
 cycle. initiateConnection() and receiveConnection() will have to be very 
 carefully synchronized otherwise, we could run into deadlocks. This code is 
 going to be difficult to maintain/modify.
 Also see: https://issues.apache.org/jira/browse/ZOOKEEPER-822

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (ZOOKEEPER-900) FLE implementation should be improved to use non-blocking sockets

[
https://issues.apache.org/jira/browse/ZOOKEEPER-900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12931497#action_12931497
]

Patrick Hunt commented on ZOOKEEPER-900:

Looking at the patch. Quite a bit changed, hard to tell which is important and
which not. In these situations I've used the -w diff trick to get just the
important changes, then applied that patch to virgin code, opened the file in
eclipse and fixed the (relatively) smaller set of formatting issues.

Also, the patch includes log4j.properties change, you don't want to include
that I'm thinking.

FLE implementation should be improved to use non-blocking sockets
-

Key: ZOOKEEPER-900
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-900
Project: Zookeeper
Issue Type: Bug
Reporter: Vishal K
Assignee: Vishal K
Priority: Critical
Fix For: 3.4.0

Attachments: ZOOKEEPER-900.patch1

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (ZOOKEEPER-900) FLE implementation should be improved to use non-blocking sockets

[
https://issues.apache.org/jira/browse/ZOOKEEPER-900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12931554#action_12931554
]

Patrick Hunt commented on ZOOKEEPER-900:

fyi, if a patch is ready for review/commit then click the submit patch link
-- will trigger the workflow.
Also if you use the same patch name (ZOOKEEPER-###.patch) and re-attach with
the same name jira will handle this correctly, more detail here:
http://wiki.apache.org/hadoop/ZooKeeper/HowToContribute
thanks!

FLE implementation should be improved to use non-blocking sockets
-

Key: ZOOKEEPER-900
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-900
Project: Zookeeper
Issue Type: Bug
Reporter: Vishal K
Assignee: Vishal K
Priority: Critical
Fix For: 3.4.0

Attachments: ZOOKEEPER-900.patch1, ZOOKEEPER-900.patch2

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (ZOOKEEPER-929) hudson qabot incorrectly reporting issues as number 909 when the patch from 908 is the one being tested

2010-11-11 Thread Patrick Hunt (JIRA)

hudson qabot incorrectly reporting issues as number 909 when the patch from 908 
is the one being tested
---

 Key: ZOOKEEPER-929
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-929
 Project: Zookeeper
  Issue Type: Bug
  Components: build
Reporter: Patrick Hunt
Assignee: Nigel Daley


Hi Nigel can you take a look at this?

Following you'll see the email I got, notice that the patch is patch 908, 
however if you look at the hudson page it's linked to the change is documented 
as 909 patch file applied
https://hudson.apache.org/hudson/job/PreCommit-ZOOKEEPER-Build/25/changes

I looked at both jiras ZOOKEEPER-908 and ZOOKEEPER-909 both of these look good 
(the right names on patches) and qabot actually updated 908 with the comment 
(failure). However the change is listed as 909 which is wrong.


[exec] -1 overall.  Here are the results of testing the latest attachment
[exec]   
http://issues.apache.org/jira/secure/attachment/12459361/ZOOKEEPER-908.patch
[exec]   against trunk revision 1033770.
[exec]
[exec] +1 @author.  The patch does not contain any @author tags.
[exec]
[exec] -1 tests included.  The patch doesn't appear to include any new 
or modified tests.
[exec] Please justify why no new tests are needed 
for this patch.
[exec] Also please list what manual steps were 
performed to verify this patch.
[exec]
[exec] +1 javadoc.  The javadoc tool did not generate any warning 
messages.
[exec]
[exec] +1 javac.  The applied patch does not increase the total number 
of javac compiler warnings.
[exec]
[exec] +1 findbugs.  The patch does not introduce any new Findbugs 
warnings.
[exec]
[exec] +1 release audit.  The applied patch does not increase the total 
number of release audit warnings.
[exec]
[exec] +1 core tests.  The patch passed core unit tests.
[exec]
[exec] +1 contrib tests.  The patch passed contrib unit tests.
[exec]
[exec] Test results: 
https://hudson.apache.org/hudson/job/PreCommit-ZOOKEEPER-Build/25//testReport/
[exec] Findbugs warnings: 
https://hudson.apache.org/hudson/job/PreCommit-ZOOKEEPER-Build/25//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
[exec] Console output: 
https://hudson.apache.org/hudson/job/PreCommit-ZOOKEEPER-Build/25//console
[exec]
[exec] This message is automatically generated.
[exec]

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (ZOOKEEPER-908) Remove code duplication and inconsistent naming in ClientCnxn.Packet creation

2010-11-11 Thread Patrick Hunt (JIRA)


 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patrick Hunt updated ZOOKEEPER-908:
---

  Resolution: Fixed
Hadoop Flags: [Reviewed]
  Status: Resolved  (was: Patch Available)

+1 change looks good to me. Thanks Thomas! Committed to trunk.

 Remove code duplication and inconsistent naming in ClientCnxn.Packet creation
 -

 Key: ZOOKEEPER-908
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-908
 Project: Zookeeper
  Issue Type: Sub-task
  Components: server
Reporter: Thomas Koch
Assignee: Thomas Koch
Priority: Minor
 Fix For: 3.4.0

 Attachments: ZOOKEEPER-908.patch


 rename record - request (since their is a counterpart record named 
 response)
 rename header - requestHeader (to distinguish from responseHeader)
 remove ByteBuffer creation code from primeConnection() method and use the 
 duplicate code in the Packet constructor. Therefor the Bytebuffer bb 
 parameter could also be removed from the constructor's parameters.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (ZOOKEEPER-780) zkCli.sh generates a ArrayIndexOutOfBoundsException

2010-11-11 Thread Patrick Hunt (JIRA)


[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12931092#action_12931092
 ] 

Patrick Hunt commented on ZOOKEEPER-780:


Agreed (no prev tests) but really this highlights that there should be. Thanks!

 zkCli.sh  generates a ArrayIndexOutOfBoundsException 
 -

 Key: ZOOKEEPER-780
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-780
 Project: Zookeeper
  Issue Type: Bug
  Components: scripts
Affects Versions: 3.3.1
 Environment: Linux Ubuntu running in VMPlayer on top of Windows XP
Reporter: Miguel Correia
Assignee: Andrei Savu
Priority: Minor
 Fix For: 3.4.0

 Attachments: ZOOKEEPER-780.patch, ZOOKEEPER-780.patch, 
 ZOOKEEPER-780.patch


 I'm starting to play with Zookeeper so I'm still running it in standalone 
 mode. This is not a big issue, but here it goes for the records. 
 I've run zkCli.sh to run some commands in the server. I created a znode 
 /groups. When I tried to create a znode client_1 inside /groups, I forgot to 
 include the data: an exception was generated and zkCli-sh crashed, instead of 
 just showing an error. I tried a few variations and it seems like the problem 
 is not including the data.
 A copy of the screen:
 [zk: localhost:2181(CONNECTED) 3] create /groups firstgroup
 Created /groups
 [zk: localhost:2181(CONNECTED) 4] create -e /groups/client_1
 Exception in thread main java.lang.ArrayIndexOutOfBoundsException: 3
   at 
 org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:678)
   at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:581)
   at 
 org.apache.zookeeper.ZooKeeperMain.executeLine(ZooKeeperMain.java:353)
   at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:311)
   at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:270)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (ZOOKEEPER-909) Extract NIO specific code from ClientCnxn

[
https://issues.apache.org/jira/browse/ZOOKEEPER-909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12930548#action_12930548
]

Patrick Hunt commented on ZOOKEEPER-909:

Hi Thomas. Still shaky legs on getting the patch queue up and working again.
Shouldn't keep us from getting this committed though.

re javadoc, this is not an issue for the other patches afaict, any idea why
it's just showing up for this patch?

There are two sets of tests, java and the c client binding. Unfortunately
hudson currently does not highlight c failures on the summary page, you need to
checkout the console (usually raw) in the case where the tests fail (but not
java test).

Looking at console I see:

[exec] [exec] ZooKeeper server process failed ZooKeeper server NOT
startedRunning

I've notified Nigel about this to see is he has insight (saw it on a couple
other jiras). So far he hasn't had a chance to look into it.

Extract NIO specific code from ClientCnxn
-

Key: ZOOKEEPER-909
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-909
Project: Zookeeper
Issue Type: Sub-task
Components: java client
Reporter: Thomas Koch
Assignee: Thomas Koch
Fix For: 3.4.0

Attachments: ClientCnxnSocketNetty.java, ZOOKEEPER-909.patch,
ZOOKEEPER-909.patch, ZOOKEEPER-909.patch, ZOOKEEPER-909.patch,
ZOOKEEPER-909.patch

This patch is mostly the same patch as my last one for ZOOKEEPER-823 minus
everything Netty related. This means this patch only extract all NIO specific
code in the class ClientCnxnSocketNIO which extends ClientCnxnSocket.
I've redone this patch from current trunk step by step now and couldn't find
any logical error. I've already done a couple of successful test runs and
will continue to do so this night.
It would be nice, if we could apply this patch as soon as possible to trunk.
This allows us to continue to work on the netty integration without blocking
the ClientCnxn class. Adding Netty after this patch should be only a matter
of adding the ClientCnxnSocketNetty class with the appropriate test cases.
You could help me by reviewing the patch and by running it on whatever test
server you have available. Please send me any complete failure log you should
encounter to thomas at koch point ro. Thx!
Update: Until now, I've collected 8 successful builds in a row!

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (ZOOKEEPER-909) Extract NIO specific code from ClientCnxn


[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12930553#action_12930553
 ] 

Patrick Hunt commented on ZOOKEEPER-909:


better, but why is javadoc failing for this but not the other patches?

 Extract NIO specific code from ClientCnxn
 -

 Key: ZOOKEEPER-909
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-909
 Project: Zookeeper
  Issue Type: Sub-task
  Components: java client
Reporter: Thomas Koch
Assignee: Thomas Koch
 Fix For: 3.4.0

 Attachments: ClientCnxnSocketNetty.java, ZOOKEEPER-909.patch, 
 ZOOKEEPER-909.patch, ZOOKEEPER-909.patch, ZOOKEEPER-909.patch, 
 ZOOKEEPER-909.patch


 This patch is mostly the same patch as my last one for ZOOKEEPER-823 minus 
 everything Netty related. This means this patch only extract all NIO specific 
 code in the class ClientCnxnSocketNIO which extends ClientCnxnSocket.
 I've redone this patch from current trunk step by step now and couldn't find 
 any logical error. I've already done a couple of successful test runs and 
 will continue to do so this night.
 It would be nice, if we could apply this patch as soon as possible to trunk. 
 This allows us to continue to work on the netty integration without blocking 
 the ClientCnxn class. Adding Netty after this patch should be only a matter 
 of adding the ClientCnxnSocketNetty class with the appropriate test cases.
 You could help me by reviewing the patch and by running it on whatever test 
 server you have available. Please send me any complete failure log you should 
 encounter to thomas at koch point ro. Thx!
 Update: Until now, I've collected 8 successful builds in a row!

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (ZOOKEEPER-896) Improve C client to support dynamic authentication schemes

[
https://issues.apache.org/jira/browse/ZOOKEEPER-896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12930648#action_12930648
]

Patrick Hunt commented on ZOOKEEPER-896:

Hi Bontond, if this is ready to go (you think it's ready for review/commit)
please click the submit patch link on the left hand side of this page. That
will trigger the necessary workflow. thanks!

Improve C client to support dynamic authentication schemes
--

Attachments: ZOOKEEPER-896.patch

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (ZOOKEEPER-905) enhance zkServer.sh for easier zookeeper automation-izing

[
https://issues.apache.org/jira/browse/ZOOKEEPER-905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12930654#action_12930654
]

Patrick Hunt commented on ZOOKEEPER-905:

Hi Nicholas, thanks!

You've currently got the workflow in inprogress mode, iirc this happens when
you resume progress or something like that (we typically don't use that part
of the workflow, if the issue is assigned to you the assumption is that you are
working on it unless we hear otw). You'll need to take the jira out of
inprogress mode and then select submit patch for this to go to the qabot
and then get reviewed/comitted by a committer.

One other FYI, this jira is assigned to be fixed in 3.4.0 (current trunk, ie
the next full trunk release). Typically you'd want to create the patch against
svn trunk. Also, the patch queue on hudson (qa bot) will only test patches
against trunk. Not a big deal (your patch may apply against trunk even if
created from 3.3.1) but I just wanted to give you that headsup.

Thanks again. Regards.

enhance zkServer.sh for easier zookeeper automation-izing
-

Key: ZOOKEEPER-905
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-905
Project: Zookeeper
Issue Type: Improvement
Components: scripts
Reporter: Nicholas Harteau
Assignee: Nicholas Harteau
Priority: Minor
Fix For: 3.4.0

Attachments: zkServer.sh.diff

zkServer.sh is good at starting zookeeper and figuring out the right options
to pass along.
unfortunately if you want to wrap zookeeper startup/shutdown in any
significant way, you have to reimplement a bunch of the logic there.
the attached patch addresses a couple simple issues:
1. add a 'start-foreground' option to zkServer.sh - this allows things that
expect to manage a foregrounded process (daemontools, launchd, etc) to use
zkServer.sh instead of rolling their own to launch zookeeper
2. add a 'print-cmd' option to zkServer.sh - rather than launching zookeeper
from the script, just give me the command you'd normally use to exec
zookeeper. I found this useful when writing automation to start/stop
zookeeper as part of smoke testing zookeeper-based applications
3. Deal more gracefully with supplying alternate configuration files to
zookeeper - currently the script assumes all config files reside in
$ZOOCFGDIR - also useful for smoke testing
4. communicate extra info (JMX enabled) about zookeeper on STDERR rather
than STDOUT (necessary for #2)
5. fixes an issue on macos where readlink doesn't have the '-f' option.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (ZOOKEEPER-784) server-side functionality for read-only mode


[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-784?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12930692#action_12930692
 ] 

Patrick Hunt commented on ZOOKEEPER-784:


No worries, I'd like to get this in given you've done a bunch of work on it, 
qabot just flagged it given it's recently working again. thanks.

 server-side functionality for read-only mode
 

 Key: ZOOKEEPER-784
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-784
 Project: Zookeeper
  Issue Type: Sub-task
  Components: server
Reporter: Sergey Doroshenko
Assignee: Sergey Doroshenko
 Fix For: 3.4.0

 Attachments: ZOOKEEPER-784.patch, ZOOKEEPER-784.patch, 
 ZOOKEEPER-784.patch, ZOOKEEPER-784.patch, ZOOKEEPER-784.patch, 
 ZOOKEEPER-784.patch, ZOOKEEPER-784.patch, ZOOKEEPER-784.patch, 
 ZOOKEEPER-784.patch, ZOOKEEPER-784.patch


 As per http://wiki.apache.org/hadoop/ZooKeeper/GSoCReadOnlyMode , create 
 ReadOnlyZooKeeperServer which comes into play when peer is partitioned.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Re: [VOTE] Release ZooKeeper 3.3.2 (candidate 0)

2010-11-10 Thread Patrick Hunt

+1 based on my testing. Included testing various cluster sizes. 9
server cluster I manually verified that killing/restarting servers
worked properly, including going below majority count and then
restarting some servers. Clients maintained connectivity and recovered
sessions correctly.

LGTM.

Patrick

On Wed, Nov 10, 2010 at 11:09 AM, Mahadev Konar maha...@yahoo-inc.com wrote:
 +1 for the release.

 Ran ant test and a couple of smoke tests. Create znodes and shutdown
 zookeeper servers to test durability. Deleted znodes to make sure they are
 deleted. Shot down servers one at a time to confirm correct behavior.

 Thanks
 mahadev


 On 11/4/10 11:17 PM, Patrick Hunt ph...@apache.org wrote:

 I've created a candidate build for ZooKeeper 3.3.2. This is a bug fix
 release addressing twenty-six issues (eight critical) -- see the
 release notes for details.

 *** Please download, test and VOTE before the
 *** vote closes 11pm pacific time, Tuesday, November 9.***

 http://people.apache.org/~phunt/zookeeper-3.3.2-candidate-0/

 Should we release this?

 Patrick

Re: ZK's hudson patch queue being worked on

2010-11-10 Thread Patrick Hunt

The ZK qabot which tests the JIRA patch queue on hudson is again
operational. I have seen a few false failures in the c client
binding tests, Nigel is going to look into it.

You can see the results of tests on the JIRA (qabot appends as a
comment), also details/list here:
https://hudson.apache.org/hudson/view/S-Z/view/ZooKeeper/job/PreCommit-ZOOKEEPER-Build/

If you login to hudson you can actually test a particular JIRA
directly by clicking build now and entering the JIRA number into the
input box. Otw (if you do not have a login) you can cancel the patch
in JIRA and re-submit. Either will trigger the patch testing workflow.
Note, the most recent attachment to the JIRA will be tested.

Patrick

On Tue, Nov 9, 2010 at 10:43 AM, Patrick Hunt ph...@apache.org wrote:
Nigel and Giri are working on fixing ZK's patch queue on hudson. You
may see some spurious messages for a bit as we get things working
again.

This is the hudson process that tests new JIRA patches. When you
attach a patch to a JIRA and submit it hudson will automatically
verify the patch and comment on the jira. Here's an example:
https://issues.apache.org/jira/browse/ZOOKEEPER-780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12930211#action_12930211

Once hadoopqa bot has given it's blessing (typically +1 overall but
there are some exceptions) committers generally will start their
detailed review for commit.

Given this workflow/automation important that you follow the details
on our how to contribute page, esp in regards to creating the patch:
http://wiki.apache.org/hadoop/ZooKeeper/HowToContribute

If you have any questions please do let us know.

Regards,

Patrick

Re: What happens to a follower if leader hangs?

2010-11-10 Thread Patrick Hunt

I'd go 3.3.3 and 3.4.0. Any of this (incl the other issues
Vishal/others have been finding recently) point to some particular set
of testing we might add to find problems like this? What are we
missing?

Once 3.3.2 is out and immediate tlp issues are addressed I'm going to
start pushing for 3.4 regardless of whether everything is in yet or
not.

Patrick

On Wed, Nov 10, 2010 at 11:48 AM, Mahadev Konar maha...@yahoo-inc.com wrote:
 Hi Vishal,
  There are periodic pings sent from the leader to the followers.

 Take a look at Leader.java:

 syncedSet.add(self.getId());
                synchronized (learners) {
                    for (LearnerHandler f : learners) {
                        if (f.synced()) {
                            syncedCount++;
                            syncedSet.add(f.getSid());
                        }
                        f.ping();
                    }
                }


 This code sends periodic pings to the followers to make sure they are
 running fine. We should keep track of these pings and see if we havent seen
 a ping packet from the leader for a long time and give up following the
 leader in case we havent heard from him for a long time. This is definitely
 worth fixing since we pride ourselves in being a highly available and
 reliable service.

 Please feel free to open a jira and work on it.
 3.4 would be a good target for this.

 Thanks
 mahadev

 On 11/10/10 12:26 PM, Vishal Kher vishalm...@gmail.com wrote:

 Hi,

 In Follower.followLeader() after syncing with the leader, the follower does:
                 while (self.isRunning()) {
                     readPacket(qp);
                     processPacket(qp);
                 }

 It looks like it relies on socket timeout expiry to figure out if the
 connection with the leader has gone down.  So a follower *with no cilents*
 may never notice a faulty leader if a Leader has a software hang, but the
 TCP connections with the peers are still valid. Since it has not cilents, it
 won't hearbeat with the Leader. If majority of followers are not connected
 to any clients, then even if other followers attempt to elect a new leader
 after detecting that the leader is unresponsive.

 Please correct me if I am wrong. If I am not mistaken, should we add code at
 the follower to monitor the heartbeat messages that it receives from the
 leader and take action if it misses heartbeats for time  (syncLimit *
 tickTime)? This certainly is a hypothetical case, however, I think it is
 worth a fix.

 Thanks.
 -Vishal

[jira] Updated: (ZOOKEEPER-928) Follower should stop following and start FLE if it does not receive pings from the leader


 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patrick Hunt updated ZOOKEEPER-928:
---

  Component/s: server
   quorum
 Priority: Critical  (was: Major)
Fix Version/s: 3.3.3

 Follower should stop following and start FLE if it does not receive pings 
 from the leader
 -

 Key: ZOOKEEPER-928
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-928
 Project: Zookeeper
  Issue Type: Bug
  Components: quorum, server
Affects Versions: 3.3.2
Reporter: Vishal K
Priority: Critical
 Fix For: 3.3.3, 3.4.0


 In Follower.followLeader() after syncing with the leader, the follower does:
 while (self.isRunning()) {
 readPacket(qp);
 processPacket(qp);
 }
 It looks like it relies on socket timeout expiry to figure out if the 
 connection with the leader has gone down.  So a follower *with no cilents* 
 may never notice a faulty leader if a Leader has a software hang, but the TCP 
 connections with the peers are still valid. Since it has no cilents, it won't 
 hearbeat with the Leader. If majority of followers are not connected to any 
 clients, then FLE will fail even if other followers attempt to elect a new 
 leader.
 We should keep track of pings received from the leader and see if we havent 
 seen
 a ping packet from the leader for (syncLimit * tickTime) time and give up 
 following the
 leader.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (ZOOKEEPER-909) Extract NIO specific code from ClientCnxn


 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patrick Hunt updated ZOOKEEPER-909:
---

Resolution: Fixed
Status: Resolved  (was: Patch Available)

committed to trunk.

Thanks for following through on this Thomas! Look forward to seeing the rest of 
it. Regards.

 Extract NIO specific code from ClientCnxn
 -

 Key: ZOOKEEPER-909
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-909
 Project: Zookeeper
  Issue Type: Sub-task
  Components: java client
Reporter: Thomas Koch
Assignee: Thomas Koch
 Fix For: 3.4.0

 Attachments: ClientCnxnSocketNetty.java, ZOOKEEPER-909.patch, 
 ZOOKEEPER-909.patch, ZOOKEEPER-909.patch, ZOOKEEPER-909.patch, 
 ZOOKEEPER-909.patch


 This patch is mostly the same patch as my last one for ZOOKEEPER-823 minus 
 everything Netty related. This means this patch only extract all NIO specific 
 code in the class ClientCnxnSocketNIO which extends ClientCnxnSocket.
 I've redone this patch from current trunk step by step now and couldn't find 
 any logical error. I've already done a couple of successful test runs and 
 will continue to do so this night.
 It would be nice, if we could apply this patch as soon as possible to trunk. 
 This allows us to continue to work on the netty integration without blocking 
 the ClientCnxn class. Adding Netty after this patch should be only a matter 
 of adding the ClientCnxnSocketNetty class with the appropriate test cases.
 You could help me by reviewing the patch and by running it on whatever test 
 server you have available. Please send me any complete failure log you should 
 encounter to thomas at koch point ro. Thx!
 Update: Until now, I've collected 8 successful builds in a row!

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (ZOOKEEPER-928) Follower should stop following and start FLE if it does not receive pings from the leader

[
https://issues.apache.org/jira/browse/ZOOKEEPER-928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12930905#action_12930905
]

Patrick Hunt commented on ZOOKEEPER-928:

according to this it's not a bug:
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4614802

specifically:

The read methods in SocketChannel (and DatagramChannel) do not
support timeouts. If you need the timeout functionality then use the read
methods of the associated Socket (or DatagramSocket) object.

notice this was asked/answered a while ago though, however I suspect it's still
true.

Follower should stop following and start FLE if it does not receive pings
from the leader
-

Key: ZOOKEEPER-928
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-928
Project: Zookeeper
Issue Type: Bug
Components: quorum, server
Affects Versions: 3.3.2
Reporter: Vishal K
Priority: Critical

In Follower.followLeader() after syncing with the leader, the follower does:
while (self.isRunning()) {
readPacket(qp);
processPacket(qp);
}
It looks like it relies on socket timeout expiry to figure out if the
connection with the leader has gone down. So a follower *with no cilents*
may never notice a faulty leader if a Leader has a software hang, but the TCP
connections with the peers are still valid. Since it has no cilents, it won't
hearbeat with the Leader. If majority of followers are not connected to any
clients, then FLE will fail even if other followers attempt to elect a new
leader.
We should keep track of pings received from the leader and see if we havent
seen
a ping packet from the leader for (syncLimit * tickTime) time and give up
following the
leader.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Re: [VOTE] Release ZooKeeper 3.3.2 (candidate 0)

2010-11-10 Thread Patrick Hunt

With 6 +1's (3 PMC) and no -1's the vote passes. I'm working to
publish the release and will send announcements as soon as that's
done.

Patrick

On Wed, Nov 10, 2010 at 4:15 PM, Stack st...@duboce.net wrote:
 +1

 I put it up on a cluster under hbase and ran loads against it over
 last few hours.  Nothing untoward in logs.  Played around w/ zkcli.
 It seems to behaving same as 3.3.1.

 St.Ack


 On Wed, Nov 10, 2010 at 3:24 PM, Henry Robinson he...@cloudera.com wrote:
 +1

 Python looks good.

 On 10 November 2010 14:51, Michi Mutsuzaki mic...@yahoo-inc.com wrote:

 +1.

 I ran my benchmark test on the release candidate for one hour, and got
 similar numbers as 3.3.0.

 --Michi

 On 11/10/10 11:09 AM, Mahadev Konar maha...@yahoo-inc.com wrote:

  +1 for the release.
 
  Ran ant test and a couple of smoke tests. Create znodes and shutdown
  zookeeper servers to test durability. Deleted znodes to make sure they
 are
  deleted. Shot down servers one at a time to confirm correct behavior.
 
  Thanks
  mahadev
 
 
  On 11/4/10 11:17 PM, Patrick Hunt ph...@apache.org wrote:
 
  I've created a candidate build for ZooKeeper 3.3.2. This is a bug fix
  release addressing twenty-six issues (eight critical) -- see the
  release notes for details.
 
  *** Please download, test and VOTE before the
  *** vote closes 11pm pacific time, Tuesday, November 9.***
 
  http://people.apache.org/~phunt/zookeeper-3.3.2-candidate-0/
 
  Should we release this?
 
  Patrick
 
 
 




 --
 Henry Robinson
 Software Engineer
 Cloudera
 415-994-6679

[jira] Commented: (ZOOKEEPER-925) Consider maven site generation to replace our forrest site and documentation generation

[
https://issues.apache.org/jira/browse/ZOOKEEPER-925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12930188#action_12930188
]

Patrick Hunt commented on ZOOKEEPER-925:

There are a few issues:

A while back I looked at replacing forrest with python's sphinx, the conversion
itself was pretty straightforward given there was a script that did most of the
work. I don't see a script for forrest-confluence, perhaps we could re-purpose
the other one I used, or just do a manual search/replace of the tags. It will
take some work to convert the formats, but not huge given the size of our
forrest based docs.

Another issue was that we lost the hadoop lookfeel. This was really the
insurmountable problem when I looked at it before. However now that we are
moving out of hadoop into our own tlp space I don't see that as an issue.
Probably we want our own look/feel anyway.

Going to maven based site gen we just need to create the toplevel pom.xml file
and a toplevel src/site directory that contains the content and the
descriptor (how to generate the site, what links, etc... that's all
configurable). We can then tell people to use both ant and mvn for the time
being. mvn would initially just be mvn site (site/doc generation) and ant for
all the things we do today.

I can create a patch that does maven site generation if there's sufficient
interest (I don't want to waste my time though if everyone's not on board).
What do you think?

Consider maven site generation to replace our forrest site and documentation
generation
---

Key: ZOOKEEPER-925
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-925
Project: Zookeeper
Issue Type: Wish
Components: documentation
Reporter: Patrick Hunt

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (ZOOKEEPER-926) Fork Hadoop common's test-patch.sh and modify for Zookeeper


[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12930189#action_12930189
 ] 

Patrick Hunt commented on ZOOKEEPER-926:


+1 Thanks Nigel  Giri!

 Fork Hadoop common's test-patch.sh and modify for Zookeeper
 ---

 Key: ZOOKEEPER-926
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-926
 Project: Zookeeper
  Issue Type: Improvement
  Components: build
Reporter: Nigel Daley
 Fix For: 3.4.0

 Attachments: ZOOKEEPER-926.patch


 Zookeeper currently uses the test-patch.sh script from the Hadoop nightly 
 dir.  This is now out of date.  I propose we just copy the updated one in 
 Hadoop common and then modify for ZK.  This will also help as ZK moves out of 
 Hadoop to it's own TLP.  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Assigned: (ZOOKEEPER-926) Fork Hadoop common's test-patch.sh and modify for Zookeeper


 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patrick Hunt reassigned ZOOKEEPER-926:
--

Assignee: Nigel Daley

 Fork Hadoop common's test-patch.sh and modify for Zookeeper
 ---

 Key: ZOOKEEPER-926
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-926
 Project: Zookeeper
  Issue Type: Improvement
  Components: build
Reporter: Nigel Daley
Assignee: Nigel Daley
 Fix For: 3.4.0

 Attachments: ZOOKEEPER-926.patch


 Zookeeper currently uses the test-patch.sh script from the Hadoop nightly 
 dir.  This is now out of date.  I propose we just copy the updated one in 
 Hadoop common and then modify for ZK.  This will also help as ZK moves out of 
 Hadoop to it's own TLP.  

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (ZOOKEEPER-922) enable faster timeout of sessions in case of unexpected socket disconnect


 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patrick Hunt updated ZOOKEEPER-922:
---

Status: Patch Available  (was: Open)

 enable faster timeout of sessions in case of unexpected socket disconnect
 -

 Key: ZOOKEEPER-922
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-922
 Project: Zookeeper
  Issue Type: Improvement
  Components: server
Reporter: Camille Fournier
Assignee: Camille Fournier
 Fix For: 3.4.0

 Attachments: ZOOKEEPER-922.patch


 In the case when a client connection is closed due to socket error instead of 
 the client calling close explicitly, it would be nice to enable the session 
 associated with that client to time out faster than the negotiated session 
 timeout. This would enable a zookeeper ensemble that is acting as a dynamic 
 discovery provider to remove ephemeral nodes for crashed clients quickly, 
 while allowing for a longer heartbeat-based timeout for java clients that 
 need to do long stop-the-world GC. 
 I propose doing this by setting the timeout associated with the crashed 
 session to minSessionTimeout.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (ZOOKEEPER-905) enhance zkServer.sh for easier zookeeper automation-izing

[
https://issues.apache.org/jira/browse/ZOOKEEPER-905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12930191#action_12930191
]

Patrick Hunt commented on ZOOKEEPER-905:

Hi Nicholas, actually the issue is that you need to use svn to create the diff
(hudson doesn't like the fact that it doesn't know what .orig file is)

See this page for instructions (basically checkout svn, make change, do svn
diff)
http://wiki.apache.org/hadoop/ZooKeeper/HowToContribute

enhance zkServer.sh for easier zookeeper automation-izing
-

Attachments: zkServer.sh.diff

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (ZOOKEEPER-922) enable faster timeout of sessions in case of unexpected socket disconnect


[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12930193#action_12930193
 ] 

Patrick Hunt commented on ZOOKEEPER-922:


Hi Camille, the patch has to be created from the top most directory (trunk) 
for hudson to apply the patch correctly, please see:
http://wiki.apache.org/hadoop/ZooKeeper/HowToContribute

(basically checkout trunk, make changes, do svn diff at the toplevel)

Thanks!

 enable faster timeout of sessions in case of unexpected socket disconnect
 -

 Key: ZOOKEEPER-922
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-922
 Project: Zookeeper
  Issue Type: Improvement
  Components: server
Reporter: Camille Fournier
Assignee: Camille Fournier
 Fix For: 3.4.0

 Attachments: ZOOKEEPER-922.patch


 In the case when a client connection is closed due to socket error instead of 
 the client calling close explicitly, it would be nice to enable the session 
 associated with that client to time out faster than the negotiated session 
 timeout. This would enable a zookeeper ensemble that is acting as a dynamic 
 discovery provider to remove ephemeral nodes for crashed clients quickly, 
 while allowing for a longer heartbeat-based timeout for java clients that 
 need to do long stop-the-world GC. 
 I propose doing this by setting the timeout associated with the crashed 
 session to minSessionTimeout.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (ZOOKEEPER-909) Extract NIO specific code from ClientCnxn

[
https://issues.apache.org/jira/browse/ZOOKEEPER-909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12930195#action_12930195
]

Patrick Hunt commented on ZOOKEEPER-909:

Hi Thomas, thanks! One more request, can you make this a single diff? Otw the
hudson patch queue doesn't work properly.
http://wiki.apache.org/hadoop/ZooKeeper/HowToContribute

(you probably need to svn add the new file for svn diff to pick it up)

Then cancel/submit the jira again to trigger the workflow.

Thanks!

Extract NIO specific code from ClientCnxn
-

Key: ZOOKEEPER-909
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-909
Project: Zookeeper
Issue Type: Sub-task
Components: java client
Reporter: Thomas Koch
Assignee: Thomas Koch
Fix For: 3.4.0

Attachments: ClientCnxnSocketNetty.java, ZOOKEEPER-909.patch,
ZOOKEEPER-909.patch, ZOOKEEPER-909.patch, ZOOKEEPER-909.patch

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (ZOOKEEPER-913) Version parser fails to parse 3.3.2-dev from build.xml.


[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12930216#action_12930216
 ] 

Patrick Hunt commented on ZOOKEEPER-913:


I propose we move to support the same format as maven supports:
http://www.sonatype.com/books/mvnref-book/reference/pom-relationships-sect-pom-syntax.html

major version.minor version.incremental version-qualifier

this is close to what we do, but allows for any qualifier after the x.y.z-


 Version parser fails to parse 3.3.2-dev from build.xml.
 -

 Key: ZOOKEEPER-913
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-913
 Project: Zookeeper
  Issue Type: Bug
  Components: build
Affects Versions: 3.3.1
Reporter: Anthony Urso
Assignee: Anthony Urso
Priority: Critical
 Fix For: 3.4.0

 Attachments: zk-build.patch, zk-version.patch


 Cannot build 3.3.1 from release tarball do to VerGen parser inability to 
 parse 3.3.2-dev.
 version-info:
  [java] All version-related parameters must be valid integers!
  [java] Exception in thread main java.lang.NumberFormatException: For 
 input string: 2-dev
  [java]   at 
 java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
  [java]   at java.lang.Integer.parseInt(Integer.java:481)
  [java]   at java.lang.Integer.parseInt(Integer.java:514)
  [java]   at 
 org.apache.zookeeper.version.util.VerGen.main(VerGen.java:131)
  [java] Java Result: 1

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[Discussion] Some proposed logging (log4j) JIRAs

2010-11-09 Thread Patrick Hunt

I wanted to highlight a couple recent JIRAs that may have impact on
users (api consumers AND admins of the service) in the 3.4 timeframe.
If you want to weigh in please comment on the respective jira:

1) proposal to move to slf4j (remove/replace log4j)
https://issues.apache.org/jira/browse/ZOOKEEPER-850

from user perspective not much should change as slf4j has full support
for log4j as an engine. But I'm not fully versed on every
particular.

Note that hbase is in the process of moving
https://issues.apache.org/jira/browse/HBASE-2608 and Avro has already
moved to slf4j, not sure about some of the other hadoop ex-subs.


2) on a related note. We did a bunch of work in the 3.3 timeframe to
improve logging where the severity levels of log messages tended to be
too verbose (many items which should have been debug/trace were info).
Much of this was based on feedback we received from the hbase
community. However there are still some rough edges.

A recent JIRA
https://issues.apache.org/jira/browse/ZOOKEEPER-912
is proposing some additional changes. It would be good for
users/admins (consumers of the client api and those involved with
running the service itself) to weigh in if they have any
insights/preferences. My primary concern is that we are still able to
help users when they run into trouble - ie sufficient logging at info
level, not losing critical detail in the weeds of debug/trace level.
It's unfortunate that we only have 3 levels to play with here. FF to
weigh in.

Regards,

Patrick

[jira] Commented: (ZOOKEEPER-902) Fix findbug issue in trunk Malicious code vulnerability


[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12930222#action_12930222
 ] 

Patrick Hunt commented on ZOOKEEPER-902:


The patch queue now has a setting:

(10:28:53 AM) nigelcdn: There's a new file in 
src/java/test/bin/test-patch.properties in which is defined the acceptable 
number of warnings
(10:29:03 AM) nigelcdn: use it very judiciously ;-)

after this issue is fixed we should adjust that file back to 0.


 Fix findbug issue in trunk Malicious code vulnerability
 -

 Key: ZOOKEEPER-902
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-902
 Project: Zookeeper
  Issue Type: Bug
  Components: quorum, server
Affects Versions: 3.4.0
Reporter: Patrick Hunt
Priority: Minor
 Fix For: 3.4.0


 https://hudson.apache.org/hudson/view/ZooKeeper/job/ZooKeeper-trunk/970/artifact/trunk/findbugs/zookeeper-findbugs-report.html#Warnings_MALICIOUS_CODE
 Malicious code vulnerability Warnings
 Code  Warning
 MSorg.apache.zookeeper.server.quorum.LeaderElection.epochGen isn't final 
 but should be

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (ZOOKEEPER-925) Consider maven site generation to replace our forrest site and documentation generation

[
https://issues.apache.org/jira/browse/ZOOKEEPER-925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12930224#action_12930224
]

Patrick Hunt commented on ZOOKEEPER-925:

We prolly wouldn't generate pdfs for the web site, no one seems to do that
anymore (although it's possible if someone would want to do it explicitly for
some reason)

We check in the source, that's a given.

We check in the generated site/docs for a reason as well. In forrest timeframe
it was mainly due to the fact that using forrest is a pita. ;-) In maven that's
less of a concern. For whirr we currently don't checkin the generated, but we
are thinking of doing so to lower the bar for new users. Doesn't really matter
much to me, we could try not committing generated first, then see if it's an
issue.

Consider maven site generation to replace our forrest site and documentation
generation
---

Key: ZOOKEEPER-925
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-925
Project: Zookeeper
Issue Type: Wish
Components: documentation
Reporter: Patrick Hunt

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (ZOOKEEPER-925) Consider maven site generation to replace our forrest site and documentation generation


[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12930226#action_12930226
 ] 

Patrick Hunt commented on ZOOKEEPER-925:


btw, given you just checkout the repo and literally type mvn site (maven 
handles all the dependency d/l) there's basically no reason to commit the 
generated docs.


 Consider maven site generation to replace our forrest site and documentation 
 generation
 ---

 Key: ZOOKEEPER-925
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-925
 Project: Zookeeper
  Issue Type: Wish
  Components: documentation
Reporter: Patrick Hunt

 See WHIRR-19 for some background.
 In whirr we looked at a number of site/doc generation facilities. In the end 
 Maven site generation plugin turned out to be by far the best option. You can 
 see our nascent site here (no attempt at styling,etc so far):
 http://incubator.apache.org/whirr/
 In particular take a look at the quick start:
 http://incubator.apache.org/whirr/quick-start-guide.html
 which was generated from
 http://svn.apache.org/repos/asf/incubator/whirr/trunk/src/site/confluence/quick-start-guide.confluence
 notice this was standard wiki markup (confluence wiki markup, same as 
 available from apache)
 You can read more about mvn site plugin here:
 http://maven.apache.org/guides/mini/guide-site.html
 Notice that other formats are available, not just confluence markup, also 
 note that you can use different markup formats if you like in the same site 
 (although probably not a great idea, but in some cases might be handy, for 
 example whirr uses the confluence wiki, so we can pretty much copy/paste 
 source docs from wiki to our site (svn) if we like)
 Re maven vs our current ant based build. It's probably a good idea for us to 
 move the build to maven at some point. We could initially move just the doc 
 generation, and then incrementally move functionality from build.xml to mvn 
 over a longer time period.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

ZK's hudson patch queue being worked on

2010-11-09 Thread Patrick Hunt

Nigel and Giri are working on fixing ZK's patch queue on hudson. You
may see some spurious messages for a bit as we get things working
again.

This is the hudson process that tests new JIRA patches. When you
attach a patch to a JIRA and submit it hudson will automatically
verify the patch and comment on the jira. Here's an example:
https://issues.apache.org/jira/browse/ZOOKEEPER-780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12930211#action_12930211

Once hadoopqa bot has given it's blessing (typically +1 overall but
there are some exceptions) committers generally will start their
detailed review for commit.

Given this workflow/automation important that you follow the details
on our how to contribute page, esp in regards to creating the patch:
http://wiki.apache.org/hadoop/ZooKeeper/HowToContribute

If you have any questions please do let us know.

Regards,

Patrick

[jira] Commented: (ZOOKEEPER-850) Switch from log4j to slf4j


[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12930229#action_12930229
 ] 

Patrick Hunt commented on ZOOKEEPER-850:


fyi hbase's jira for similar change: HBASE-2608 (looks like this patch is 
proposing similar to 2 in hbase jira?)

 Switch from log4j to slf4j
 --

 Key: ZOOKEEPER-850
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-850
 Project: Zookeeper
  Issue Type: Improvement
  Components: java client
Affects Versions: 3.3.1
Reporter: Olaf Krische
Assignee: Olaf Krische
 Fix For: 3.4.0

 Attachments: ZOOKEEPER-3.3.1-log4j-slf4j-20101031.patch.bz2, 
 ZOOKEEPER-3.4.0-log4j-slf4j-20101102.patch.bz2


 Hello,
 i would like to see slf4j integrated into the zookeeper instead of relying 
 explicitly on log4j.
 slf4j is an abstract logging framework. There are adapters from slf4j to many 
 logger implementations, one of them is log4j.
 The decision which log engine to use i dont like to make so early.
 This would help me to embed zookeeper in my own applications (which use a 
 different logger implemenation, but slf4j is the basis)
 What do you think?
 (as i can see, those slf4j request flood all other projects on apache as well 
 :-)
 Maybe for 3.4 or 4.0?
 I can offer a patchset, i have experience in such an migration already. :-)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (ZOOKEEPER-850) Switch from log4j to slf4j


 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patrick Hunt updated ZOOKEEPER-850:
---

Attachment: ZOOKEEPER-850.patch

Attaching Olaf's patch as raw patch file so that hudson can do it's magic.

 Switch from log4j to slf4j
 --

 Key: ZOOKEEPER-850
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-850
 Project: Zookeeper
  Issue Type: Improvement
  Components: java client
Affects Versions: 3.3.1
Reporter: Olaf Krische
Assignee: Olaf Krische
 Fix For: 3.4.0

 Attachments: ZOOKEEPER-3.3.1-log4j-slf4j-20101031.patch.bz2, 
 ZOOKEEPER-3.4.0-log4j-slf4j-20101102.patch.bz2, ZOOKEEPER-850.patch


 Hello,
 i would like to see slf4j integrated into the zookeeper instead of relying 
 explicitly on log4j.
 slf4j is an abstract logging framework. There are adapters from slf4j to many 
 logger implementations, one of them is log4j.
 The decision which log engine to use i dont like to make so early.
 This would help me to embed zookeeper in my own applications (which use a 
 different logger implemenation, but slf4j is the basis)
 What do you think?
 (as i can see, those slf4j request flood all other projects on apache as well 
 :-)
 Maybe for 3.4 or 4.0?
 I can offer a patchset, i have experience in such an migration already. :-)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (ZOOKEEPER-850) Switch from log4j to slf4j

[
https://issues.apache.org/jira/browse/ZOOKEEPER-850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12930232#action_12930232
]

Patrick Hunt commented on ZOOKEEPER-850:

Olaf, re 5 could you add similar comments to this JIRA in the release notes
section? We'll need that when doing the release itself.

Switch from log4j to slf4j
--

Key: ZOOKEEPER-850
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-850
Project: Zookeeper
Issue Type: Improvement
Components: java client
Affects Versions: 3.3.1
Reporter: Olaf Krische
Assignee: Olaf Krische
Fix For: 3.4.0

Attachments: ZOOKEEPER-3.3.1-log4j-slf4j-20101031.patch.bz2,
ZOOKEEPER-3.4.0-log4j-slf4j-20101102.patch.bz2, ZOOKEEPER-850.patch

Hello,
i would like to see slf4j integrated into the zookeeper instead of relying
explicitly on log4j.
slf4j is an abstract logging framework. There are adapters from slf4j to many
logger implementations, one of them is log4j.
The decision which log engine to use i dont like to make so early.
This would help me to embed zookeeper in my own applications (which use a
different logger implemenation, but slf4j is the basis)
What do you think?
(as i can see, those slf4j request flood all other projects on apache as well
:-)
Maybe for 3.4 or 4.0?
I can offer a patchset, i have experience in such an migration already. :-)

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (ZOOKEEPER-925) Consider maven site generation to replace our forrest site and documentation generation


 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patrick Hunt updated ZOOKEEPER-925:
---

Attachment: ZOOKEEPER-925.patch

apply this patch, then in the toplevel directory type mvn site:site, then 
open target/site/index.html in your browser.

Notice the index.confluence src page, try editing that (confluence wiki markup 
http://maven.apache.org/doxia/modules/index.html#Confluence) and 
regenerating/viewing the updated site. site.xml controls the layout and which 
links are put into the generated site.


 Consider maven site generation to replace our forrest site and documentation 
 generation
 ---

 Key: ZOOKEEPER-925
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-925
 Project: Zookeeper
  Issue Type: Wish
  Components: documentation
Reporter: Patrick Hunt
 Attachments: ZOOKEEPER-925.patch


 See WHIRR-19 for some background.
 In whirr we looked at a number of site/doc generation facilities. In the end 
 Maven site generation plugin turned out to be by far the best option. You can 
 see our nascent site here (no attempt at styling,etc so far):
 http://incubator.apache.org/whirr/
 In particular take a look at the quick start:
 http://incubator.apache.org/whirr/quick-start-guide.html
 which was generated from
 http://svn.apache.org/repos/asf/incubator/whirr/trunk/src/site/confluence/quick-start-guide.confluence
 notice this was standard wiki markup (confluence wiki markup, same as 
 available from apache)
 You can read more about mvn site plugin here:
 http://maven.apache.org/guides/mini/guide-site.html
 Notice that other formats are available, not just confluence markup, also 
 note that you can use different markup formats if you like in the same site 
 (although probably not a great idea, but in some cases might be handy, for 
 example whirr uses the confluence wiki, so we can pretty much copy/paste 
 source docs from wiki to our site (svn) if we like)
 Re maven vs our current ant based build. It's probably a good idea for us to 
 move the build to maven at some point. We could initially move just the doc 
 generation, and then incrementally move functionality from build.xml to mvn 
 over a longer time period.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Assigned: (ZOOKEEPER-925) Consider maven site generation to replace our forrest site and documentation generation


 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patrick Hunt reassigned ZOOKEEPER-925:
--

Assignee: Patrick Hunt

 Consider maven site generation to replace our forrest site and documentation 
 generation
 ---

 Key: ZOOKEEPER-925
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-925
 Project: Zookeeper
  Issue Type: Wish
  Components: documentation
Reporter: Patrick Hunt
Assignee: Patrick Hunt
 Attachments: ZOOKEEPER-925.patch


 See WHIRR-19 for some background.
 In whirr we looked at a number of site/doc generation facilities. In the end 
 Maven site generation plugin turned out to be by far the best option. You can 
 see our nascent site here (no attempt at styling,etc so far):
 http://incubator.apache.org/whirr/
 In particular take a look at the quick start:
 http://incubator.apache.org/whirr/quick-start-guide.html
 which was generated from
 http://svn.apache.org/repos/asf/incubator/whirr/trunk/src/site/confluence/quick-start-guide.confluence
 notice this was standard wiki markup (confluence wiki markup, same as 
 available from apache)
 You can read more about mvn site plugin here:
 http://maven.apache.org/guides/mini/guide-site.html
 Notice that other formats are available, not just confluence markup, also 
 note that you can use different markup formats if you like in the same site 
 (although probably not a great idea, but in some cases might be handy, for 
 example whirr uses the confluence wiki, so we can pretty much copy/paste 
 source docs from wiki to our site (svn) if we like)
 Re maven vs our current ant based build. It's probably a good idea for us to 
 move the build to maven at some point. We could initially move just the doc 
 generation, and then incrementally move functionality from build.xml to mvn 
 over a longer time period.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Assigned: (ZOOKEEPER-913) Version parser fails to parse 3.3.2-dev from build.xml.


 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patrick Hunt reassigned ZOOKEEPER-913:
--

Assignee: Patrick Hunt  (was: Anthony Urso)

 Version parser fails to parse 3.3.2-dev from build.xml.
 -

 Key: ZOOKEEPER-913
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-913
 Project: Zookeeper
  Issue Type: Bug
  Components: build
Affects Versions: 3.3.1
Reporter: Anthony Urso
Assignee: Patrick Hunt
Priority: Critical
 Fix For: 3.4.0

 Attachments: zk-build.patch, zk-version.patch


 Cannot build 3.3.1 from release tarball do to VerGen parser inability to 
 parse 3.3.2-dev.
 version-info:
  [java] All version-related parameters must be valid integers!
  [java] Exception in thread main java.lang.NumberFormatException: For 
 input string: 2-dev
  [java]   at 
 java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
  [java]   at java.lang.Integer.parseInt(Integer.java:481)
  [java]   at java.lang.Integer.parseInt(Integer.java:514)
  [java]   at 
 org.apache.zookeeper.version.util.VerGen.main(VerGen.java:131)
  [java] Java Result: 1

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (ZOOKEEPER-913) Version parser fails to parse 3.3.2-dev from build.xml.


 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patrick Hunt updated ZOOKEEPER-913:
---

Fix Version/s: 3.3.3

 Version parser fails to parse 3.3.2-dev from build.xml.
 -

 Key: ZOOKEEPER-913
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-913
 Project: Zookeeper
  Issue Type: Bug
  Components: build
Affects Versions: 3.3.1
Reporter: Anthony Urso
Assignee: Patrick Hunt
Priority: Critical
 Fix For: 3.3.3, 3.4.0

 Attachments: zk-build.patch, zk-version.patch


 Cannot build 3.3.1 from release tarball do to VerGen parser inability to 
 parse 3.3.2-dev.
 version-info:
  [java] All version-related parameters must be valid integers!
  [java] Exception in thread main java.lang.NumberFormatException: For 
 input string: 2-dev
  [java]   at 
 java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
  [java]   at java.lang.Integer.parseInt(Integer.java:481)
  [java]   at java.lang.Integer.parseInt(Integer.java:514)
  [java]   at 
 org.apache.zookeeper.version.util.VerGen.main(VerGen.java:131)
  [java] Java Result: 1

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (ZOOKEEPER-913) Version parser fails to parse 3.3.2-dev from build.xml.


 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patrick Hunt updated ZOOKEEPER-913:
---

Status: Patch Available  (was: Open)

 Version parser fails to parse 3.3.2-dev from build.xml.
 -

 Key: ZOOKEEPER-913
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-913
 Project: Zookeeper
  Issue Type: Bug
  Components: build
Affects Versions: 3.3.1
Reporter: Anthony Urso
Assignee: Patrick Hunt
Priority: Critical
 Fix For: 3.3.3, 3.4.0

 Attachments: zk-build.patch, zk-version.patch, ZOOKEEPER-913.patch


 Cannot build 3.3.1 from release tarball do to VerGen parser inability to 
 parse 3.3.2-dev.
 version-info:
  [java] All version-related parameters must be valid integers!
  [java] Exception in thread main java.lang.NumberFormatException: For 
 input string: 2-dev
  [java]   at 
 java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
  [java]   at java.lang.Integer.parseInt(Integer.java:481)
  [java]   at java.lang.Integer.parseInt(Integer.java:514)
  [java]   at 
 org.apache.zookeeper.version.util.VerGen.main(VerGen.java:131)
  [java] Java Result: 1

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (ZOOKEEPER-913) Version parser fails to parse 3.3.2-dev from build.xml.


 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patrick Hunt updated ZOOKEEPER-913:
---

Attachment: ZOOKEEPER-913.patch

this most recent patch implements support for a version format similar to what 
maven does. 

Added tests, also verified on the command line (build) using -Dversion option. 
Seems to work ok.

 Version parser fails to parse 3.3.2-dev from build.xml.
 -

 Key: ZOOKEEPER-913
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-913
 Project: Zookeeper
  Issue Type: Bug
  Components: build
Affects Versions: 3.3.1
Reporter: Anthony Urso
Assignee: Patrick Hunt
Priority: Critical
 Fix For: 3.3.3, 3.4.0

 Attachments: zk-build.patch, zk-version.patch, ZOOKEEPER-913.patch


 Cannot build 3.3.1 from release tarball do to VerGen parser inability to 
 parse 3.3.2-dev.
 version-info:
  [java] All version-related parameters must be valid integers!
  [java] Exception in thread main java.lang.NumberFormatException: For 
 input string: 2-dev
  [java]   at 
 java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
  [java]   at java.lang.Integer.parseInt(Integer.java:481)
  [java]   at java.lang.Integer.parseInt(Integer.java:514)
  [java]   at 
 org.apache.zookeeper.version.util.VerGen.main(VerGen.java:131)
  [java] Java Result: 1

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Resolved: (ZOOKEEPER-891) Allow non-numeric version strings


 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patrick Hunt resolved ZOOKEEPER-891.


Resolution: Duplicate

This is a dup of ZOOKEEPER-913  See the patch there (handles this case).

 Allow non-numeric version strings
 -

 Key: ZOOKEEPER-891
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-891
 Project: Zookeeper
  Issue Type: Improvement
  Components: build
Reporter: Eli Collins
Priority: Minor
 Fix For: 3.4.0


 Non-numeric version strings (eg -dev) or -are not currently accepted, you 
 either get an error (Invalid version number format, must be x.y.z) or if 
 you pass x.y.z-dev or x.y.z+1 you'll get a NumberFormatException.  Would be 
 useful to allow non-numeric versions. 
 {noformat}
 version-info:
  [java] All version-related parameters must be valid integers!
  [java] Exception in thread main java.lang.NumberFormatException: For 
 input string: 3-dev
  [java]   at 
 java.lang.NumberFormatException.forInputString(NumberFormatException.java:48)
  [java]   at java.lang.Integer.parseInt(Integer.java:458)
  [java]   at java.lang.Integer.parseInt(Integer.java:499)
  [java]   at 
 org.apache.zookeeper.version.util.VerGen.main(VerGen.java:131)
  [java] Java Result: 1
 {noformat}

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (ZOOKEEPER-920) L7 (application layer) ping support

[
https://issues.apache.org/jira/browse/ZOOKEEPER-920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12930399#action_12930399
]

Patrick Hunt commented on ZOOKEEPER-920:

In general unit tests for the c binding is an area we could use more help with
(more tests). If you're interested. :-)

L7 (application layer) ping support
---

Key: ZOOKEEPER-920
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-920
Project: Zookeeper
Issue Type: New Feature
Components: c client
Affects Versions: 3.3.1
Reporter: Chang Song
Assignee: Chang Song
Priority: Minor
Fix For: 3.4.0

Attachments: ZOOKEEPER-920.patch

Zookeeper is used in applications where fault tolerance is important. Its
client i/o thread send/recv heartbeats to/fro Zookeeper ensemble to stay
connected. However healthy heartbeat does not always means that the
application that uses Zookeeper client is in good health, it only means that
ZK client thread is in good health.
This I needed something that can tagged onto Zookeeper ping that represents
L7 (application) health as well.
I have modified C client source to support this in minimal way. I am new to
Zookeeper, so please code review this code. I am actually using this code in
our in-house solution.
https://github.com/tru64ufs/zookeeper/commit/2196d6d5114a2fd2c0a3bc9a55f4494d47d2aece
Thank you very much.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (ZOOKEEPER-922) enable faster timeout of sessions in case of unexpected socket disconnect

[
https://issues.apache.org/jira/browse/ZOOKEEPER-922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12930407#action_12930407
]

Patrick Hunt commented on ZOOKEEPER-922:

@camille NP, although it makes it easier for us (reviewers) if all the patches
are consistent. For future reference then. Thanks.

ps. you might get more insightful review if you post to apache's new
reviewboard server: https://reviews.apache.org

Regards.

enable faster timeout of sessions in case of unexpected socket disconnect
-

Key: ZOOKEEPER-922
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-922
Project: Zookeeper
Issue Type: Improvement
Components: server
Reporter: Camille Fournier
Assignee: Camille Fournier
Fix For: 3.4.0

Attachments: ZOOKEEPER-922.patch

In the case when a client connection is closed due to socket error instead of
the client calling close explicitly, it would be nice to enable the session
associated with that client to time out faster than the negotiated session
timeout. This would enable a zookeeper ensemble that is acting as a dynamic
discovery provider to remove ephemeral nodes for crashed clients quickly,
while allowing for a longer heartbeat-based timeout for java clients that
need to do long stop-the-world GC.
I propose doing this by setting the timeout associated with the crashed
session to minSessionTimeout.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (ZOOKEEPER-906) Improve C client connection reliability by making it sleep between reconnect attempts as in Java Client

[
https://issues.apache.org/jira/browse/ZOOKEEPER-906?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Patrick Hunt updated ZOOKEEPER-906:
---

Status: Open (was: Patch Available)

Cancelling patch, needs test(s).

Improve C client connection reliability by making it sleep between reconnect
attempts as in Java Client
---

Key: ZOOKEEPER-906
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-906
Project: Zookeeper
Issue Type: Improvement
Components: c client
Affects Versions: 3.3.1
Reporter: Radu Marin
Assignee: Radu Marin
Fix For: 3.4.0

Attachments: ZOOKEEPER-906.patch

Original Estimate: 24h
Remaining Estimate: 24h

Currently, when a C client get disconnected, it retries a couple of hosts
(not all) with no delay between attempts and then if it doesn't succeed it
sleeps for 1/3 session expiration timeout period before trying again.
In the worst case the disconnect event can occur after 2/3 of session
expiration timeout has past, and sleeping for even more 1/3 session timeout
will cause a session loss in most of the times.
A better approach is to check all hosts but with random delay between
reconnect attempts. Also the delay must be independent of session timeout so
if we increase the session timeout we also increase the number of available
attempts.
This improvement covers the case when the C client experiences network
problems for a short period of time and is not able to reach any zookeeper
hosts.
Java client already uses this logic and works very good.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (ZOOKEEPER-877) zkpython does not work with python3.1


 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patrick Hunt updated ZOOKEEPER-877:
---

Status: Open  (was: Patch Available)

Hi TuxRacer, is it possible for you to re-submit this as a single patch file? 
We generally request all changes in that format, to ensure that it's committed 
the way you intended (also helps with a bunch of other things like reviewing 
and hudsonqabot, etc...) here are the basic details: 
http://wiki.apache.org/hadoop/ZooKeeper/HowToContribute
(basically: svn checkout the code, make changes, svn diff and submit the 
result as ZOOKEEPER-877.patch, you may need to svn add if you are adding new 
files).

 zkpython does not work with python3.1
 -

 Key: ZOOKEEPER-877
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-877
 Project: Zookeeper
  Issue Type: Bug
  Components: contrib-bindings
Affects Versions: 3.3.1
 Environment: linux+python3.1
Reporter: TuxRacer
Assignee: TuxRacer
 Fix For: 3.4.0

 Attachments: Doc.tgz, tests_py3k.tgz, zookeeper.c, 
 zookeeper.c.patch.v1, zookeeper.c.patch.v2, zookeeper.c.v2, zookeeper.rst


 as written in the contrib/zkpython/README file:
 Python = 2.6 is required. We have tested against 2.6. We have not tested 
 against 3.x.
 this is probably more a 'new feature' request than a bug; anyway compiling 
 the pythn module and calling it returns an error at load time:
 python3.1
 Python 3.1.2 (r312:79147, May  8 2010, 16:36:46) 
 [GCC 4.4.4] on linux2
 Type help, copyright, credits or license for more information.
  import zookeeper
 Traceback (most recent call last):
   File stdin, line 1, in module
 ImportError: /usr/local/lib/python3.1/dist-packages/zookeeper.so: undefined 
 symbol: PyString_AsString
 are there any plan to support Python3.X?
 I also tried to write a 3.1 ctypes wrapper but the C API seems in fact to be 
 written in C++, so python ctypes cannot be used.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (ZOOKEEPER-874) FileTxnSnapLog.restore does not call listener


 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patrick Hunt updated ZOOKEEPER-874:
---

Status: Open  (was: Patch Available)

Cancelling patch, could you provide a tests that verifies this issue is 
addressed? Thanks.

 FileTxnSnapLog.restore does not call listener
 -

 Key: ZOOKEEPER-874
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-874
 Project: Zookeeper
  Issue Type: Bug
  Components: leaderElection
Affects Versions: 3.3.1
Reporter: Diogo
Assignee: Diogo
Priority: Trivial
 Fix For: 3.4.0

 Attachments: ZOOKEEPER-874.patch


 FileTxnSnapLog.restore() does not call listener passed as parameter. The 
 result is that the commitLogs list is empty. When a follower connects to the 
 leader, the leader is forced to send a snapshot to the follower instead of a 
 couple of requests and commits.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (ZOOKEEPER-823) update ZooKeeper java client to optionally use Netty for connections


 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patrick Hunt updated ZOOKEEPER-823:
---

Status: Open  (was: Patch Available)

cancelling for now given Thomas is working on this via a new avenue.

 update ZooKeeper java client to optionally use Netty for connections
 

 Key: ZOOKEEPER-823
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-823
 Project: Zookeeper
  Issue Type: New Feature
  Components: java client
Reporter: Patrick Hunt
Assignee: Patrick Hunt
 Fix For: 3.4.0

 Attachments: NettyNettySuiteTest.rtf, 
 TEST-org.apache.zookeeper.test.NettyNettySuiteTest.txt.gz, 
 testDisconnectedAddAuth_FAILURE, testWatchAutoResetWithPending_FAILURE, 
 ZOOKEEPER-823.patch, ZOOKEEPER-823.patch, ZOOKEEPER-823.patch, 
 ZOOKEEPER-823.patch, ZOOKEEPER-823.patch, ZOOKEEPER-823.patch, 
 ZOOKEEPER-823.patch, ZOOKEEPER-823.patch, ZOOKEEPER-823.patch, 
 ZOOKEEPER-823.patch, ZOOKEEPER-823.patch


 This jira will port the client side connection code to use netty rather than 
 direct nio.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (ZOOKEEPER-784) server-side functionality for read-only mode


 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patrick Hunt updated ZOOKEEPER-784:
---

Status: Open  (was: Patch Available)

Looks like the patch is failing to apply in one hunk, please resubmit. thanks.

 [exec] 1 out of 2 hunks FAILED -- saving rejects to file 
src/java/main/org/apache/zookeeper/Watcher.java.rej


 server-side functionality for read-only mode
 

 Key: ZOOKEEPER-784
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-784
 Project: Zookeeper
  Issue Type: Sub-task
  Components: server
Reporter: Sergey Doroshenko
Assignee: Sergey Doroshenko
 Fix For: 3.4.0

 Attachments: ZOOKEEPER-784.patch, ZOOKEEPER-784.patch, 
 ZOOKEEPER-784.patch, ZOOKEEPER-784.patch, ZOOKEEPER-784.patch, 
 ZOOKEEPER-784.patch, ZOOKEEPER-784.patch, ZOOKEEPER-784.patch, 
 ZOOKEEPER-784.patch, ZOOKEEPER-784.patch


 As per http://wiki.apache.org/hadoop/ZooKeeper/GSoCReadOnlyMode , create 
 ReadOnlyZooKeeperServer which comes into play when peer is partitioned.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (ZOOKEEPER-780) zkCli.sh generates a ArrayIndexOutOfBoundsException


 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patrick Hunt updated ZOOKEEPER-780:
---

Status: Open  (was: Patch Available)

Andrei, this is a good fix, could you create a test for this? Thanks.

 zkCli.sh  generates a ArrayIndexOutOfBoundsException 
 -

 Key: ZOOKEEPER-780
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-780
 Project: Zookeeper
  Issue Type: Bug
  Components: scripts
Affects Versions: 3.3.1
 Environment: Linux Ubuntu running in VMPlayer on top of Windows XP
Reporter: Miguel Correia
Assignee: Andrei Savu
Priority: Minor
 Fix For: 3.4.0

 Attachments: ZOOKEEPER-780.patch, ZOOKEEPER-780.patch, 
 ZOOKEEPER-780.patch


 I'm starting to play with Zookeeper so I'm still running it in standalone 
 mode. This is not a big issue, but here it goes for the records. 
 I've run zkCli.sh to run some commands in the server. I created a znode 
 /groups. When I tried to create a znode client_1 inside /groups, I forgot to 
 include the data: an exception was generated and zkCli-sh crashed, instead of 
 just showing an error. I tried a few variations and it seems like the problem 
 is not including the data.
 A copy of the screen:
 [zk: localhost:2181(CONNECTED) 3] create /groups firstgroup
 Created /groups
 [zk: localhost:2181(CONNECTED) 4] create -e /groups/client_1
 Exception in thread main java.lang.ArrayIndexOutOfBoundsException: 3
   at 
 org.apache.zookeeper.ZooKeeperMain.processZKCmd(ZooKeeperMain.java:678)
   at org.apache.zookeeper.ZooKeeperMain.processCmd(ZooKeeperMain.java:581)
   at 
 org.apache.zookeeper.ZooKeeperMain.executeLine(ZooKeeperMain.java:353)
   at org.apache.zookeeper.ZooKeeperMain.run(ZooKeeperMain.java:311)
   at org.apache.zookeeper.ZooKeeperMain.main(ZooKeeperMain.java:270)

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (ZOOKEEPER-740) zkpython leading to segfault on zookeeper


 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patrick Hunt updated ZOOKEEPER-740:
---

Status: Open  (was: Patch Available)

Looks like the patch is failing to apply. Could someone update and resubmit?

 zkpython leading to segfault on zookeeper
 -

 Key: ZOOKEEPER-740
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-740
 Project: Zookeeper
  Issue Type: Bug
Affects Versions: 3.3.0
Reporter: Federico
Assignee: Henry Robinson
Priority: Critical
 Fix For: 3.4.0

 Attachments: ZOOKEEPER-740.patch


 The program that we are implementing uses the python binding for zookeeper 
 but sometimes it crash with segfault; here is the bt from gdb:
 Program received signal SIGSEGV, Segmentation fault.
 [Switching to Thread 0xad244b70 (LWP 28216)]
 0x080611d5 in PyObject_Call (func=0x862fab0, arg=0x8837194, kw=0x0)
 at ../Objects/abstract.c:2488
 2488../Objects/abstract.c: No such file or directory.
 in ../Objects/abstract.c
 (gdb) bt
 #0  0x080611d5 in PyObject_Call (func=0x862fab0, arg=0x8837194, kw=0x0)
 at ../Objects/abstract.c:2488
 #1  0x080d6ef2 in PyEval_CallObjectWithKeywords (func=0x862fab0,
 arg=0x8837194, kw=0x0) at ../Python/ceval.c:3575
 #2  0x080612a0 in PyObject_CallObject (o=0x862fab0, a=0x8837194)
 at ../Objects/abstract.c:2480
 #3  0x0047af42 in watcher_dispatch (zzh=0x86174e0, type=-1, state=1,
 path=0x86337c8 , context=0x8588660) at src/c/zookeeper.c:314
 #4  0x00496559 in do_foreach_watcher (zh=0x86174e0, type=-1, state=1,
 path=0x86337c8 , list=0xa5354140) at src/zk_hashtable.c:275
 #5  deliverWatchers (zh=0x86174e0, type=-1, state=1, path=0x86337c8 ,
 list=0xa5354140) at src/zk_hashtable.c:317
 #6  0x0048ae3c in process_completions (zh=0x86174e0) at src/zookeeper.c:1766
 #7  0x0049706b in do_completion (v=0x86174e0) at src/mt_adaptor.c:333
 #8  0x0013380e in start_thread () from /lib/tls/i686/cmov/libpthread.so.0
 #9  0x002578de in clone () from /lib/tls/i686/cmov/libc.so.6

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (ZOOKEEPER-756) some cleanup and improvements for zooinspector

[
https://issues.apache.org/jira/browse/ZOOKEEPER-756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Patrick Hunt updated ZOOKEEPER-756:
---

Status: Open (was: Patch Available)

The patch is failing to apply, please update against the latest source base and
resubmit, thanks!

some cleanup and improvements for zooinspector
--

Attachments: zooInspectorChanges.patch

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (ZOOKEEPER-906) Improve C client connection reliability by making it sleep between reconnect attempts as in Java Client

[
https://issues.apache.org/jira/browse/ZOOKEEPER-906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12930429#action_12930429
]

Patrick Hunt commented on ZOOKEEPER-906:

Hi Radu, yes I agree, difficult to parse in some cases. Nigel/Giri/buildteam
are working to improve, WIP.

bq. -1 tests included. The patch doesn't appear to include any new or modified
tests.

this is the main issue - the patch doesn't include any tests validating the
modified functionality.

bq. -1 on the core tests

You can't see it in the summary but if you look at the hudson raw console,
near the end, the c tests have failed. (d/l the console and open in editor)

This might be a false failure though, I saw similar on another test. I've
notified Nigel about it and he's going to take a look.

Improve C client connection reliability by making it sleep between reconnect
attempts as in Java Client
---

Attachments: ZOOKEEPER-906.patch

Original Estimate: 24h
Remaining Estimate: 24h

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (ZOOKEEPER-922) enable faster timeout of sessions in case of unexpected socket disconnect


 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patrick Hunt updated ZOOKEEPER-922:
---

Fix Version/s: 3.4.0
 Assignee: Camille Fournier
   Status: Patch Available  (was: Open)

Marking this as pa so we don't lose it, Camille is asking for f/b.

 enable faster timeout of sessions in case of unexpected socket disconnect
 -

 Key: ZOOKEEPER-922
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-922
 Project: Zookeeper
  Issue Type: Improvement
  Components: server
Reporter: Camille Fournier
Assignee: Camille Fournier
 Fix For: 3.4.0

 Attachments: ZOOKEEPER-922.patch


 In the case when a client connection is closed due to socket error instead of 
 the client calling close explicitly, it would be nice to enable the session 
 associated with that client to time out faster than the negotiated session 
 timeout. This would enable a zookeeper ensemble that is acting as a dynamic 
 discovery provider to remove ephemeral nodes for crashed clients quickly, 
 while allowing for a longer heartbeat-based timeout for java clients that 
 need to do long stop-the-world GC. 
 I propose doing this by setting the timeout associated with the crashed 
 session to minSessionTimeout.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (ZOOKEEPER-921) zkPython interferes with/corrupts Python's 'logging' module


 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patrick Hunt updated ZOOKEEPER-921:
---

Fix Version/s: 3.4.0
   3.3.3
 Assignee: Nicholas Knight

 zkPython interferes with/corrupts Python's 'logging' module
 ---

 Key: ZOOKEEPER-921
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-921
 Project: Zookeeper
  Issue Type: Bug
  Components: contrib-bindings
Affects Versions: 3.3.1, 3.4.0
 Environment: Mac OS X 10.6.4, included Python 2.6.1
Reporter: Nicholas Knight
Assignee: Nicholas Knight
 Fix For: 3.3.3, 3.4.0

 Attachments: zktest.py


 Calling {{zookeeper.create()}} seems, under certain circumstances, to be 
 corrupting a subsequent call to Python's {{logging}} module.
 Specifically, if the node does not exist (but its parent does), I end up with 
 a traceback like this when I try to make the logging call:
 {noformat}
 Traceback (most recent call last):
   File zktest.py, line 21, in module
 logger.error(Boom?)
   File 
 /System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/logging/__init__.py,
  line 1046, in error
 if self.isEnabledFor(ERROR):
   File 
 /System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/logging/__init__.py,
  line 1206, in isEnabledFor
 return level = self.getEffectiveLevel()
   File 
 /System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/logging/__init__.py,
  line 1194, in getEffectiveLevel
 while logger:
 TypeError: an integer is required
 {noformat}
 But if the node already exists, or the parent does not exist, I get the 
 appropriate NodeExists or NoNode exceptions.
 I'll be attaching a test script that can be used to reproduce this behavior.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (ZOOKEEPER-920) L7 (application layer) ping support

[
https://issues.apache.org/jira/browse/ZOOKEEPER-920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Patrick Hunt updated ZOOKEEPER-920:
---

Fix Version/s: 3.4.0
Assignee: Chang Song

Hi Chang Song, you need to provide this as a patch file in order for us to
consider, see this:
http://wiki.apache.org/hadoop/ZooKeeper/HowToContribute

basically do a svn diff at the toplevel and attach to this jira. Regards.

L7 (application layer) ping support
---

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Created: (ZOOKEEPER-925) Consider maven site generation to replace our forrest site and documentation generation

Consider maven site generation to replace our forrest site and documentation 
generation
---

 Key: ZOOKEEPER-925
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-925
 Project: Zookeeper
  Issue Type: Wish
  Components: documentation
Reporter: Patrick Hunt


See WHIRR-19 for some background.

In whirr we looked at a number of site/doc generation facilities. In the end 
Maven site generation plugin turned out to be by far the best option. You can 
see our nascent site here (no attempt at styling,etc so far):
http://incubator.apache.org/whirr/

In particular take a look at the quick start:
http://incubator.apache.org/whirr/quick-start-guide.html
which was generated from
http://svn.apache.org/repos/asf/incubator/whirr/trunk/src/site/confluence/quick-start-guide.confluence
notice this was standard wiki markup (confluence wiki markup, same as available 
from apache)

You can read more about mvn site plugin here:
http://maven.apache.org/guides/mini/guide-site.html
Notice that other formats are available, not just confluence markup, also note 
that you can use different markup formats if you like in the same site 
(although probably not a great idea, but in some cases might be handy, for 
example whirr uses the confluence wiki, so we can pretty much copy/paste source 
docs from wiki to our site (svn) if we like)


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (ZOOKEEPER-925) Consider maven site generation to replace our forrest site and documentation generation


 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patrick Hunt updated ZOOKEEPER-925:
---

Description: 
See WHIRR-19 for some background.

In whirr we looked at a number of site/doc generation facilities. In the end 
Maven site generation plugin turned out to be by far the best option. You can 
see our nascent site here (no attempt at styling,etc so far):
http://incubator.apache.org/whirr/

In particular take a look at the quick start:
http://incubator.apache.org/whirr/quick-start-guide.html
which was generated from
http://svn.apache.org/repos/asf/incubator/whirr/trunk/src/site/confluence/quick-start-guide.confluence
notice this was standard wiki markup (confluence wiki markup, same as available 
from apache)

You can read more about mvn site plugin here:
http://maven.apache.org/guides/mini/guide-site.html
Notice that other formats are available, not just confluence markup, also note 
that you can use different markup formats if you like in the same site 
(although probably not a great idea, but in some cases might be handy, for 
example whirr uses the confluence wiki, so we can pretty much copy/paste source 
docs from wiki to our site (svn) if we like)


Re maven vs our current ant based build. It's probably a good idea for us to 
move the build to maven at some point. We could initially move just the doc 
generation, and then incrementally move functionality from build.xml to mvn 
over a longer time period.


  was:
See WHIRR-19 for some background.

In whirr we looked at a number of site/doc generation facilities. In the end 
Maven site generation plugin turned out to be by far the best option. You can 
see our nascent site here (no attempt at styling,etc so far):
http://incubator.apache.org/whirr/

In particular take a look at the quick start:
http://incubator.apache.org/whirr/quick-start-guide.html
which was generated from
http://svn.apache.org/repos/asf/incubator/whirr/trunk/src/site/confluence/quick-start-guide.confluence
notice this was standard wiki markup (confluence wiki markup, same as available 
from apache)

You can read more about mvn site plugin here:
http://maven.apache.org/guides/mini/guide-site.html
Notice that other formats are available, not just confluence markup, also note 
that you can use different markup formats if you like in the same site 
(although probably not a great idea, but in some cases might be handy, for 
example whirr uses the confluence wiki, so we can pretty much copy/paste source 
docs from wiki to our site (svn) if we like)



 Consider maven site generation to replace our forrest site and documentation 
 generation
 ---

 Key: ZOOKEEPER-925
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-925
 Project: Zookeeper
  Issue Type: Wish
  Components: documentation
Reporter: Patrick Hunt

 See WHIRR-19 for some background.
 In whirr we looked at a number of site/doc generation facilities. In the end 
 Maven site generation plugin turned out to be by far the best option. You can 
 see our nascent site here (no attempt at styling,etc so far):
 http://incubator.apache.org/whirr/
 In particular take a look at the quick start:
 http://incubator.apache.org/whirr/quick-start-guide.html
 which was generated from
 http://svn.apache.org/repos/asf/incubator/whirr/trunk/src/site/confluence/quick-start-guide.confluence
 notice this was standard wiki markup (confluence wiki markup, same as 
 available from apache)
 You can read more about mvn site plugin here:
 http://maven.apache.org/guides/mini/guide-site.html
 Notice that other formats are available, not just confluence markup, also 
 note that you can use different markup formats if you like in the same site 
 (although probably not a great idea, but in some cases might be handy, for 
 example whirr uses the confluence wiki, so we can pretty much copy/paste 
 source docs from wiki to our site (svn) if we like)
 Re maven vs our current ant based build. It's probably a good idea for us to 
 move the build to maven at some point. We could initially move just the doc 
 generation, and then incrementally move functionality from build.xml to mvn 
 over a longer time period.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Replace forrest with maven site generation?

2010-11-08 Thread Patrick Hunt

Ben pinged me about what might we do about the forrest situation,
here's one idea:

https://issues.apache.org/jira/browse/ZOOKEEPER-925

Patrick

[VOTE] Release ZooKeeper 3.3.2 (candidate 0)

2010-11-05 Thread Patrick Hunt

I've created a candidate build for ZooKeeper 3.3.2. This is a bug fix
release addressing twenty-six issues (eight critical) -- see the
release notes for details.

*** Please download, test and VOTE before the
*** vote closes 11pm pacific time, Tuesday, November 9.***

http://people.apache.org/~phunt/zookeeper-3.3.2-candidate-0/

Should we release this?

Patrick

[jira] Commented: (ZOOKEEPER-918) Review of BookKeeper Documentation (Sequence flow and failure scenarios)

2010-11-05 Thread Patrick Hunt (JIRA)


[ 
https://issues.apache.org/jira/browse/ZOOKEEPER-918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12928697#action_12928697
 ] 

Patrick Hunt commented on ZOOKEEPER-918:


There are really two options for docs (today):

1) put it into svn as a forrest doc. typically this is for documentation that's 
version specific - needs to be versioned along with the code
2) put it into wiki, usually this is non-version specific detail.

putting into svn requires a patch for each change, which adds to the overhead. 
another way to go is to start on the wiki, once the doc is fairly stable move 
it to svn.



 Review of BookKeeper Documentation (Sequence flow and failure scenarios)
 

 Key: ZOOKEEPER-918
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-918
 Project: Zookeeper
  Issue Type: Task
  Components: documentation
Reporter: Amit Jaiswal
Priority: Trivial
 Fix For: 3.3.3, 3.4.0

 Attachments: BookKeeperInternals.doc, BookKeeperInternals.pdf

   Original Estimate: 2h
  Remaining Estimate: 2h

 I have prepared a document describing some of the internals of bookkeeper in 
 terms of:
 1. Sequence of operations
 2. Files layout
 3. Failure scenarios
 The document is prepared by mostly by reading the code. Can somebody who 
 understands the design review the same.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Updated: (ZOOKEEPER-918) Review of BookKeeper Documentation (Sequence flow and failure scenarios)

2010-11-05 Thread Patrick Hunt (JIRA)


 [ 
https://issues.apache.org/jira/browse/ZOOKEEPER-918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Patrick Hunt updated ZOOKEEPER-918:
---

Assignee: Amit Jaiswal

 Review of BookKeeper Documentation (Sequence flow and failure scenarios)
 

 Key: ZOOKEEPER-918
 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-918
 Project: Zookeeper
  Issue Type: Task
  Components: documentation
Reporter: Amit Jaiswal
Assignee: Amit Jaiswal
Priority: Trivial
 Fix For: 3.3.3, 3.4.0

 Attachments: BookKeeperInternals.doc, BookKeeperInternals.pdf

   Original Estimate: 2h
  Remaining Estimate: 2h

 I have prepared a document describing some of the internals of bookkeeper in 
 terms of:
 1. Sequence of operations
 2. Files layout
 3. Failure scenarios
 The document is prepared by mostly by reading the code. Can somebody who 
 understands the design review the same.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Re: Windows port of ZK C api

2010-11-04 Thread Patrick Hunt

If possible please update the poweredby page with some detail:
http://wiki.apache.org/hadoop/ZooKeeper/PoweredBy
useful for others to see how zk is being used.

Regards,

Patrick

On Thu, Nov 4, 2010 at 12:57 AM, jriepsh...@ujam.com
jriepsh...@ujam.com wrote:
 Regarding cygwin (just to let u guys know):

 We've built our zookeeper environment 2 month ago and one of the different 
 server types is (unfortunately) still running on Windows (Server 2008 64 bit) 
 only.
 Zookeeper + cygwin works perfectly for us.


 Jan




 On Nov 3, 2010, at 5:01 PM, Patrick Hunt wrote:

 Notice that buildbot now has windows 7 support:
 http://ci.apache.org/buildbot.html
 also I notice that hudson has windows 2008 server
 https://hudson.apache.org/hudson/computer/windows1/

 We should see if we can get ZK up on these to help with compatibility
 testing. Would be nice to catch cygwin as well - note that we support
 unix/cygwin today (although it's been a while since anyone tried
 cygwin afaik).

 Any interest in adding support for cmake? (the benefit of cmake is
 that it can generate native build environment, VS project for windows,
 makefiles for unix, etc...)

 Patrick

 On Wed, Nov 3, 2010 at 8:46 AM, Fournier, Camille F. [Tech]
 camille.fourn...@gs.com wrote:
 Thanks Mahadev. We are using those C# bindings but also need native windows 
 C/C++. Every language all the time!

 C

 -Original Message-
 From: Mahadev Konar [mailto:maha...@yahoo-inc.com]
 Sent: Wednesday, November 03, 2010 11:06 AM
 To: zookeeper-dev@hadoop.apache.org
 Subject: Re: Windows port of ZK C api

 Hi Camille,
  I think definitely there is. I think a build script with a set of
 requirements and a nice set of docs on how to start using it would be great.
 BTW, there is a C# binding which someone wrote earlier

 http://wiki.apache.org/hadoop/ZooKeeper/ZKClientBindings

 You can take a look at that and see if you want to extend that or write your
 own.

 Thanks
 mahadev


 On 11/3/10 7:18 AM, Fournier, Camille F. [Tech] camille.fourn...@gs.com
 wrote:

 Hi everyone,

 We have a requirement for a native windows-compatible version of the ZK C 
 api.
 We're currently working on various ways to do this port, but would very 
 much
 like to submit this back to you all when we are finished so that we don't 
 have
 to maintain the code ourselves through future releases. Is there interest 
 in
 having this? What would you need with this patch (build scripts, etc) to
 accept it?

 Thanks,
 Camille

[jira] Commented: (ZOOKEEPER-917) Leader election selected incorrect leader

[
https://issues.apache.org/jira/browse/ZOOKEEPER-917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12928240#action_12928240
]

Patrick Hunt commented on ZOOKEEPER-917:

Sounds like we need more documentation detailing the election process and what
expected behavior is. Flavio perhaps you could
create a JIRA for that and start collecting this type of information? In
particular you could link to jiras of this type, with the intent
of general documentation, including detail about these specific types of
questions.

Leader election selected incorrect leader
-

Key: ZOOKEEPER-917
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-917
Project: Zookeeper
Issue Type: Bug
Components: leaderElection, server
Affects Versions: 3.2.2
Environment: Cloudera distribution of zookeeper (patched to never
cache DNS entries)
Debian lenny
Reporter: Alexandre Hardy
Priority: Critical
Fix For: 3.3.3, 3.4.0

Attachments: zklogs-20101102144159SAST.tar.gz

We had three nodes running zookeeper:
* 192.168.130.10
* 192.168.130.11
* 192.168.130.14
192.168.130.11 failed, and was replaced by a new node 192.168.130.13
(automated startup). The new node had not participated in any zookeeper
quorum previously. The node 192.148.130.11 was permanently removed from
service and could not contribute to the quorum any further (powered off).
DNS entries were updated for the new node to allow all the zookeeper servers
to find the new node.
The new node 192.168.130.13 was selected as the LEADER, despite the fact that
it had not seen the latest zxid.
This particular problem has not been verified with later versions of
zookeeper, and no attempt has been made to reproduce this problem as yet.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

Update on ZooKeeper 3.3.2 release

2010-11-04 Thread Patrick Hunt

I'm starting work on creating a release candidate for ZooKeeper 3.3.2

26 issues are addressed in this release http://bit.ly/9HP5J3 8 of
which were considered blockers.

Patrick

[jira] Commented: (ZOOKEEPER-912) ZooKeeper client logs trace and debug messages at level INFO

[
https://issues.apache.org/jira/browse/ZOOKEEPER-912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12928430#action_12928430
]

Patrick Hunt commented on ZOOKEEPER-912:

Hi John, thanks for the feedback

bq. I've actually had to repackage the zk jar without log4j.xml to tone down
the logging.

well we don't have a log4j.xml at all, we do have a log4j.properties but that's
not included in our primary or maven bin jar files. I just looked again and I
don't see it included (at least in 3.3.1) Perhaps this is being pulled in from
elsewhere in your environment?

We do provide a default log4j.properties in the conf directory of the release
artifact. However the intent is for users to either use that directly or
customize it based on their needs, which is why it's a config file and not
hardcoded.

bq. If I could weigh in on this one too... logging levels

That's basically what we do. I grant that we skew to pushing more detail at
INFO that might be DEBUG, and some DEBUG that should be TRACE. We have been
working on that of late (see JIRA, a number of logging level changes went into
3.3). We could push a few more of the info messages to debug, but I'm reticent
for a few reasons. Primarily the fact that we often get reports from users who
see issues where the only detail we have to go on (esp given this is a complex
distributed system) is the info (and higher) level logs. My experience
supporting a number of production teams (15+) tells me this would compromise
our ability to help them resolve issues.

ZooKeeper client logs trace and debug messages at level INFO

Key: ZOOKEEPER-912
URL: https://issues.apache.org/jira/browse/ZOOKEEPER-912
Project: Zookeeper
Issue Type: Improvement
Components: java client
Affects Versions: 3.3.1
Reporter: Anthony Urso
Assignee: Anthony Urso
Priority: Minor
Fix For: 3.4.0

Attachments: zk-loglevel.patch

ZK logs a lot of uninformative trace and debug messages to level INFO. This
fuzzes up everything and makes it easy to miss useful log info.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (ZOOKEEPER-912) ZooKeeper client logs trace and debug messages at level INFO

[
https://issues.apache.org/jira/browse/ZOOKEEPER-912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12928436#action_12928436
]

Patrick Hunt commented on ZOOKEEPER-912:

John could you give some examples of the log messages (ones actually output to
the log) that you thought were excessive? (did you mean client or server?
both?) It might help to frame the conversation. We might be able to address
some of the more egregious ones.

Here's a client session (default log4j.properties) where I created a client,
ran a few commands, then quit (cli shell):

2010-11-04 17:16:21,319 - INFO [main:zookee...@373] - Initiating client
connection, connectString= sessionTimeout=3
watcher=org.apache.zookeeper.zookeepermain$mywatc...@2c6f7ce9
2010-11-04 17:16:21,347 - INFO [main-SendThread():clientcnxn$sendthr...@1000]
- Opening socket connection to server localhost/127.0.0.1:2181
2010-11-04 17:16:21,392 - INFO
[main-SendThread(localhost:2181):clientcnxn$sendthr...@908] - Socket connection
established to localhost/127.0.0.1:2181, initiating session
2010-11-04 17:16:21,486 - INFO
[main-SendThread(localhost:2181):clientcnxn$sendthr...@701] - Session
establishment complete on server localhost/127.0.0.1:2181, sessionid =
0x12c1963f821, negotiated timeout = 3

then nothing until I quit

2010-11-04 17:16:49,401 - INFO [main:zookee...@538] - Session:
0x12c1963f821 closed

So the _entirety_ of the logging today is just 4 messages for the client
establishment, one for closing the client. During the time that the client has
an active session established there are no messages output. I don't see that as
excessive personally, but others might not think the same. In my experience
these are some very important messages when helping postmortem production
failures.

I could see where we might drop msgs 1-3 to debug level (keep 4, the
established msg), as long as we highlight connection attempts that fail.
Having detail about the sessionid and negotiated timeout is pretty critical
though, from an informational perspective.

ZooKeeper client logs trace and debug messages at level INFO

Attachments: zk-loglevel.patch

ZK logs a lot of uninformative trace and debug messages to level INFO. This
fuzzes up everything and makes it easy to miss useful log info.

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

[jira] Commented: (ZOOKEEPER-914) QuorumCnxManager blocks forever