[jira] Updated: (ZOOKEEPER-930) Hedwig c++ client uses a non thread safe logging library
[ https://issues.apache.org/jira/browse/ZOOKEEPER-930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Kelly updated ZOOKEEPER-930: - Status: Open (was: Patch Available) Hedwig c++ client uses a non thread safe logging library Key: ZOOKEEPER-930 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-930 Project: Zookeeper Issue Type: Bug Components: contrib-hedwig Affects Versions: 3.3.2 Reporter: Ivan Kelly Assignee: Ivan Kelly Attachments: ZOOKEEPER-930.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-930) Hedwig c++ client uses a non thread safe logging library
[ https://issues.apache.org/jira/browse/ZOOKEEPER-930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Kelly updated ZOOKEEPER-930: - Attachment: ZOOKEEPER-930.patch Hedwig c++ client uses a non thread safe logging library Key: ZOOKEEPER-930 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-930 Project: Zookeeper Issue Type: Bug Components: contrib-hedwig Affects Versions: 3.3.2 Reporter: Ivan Kelly Assignee: Ivan Kelly Attachments: ZOOKEEPER-930.patch, ZOOKEEPER-930.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-930) Hedwig c++ client uses a non thread safe logging library
[ https://issues.apache.org/jira/browse/ZOOKEEPER-930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12932379#action_12932379 ] Ivan Kelly commented on ZOOKEEPER-930: -- As the title says, log4cpp is not thread safe and not really maintained afaics. Moving to log4cxx, which is an apache project. Uploaded a new diff with indentation fixes. Hedwig c++ client uses a non thread safe logging library Key: ZOOKEEPER-930 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-930 Project: Zookeeper Issue Type: Bug Components: contrib-hedwig Affects Versions: 3.3.2 Reporter: Ivan Kelly Assignee: Ivan Kelly Attachments: ZOOKEEPER-930.patch, ZOOKEEPER-930.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-930) Hedwig c++ client uses a non thread safe logging library
[ https://issues.apache.org/jira/browse/ZOOKEEPER-930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Kelly updated ZOOKEEPER-930: - Status: Patch Available (was: Open) Hedwig c++ client uses a non thread safe logging library Key: ZOOKEEPER-930 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-930 Project: Zookeeper Issue Type: Bug Components: contrib-hedwig Affects Versions: 3.3.2 Reporter: Ivan Kelly Assignee: Ivan Kelly Attachments: ZOOKEEPER-930.patch, ZOOKEEPER-930.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-930) Hedwig c++ client uses a non thread safe logging library
[ https://issues.apache.org/jira/browse/ZOOKEEPER-930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12932387#action_12932387 ] Hadoop QA commented on ZOOKEEPER-930: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12459691/ZOOKEEPER-930.patch against trunk revision 1034003. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 24 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://hudson.apache.org/hudson/job/PreCommit-ZOOKEEPER-Build/30//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-ZOOKEEPER-Build/30//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-ZOOKEEPER-Build/30//console This message is automatically generated. Hedwig c++ client uses a non thread safe logging library Key: ZOOKEEPER-930 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-930 Project: Zookeeper Issue Type: Bug Components: contrib-hedwig Affects Versions: 3.3.2 Reporter: Ivan Kelly Assignee: Ivan Kelly Attachments: ZOOKEEPER-930.patch, ZOOKEEPER-930.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-905) enhance zkServer.sh for easier zookeeper automation-izing
[ https://issues.apache.org/jira/browse/ZOOKEEPER-905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12932498#action_12932498 ] Mahadev konar commented on ZOOKEEPER-905: - The patch looks good and applies to trunk. +1 to the patch, I am marking this as PA. enhance zkServer.sh for easier zookeeper automation-izing - Key: ZOOKEEPER-905 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-905 Project: Zookeeper Issue Type: Improvement Components: scripts Reporter: Nicholas Harteau Assignee: Nicholas Harteau Priority: Minor Fix For: 3.4.0 Attachments: zkServer.sh.diff zkServer.sh is good at starting zookeeper and figuring out the right options to pass along. unfortunately if you want to wrap zookeeper startup/shutdown in any significant way, you have to reimplement a bunch of the logic there. the attached patch addresses a couple simple issues: 1. add a 'start-foreground' option to zkServer.sh - this allows things that expect to manage a foregrounded process (daemontools, launchd, etc) to use zkServer.sh instead of rolling their own to launch zookeeper 2. add a 'print-cmd' option to zkServer.sh - rather than launching zookeeper from the script, just give me the command you'd normally use to exec zookeeper. I found this useful when writing automation to start/stop zookeeper as part of smoke testing zookeeper-based applications 3. Deal more gracefully with supplying alternate configuration files to zookeeper - currently the script assumes all config files reside in $ZOOCFGDIR - also useful for smoke testing 4. communicate extra info (JMX enabled) about zookeeper on STDERR rather than STDOUT (necessary for #2) 5. fixes an issue on macos where readlink doesn't have the '-f' option. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-922) enable faster timeout of sessions in case of unexpected socket disconnect
[ https://issues.apache.org/jira/browse/ZOOKEEPER-922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12932500#action_12932500 ] Flavio Junqueira commented on ZOOKEEPER-922: Hi! I'm confused by this proposal. What happens if the client disconnects form one server and moves to another? Or you want to be able to disable that feature as well? enable faster timeout of sessions in case of unexpected socket disconnect - Key: ZOOKEEPER-922 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-922 Project: Zookeeper Issue Type: Improvement Components: server Reporter: Camille Fournier Assignee: Camille Fournier Fix For: 3.4.0 Attachments: ZOOKEEPER-922.patch In the case when a client connection is closed due to socket error instead of the client calling close explicitly, it would be nice to enable the session associated with that client to time out faster than the negotiated session timeout. This would enable a zookeeper ensemble that is acting as a dynamic discovery provider to remove ephemeral nodes for crashed clients quickly, while allowing for a longer heartbeat-based timeout for java clients that need to do long stop-the-world GC. I propose doing this by setting the timeout associated with the crashed session to minSessionTimeout. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-905) enhance zkServer.sh for easier zookeeper automation-izing
[ https://issues.apache.org/jira/browse/ZOOKEEPER-905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated ZOOKEEPER-905: Status: Patch Available (was: Open) enhance zkServer.sh for easier zookeeper automation-izing - Key: ZOOKEEPER-905 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-905 Project: Zookeeper Issue Type: Improvement Components: scripts Reporter: Nicholas Harteau Assignee: Nicholas Harteau Priority: Minor Fix For: 3.4.0 Attachments: zkServer.sh.diff zkServer.sh is good at starting zookeeper and figuring out the right options to pass along. unfortunately if you want to wrap zookeeper startup/shutdown in any significant way, you have to reimplement a bunch of the logic there. the attached patch addresses a couple simple issues: 1. add a 'start-foreground' option to zkServer.sh - this allows things that expect to manage a foregrounded process (daemontools, launchd, etc) to use zkServer.sh instead of rolling their own to launch zookeeper 2. add a 'print-cmd' option to zkServer.sh - rather than launching zookeeper from the script, just give me the command you'd normally use to exec zookeeper. I found this useful when writing automation to start/stop zookeeper as part of smoke testing zookeeper-based applications 3. Deal more gracefully with supplying alternate configuration files to zookeeper - currently the script assumes all config files reside in $ZOOCFGDIR - also useful for smoke testing 4. communicate extra info (JMX enabled) about zookeeper on STDERR rather than STDOUT (necessary for #2) 5. fixes an issue on macos where readlink doesn't have the '-f' option. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-922) enable faster timeout of sessions in case of unexpected socket disconnect
[ https://issues.apache.org/jira/browse/ZOOKEEPER-922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12932502#action_12932502 ] Camille Fournier commented on ZOOKEEPER-922: If the client connects to another server within whatever the time they set the minSessionTimeout to, the client should heartbeat the session and it shouldn't get timed out. Otherwise, their session will be expired and they'll get the standard expired session scenario. If you are working in a setup where you think that unexpected disconnects will largely be due to clients crashing and you want ephemeral data aggressively removed in that scenario, with this design you set the minSessionTimeout to a low value and allow the ZK to quickly timeout those sessions. If you are working in a setup where unexpected disconnects are more likely to be due to network problems, or you want to give data a longer time to survive, you have the option of setting the timeout to a higher value (ideally the same as the negotiated session timeout, but that might require a code change to match negotiation), which should give the same behavior as now. enable faster timeout of sessions in case of unexpected socket disconnect - Key: ZOOKEEPER-922 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-922 Project: Zookeeper Issue Type: Improvement Components: server Reporter: Camille Fournier Assignee: Camille Fournier Fix For: 3.4.0 Attachments: ZOOKEEPER-922.patch In the case when a client connection is closed due to socket error instead of the client calling close explicitly, it would be nice to enable the session associated with that client to time out faster than the negotiated session timeout. This would enable a zookeeper ensemble that is acting as a dynamic discovery provider to remove ephemeral nodes for crashed clients quickly, while allowing for a longer heartbeat-based timeout for java clients that need to do long stop-the-world GC. I propose doing this by setting the timeout associated with the crashed session to minSessionTimeout. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-756) some cleanup and improvements for zooinspector
[ https://issues.apache.org/jira/browse/ZOOKEEPER-756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12932503#action_12932503 ] Mahadev konar commented on ZOOKEEPER-756: - @colin, Any updates on the patch? As patrick mentioned above, the patch doesnt apply to the trunk. some cleanup and improvements for zooinspector -- Key: ZOOKEEPER-756 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-756 Project: Zookeeper Issue Type: Improvement Components: contrib Affects Versions: 3.3.0 Reporter: Thomas Koch Assignee: Colin Goodheart-Smithe Fix For: 3.4.0 Attachments: zooInspectorChanges.patch Copied from the already closed ZOOKEEPER-678: * specify the exact URL, where the icons are from. It's best to include the link also in the NOTICE.txt file. It seems, that zooinspector finds it's icons only if the icons folder is in the current path. But when I install zooinspector as part of the Zookeeper Debian package, I want to be able to call it regardless of the current path. Could you use getRessources or something so that I can point to the icons location from the wrapper shell script? Can I place the zooinspector config files in /etc/zookeeper/zooinspector/ ? Could I give zooinspector a property to point to the config file location? There are several places, where viewers is missspelled as Veiwers. Please do a case insensitive search for veiw to correct these. Even the config file defaultNodeVeiwers.cfg is missspelled like this. This has the potential to confuse the hell out of people when debugging something! -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-922) enable faster timeout of sessions in case of unexpected socket disconnect
[ https://issues.apache.org/jira/browse/ZOOKEEPER-922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12932512#action_12932512 ] Flavio Junqueira commented on ZOOKEEPER-922: I think I understand your motivation, but I'm not sure it will work the way you expect it to work. I'm afraid that you might end end up getting lots of false positives due to delays introduced by the environment (e.g., jvm gc). Let me clarify one thing first: when you refer to clients crashing, are you thinking about the jvm crashing or the whole machine becoming unavailable? Basically my question is if you really expect connections to be cleanly closed or not. enable faster timeout of sessions in case of unexpected socket disconnect - Key: ZOOKEEPER-922 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-922 Project: Zookeeper Issue Type: Improvement Components: server Reporter: Camille Fournier Assignee: Camille Fournier Fix For: 3.4.0 Attachments: ZOOKEEPER-922.patch In the case when a client connection is closed due to socket error instead of the client calling close explicitly, it would be nice to enable the session associated with that client to time out faster than the negotiated session timeout. This would enable a zookeeper ensemble that is acting as a dynamic discovery provider to remove ephemeral nodes for crashed clients quickly, while allowing for a longer heartbeat-based timeout for java clients that need to do long stop-the-world GC. I propose doing this by setting the timeout associated with the crashed session to minSessionTimeout. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-905) enhance zkServer.sh for easier zookeeper automation-izing
[ https://issues.apache.org/jira/browse/ZOOKEEPER-905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12932514#action_12932514 ] Hadoop QA commented on ZOOKEEPER-905: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12459259/zkServer.sh.diff against trunk revision 1034003. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://hudson.apache.org/hudson/job/PreCommit-ZOOKEEPER-Build/31//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-ZOOKEEPER-Build/31//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-ZOOKEEPER-Build/31//console This message is automatically generated. enhance zkServer.sh for easier zookeeper automation-izing - Key: ZOOKEEPER-905 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-905 Project: Zookeeper Issue Type: Improvement Components: scripts Reporter: Nicholas Harteau Assignee: Nicholas Harteau Priority: Minor Fix For: 3.4.0 Attachments: zkServer.sh.diff zkServer.sh is good at starting zookeeper and figuring out the right options to pass along. unfortunately if you want to wrap zookeeper startup/shutdown in any significant way, you have to reimplement a bunch of the logic there. the attached patch addresses a couple simple issues: 1. add a 'start-foreground' option to zkServer.sh - this allows things that expect to manage a foregrounded process (daemontools, launchd, etc) to use zkServer.sh instead of rolling their own to launch zookeeper 2. add a 'print-cmd' option to zkServer.sh - rather than launching zookeeper from the script, just give me the command you'd normally use to exec zookeeper. I found this useful when writing automation to start/stop zookeeper as part of smoke testing zookeeper-based applications 3. Deal more gracefully with supplying alternate configuration files to zookeeper - currently the script assumes all config files reside in $ZOOCFGDIR - also useful for smoke testing 4. communicate extra info (JMX enabled) about zookeeper on STDERR rather than STDOUT (necessary for #2) 5. fixes an issue on macos where readlink doesn't have the '-f' option. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-922) enable faster timeout of sessions in case of unexpected socket disconnect
[ https://issues.apache.org/jira/browse/ZOOKEEPER-922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12932530#action_12932530 ] Camille Fournier commented on ZOOKEEPER-922: This only changes the timeout on case of a detected exception that closes the socket (see the patch). The purpose in fact is to enable environments with machines that may have long GC pause times to have long max session timeouts, while still clearing the ephemeral nodes of crashed clients more quickly. The only crash I am intending to deal with here is the crash that causes an exception closing the socket on the server side. This can also be caused by a switch failure, but in my environment it is much much much much more likely to be caused by the client process crashing. I don't expect to be able to perfectly deal with all cases of client crash, because there are some that don't cause the socket to close and that can't be differentiated from a client doing a long full GC. enable faster timeout of sessions in case of unexpected socket disconnect - Key: ZOOKEEPER-922 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-922 Project: Zookeeper Issue Type: Improvement Components: server Reporter: Camille Fournier Assignee: Camille Fournier Fix For: 3.4.0 Attachments: ZOOKEEPER-922.patch In the case when a client connection is closed due to socket error instead of the client calling close explicitly, it would be nice to enable the session associated with that client to time out faster than the negotiated session timeout. This would enable a zookeeper ensemble that is acting as a dynamic discovery provider to remove ephemeral nodes for crashed clients quickly, while allowing for a longer heartbeat-based timeout for java clients that need to do long stop-the-world GC. I propose doing this by setting the timeout associated with the crashed session to minSessionTimeout. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-900) FLE implementation should be improved to use non-blocking sockets
[ https://issues.apache.org/jira/browse/ZOOKEEPER-900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vishal K updated ZOOKEEPER-900: --- Attachment: ZOOKEEPER-900.patch Hi Flavio, Pat, I have attached the patch with suggested modifications. I did some more testing and also ran SimpleSysTest. Other than the changes suggested above and some cosmetic changes to error messages, I have added a delay on 1 second in Listener.run() in case we see an IOException. Let me know if you have more comments. -Vishal FLE implementation should be improved to use non-blocking sockets - Key: ZOOKEEPER-900 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-900 Project: Zookeeper Issue Type: Bug Reporter: Vishal K Assignee: Vishal K Priority: Critical Fix For: 3.4.0 Attachments: ZOOKEEPER-900.patch, ZOOKEEPER-900.patch1, ZOOKEEPER-900.patch2 From earlier email exchanges: 1. Blocking connects and accepts: a) The first problem is in manager.toSend(). This invokes connectOne(), which does a blocking connect. While testing, I changed the code so that connectOne() starts a new thread called AsyncConnct(). AsyncConnect.run() does a socketChannel.connect(). After starting AsyncConnect, connectOne starts a timer. connectOne continues with normal operations if the connection is established before the timer expires, otherwise, when the timer expires it interrupts AsyncConnect() thread and returns. In this way, I can have an upper bound on the amount of time we need to wait for connect to succeed. Of course, this was a quick fix for my testing. Ideally, we should use Selector to do non-blocking connects/accepts. I am planning to do that later once we at least have a quick fix for the problem and consensus from others for the real fix (this problem is big blocker for us). Note that it is OK to do blocking IO in SenderWorker and RecvWorker threads since they block IO to the respective ! peer. b) The blocking IO problem is not just restricted to connectOne(), but also in receiveConnection(). The Listener thread calls receiveConnection() for each incoming connection request. receiveConnection does blocking IO to get peer's info (s.read(msgBuffer)). Worse, it invokes connectOne() back to the peer that had sent the connection request. All of this is happening from the Listener. In short, if a peer fails after initiating a connection, the Listener thread won't be able to accept connections from other peers, because it would be stuck in read() or connetOne(). Also the code has an inherent cycle. initiateConnection() and receiveConnection() will have to be very carefully synchronized otherwise, we could run into deadlocks. This code is going to be difficult to maintain/modify. Also see: https://issues.apache.org/jira/browse/ZOOKEEPER-822 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-900) FLE implementation should be improved to use non-blocking sockets
[ https://issues.apache.org/jira/browse/ZOOKEEPER-900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vishal K updated ZOOKEEPER-900: --- Status: Patch Available (was: Open) patch submitted. FLE implementation should be improved to use non-blocking sockets - Key: ZOOKEEPER-900 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-900 Project: Zookeeper Issue Type: Bug Reporter: Vishal K Assignee: Vishal K Priority: Critical Fix For: 3.4.0 Attachments: ZOOKEEPER-900.patch, ZOOKEEPER-900.patch1, ZOOKEEPER-900.patch2 From earlier email exchanges: 1. Blocking connects and accepts: a) The first problem is in manager.toSend(). This invokes connectOne(), which does a blocking connect. While testing, I changed the code so that connectOne() starts a new thread called AsyncConnct(). AsyncConnect.run() does a socketChannel.connect(). After starting AsyncConnect, connectOne starts a timer. connectOne continues with normal operations if the connection is established before the timer expires, otherwise, when the timer expires it interrupts AsyncConnect() thread and returns. In this way, I can have an upper bound on the amount of time we need to wait for connect to succeed. Of course, this was a quick fix for my testing. Ideally, we should use Selector to do non-blocking connects/accepts. I am planning to do that later once we at least have a quick fix for the problem and consensus from others for the real fix (this problem is big blocker for us). Note that it is OK to do blocking IO in SenderWorker and RecvWorker threads since they block IO to the respective ! peer. b) The blocking IO problem is not just restricted to connectOne(), but also in receiveConnection(). The Listener thread calls receiveConnection() for each incoming connection request. receiveConnection does blocking IO to get peer's info (s.read(msgBuffer)). Worse, it invokes connectOne() back to the peer that had sent the connection request. All of this is happening from the Listener. In short, if a peer fails after initiating a connection, the Listener thread won't be able to accept connections from other peers, because it would be stuck in read() or connetOne(). Also the code has an inherent cycle. initiateConnection() and receiveConnection() will have to be very carefully synchronized otherwise, we could run into deadlocks. This code is going to be difficult to maintain/modify. Also see: https://issues.apache.org/jira/browse/ZOOKEEPER-822 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-930) Hedwig c++ client uses a non thread safe logging library
[ https://issues.apache.org/jira/browse/ZOOKEEPER-930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Reed updated ZOOKEEPER-930: Resolution: Fixed Hadoop Flags: [Reviewed] Status: Resolved (was: Patch Available) Committed revision 1035727. Hedwig c++ client uses a non thread safe logging library Key: ZOOKEEPER-930 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-930 Project: Zookeeper Issue Type: Bug Components: contrib-hedwig Affects Versions: 3.3.2 Reporter: Ivan Kelly Assignee: Ivan Kelly Attachments: ZOOKEEPER-930.patch, ZOOKEEPER-930.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-930) Hedwig c++ client uses a non thread safe logging library
[ https://issues.apache.org/jira/browse/ZOOKEEPER-930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12932583#action_12932583 ] Benjamin Reed commented on ZOOKEEPER-930: - thanx ivan! Hedwig c++ client uses a non thread safe logging library Key: ZOOKEEPER-930 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-930 Project: Zookeeper Issue Type: Bug Components: contrib-hedwig Affects Versions: 3.3.2 Reporter: Ivan Kelly Assignee: Ivan Kelly Attachments: ZOOKEEPER-930.patch, ZOOKEEPER-930.patch -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Created: (ZOOKEEPER-931) Documentation for Hedwig
Documentation for Hedwig Key: ZOOKEEPER-931 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-931 Project: Zookeeper Issue Type: Task Components: contrib-hedwig Reporter: Erwin Tam This is a tracking jira for providing documentation to Hedwig on the Apache site. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-925) Consider maven site generation to replace our forrest site and documentation generation
[ https://issues.apache.org/jira/browse/ZOOKEEPER-925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12932590#action_12932590 ] Patrick Hunt commented on ZOOKEEPER-925: At this point to move fwd we need to work on 2 main areas: 1) site gen 2) doc conversion item 1) is looking pretty good, but some work yet to be done, icons and look/feel mainly item 2) just requires us to decide which if any format we want to standardize on and then try moving some docs to that format. I would highly suggest that our standard/preferred format be confluence format -- we can move our wiki to there at some point, which will match up nicely. Consider maven site generation to replace our forrest site and documentation generation --- Key: ZOOKEEPER-925 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-925 Project: Zookeeper Issue Type: Wish Components: documentation Reporter: Patrick Hunt Assignee: Patrick Hunt Attachments: ZOOKEEPER-925.patch See WHIRR-19 for some background. In whirr we looked at a number of site/doc generation facilities. In the end Maven site generation plugin turned out to be by far the best option. You can see our nascent site here (no attempt at styling,etc so far): http://incubator.apache.org/whirr/ In particular take a look at the quick start: http://incubator.apache.org/whirr/quick-start-guide.html which was generated from http://svn.apache.org/repos/asf/incubator/whirr/trunk/src/site/confluence/quick-start-guide.confluence notice this was standard wiki markup (confluence wiki markup, same as available from apache) You can read more about mvn site plugin here: http://maven.apache.org/guides/mini/guide-site.html Notice that other formats are available, not just confluence markup, also note that you can use different markup formats if you like in the same site (although probably not a great idea, but in some cases might be handy, for example whirr uses the confluence wiki, so we can pretty much copy/paste source docs from wiki to our site (svn) if we like) Re maven vs our current ant based build. It's probably a good idea for us to move the build to maven at some point. We could initially move just the doc generation, and then incrementally move functionality from build.xml to mvn over a longer time period. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-925) Consider maven site generation to replace our forrest site and documentation generation
[ https://issues.apache.org/jira/browse/ZOOKEEPER-925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12932592#action_12932592 ] Benjamin Reed commented on ZOOKEEPER-925: - +1 for confluence it would be great to target 1) for when we move to tlp. Consider maven site generation to replace our forrest site and documentation generation --- Key: ZOOKEEPER-925 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-925 Project: Zookeeper Issue Type: Wish Components: documentation Reporter: Patrick Hunt Assignee: Patrick Hunt Attachments: ZOOKEEPER-925.patch See WHIRR-19 for some background. In whirr we looked at a number of site/doc generation facilities. In the end Maven site generation plugin turned out to be by far the best option. You can see our nascent site here (no attempt at styling,etc so far): http://incubator.apache.org/whirr/ In particular take a look at the quick start: http://incubator.apache.org/whirr/quick-start-guide.html which was generated from http://svn.apache.org/repos/asf/incubator/whirr/trunk/src/site/confluence/quick-start-guide.confluence notice this was standard wiki markup (confluence wiki markup, same as available from apache) You can read more about mvn site plugin here: http://maven.apache.org/guides/mini/guide-site.html Notice that other formats are available, not just confluence markup, also note that you can use different markup formats if you like in the same site (although probably not a great idea, but in some cases might be handy, for example whirr uses the confluence wiki, so we can pretty much copy/paste source docs from wiki to our site (svn) if we like) Re maven vs our current ant based build. It's probably a good idea for us to move the build to maven at some point. We could initially move just the doc generation, and then incrementally move functionality from build.xml to mvn over a longer time period. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-896) Improve C client to support dynamic authentication schemes
[ https://issues.apache.org/jira/browse/ZOOKEEPER-896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koontz updated ZOOKEEPER-896: Hi Botond, I am planning on developing a Java version of your patch; are you working on this? Improve C client to support dynamic authentication schemes -- Key: ZOOKEEPER-896 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-896 Project: Zookeeper Issue Type: Improvement Components: c client Affects Versions: 3.3.1 Reporter: Botond Hejj Assignee: Botond Hejj Fix For: 3.4.0 Attachments: ZOOKEEPER-896.patch When we started exploring zookeeper for our requirements we found the authentication mechanism is not flexible enough. We want to use kerberos for authentication but using the current API we ran into a few problems. The idea is that we get a kerberos token on the client side and than send that token to the server with a kerberos scheme. A server side authentication plugin can use that token to authenticate the client and also use the token for authorization. We ran into two problems with this approach: 1. A different kerberos token is needed for each different server that client can connect to since kerberos uses mutual authentication. That means when the client acquires this kerberos token it has to know which server it connects to and generate the token according to that. The client currently can't generate a token for a specific server. The token stored in the auth_info is used for all the servers. 2. The kerberos token might have an expiry time so if the client loses the connection to the server and than it tries to reconnect it should acquire a new token. That is not possible currently since the token is stored in auth_info and reused for every connection. The problem can be solved if we allow the client to register a callback for authentication instead a static token. This can be a callback with an argument which passes the current host string. The zookeeper client code could call this callback before it sends the authentication info to the server to get a fresh server specific token. This would solve our problem with the kerberos authentication and also could be used for other more dynamic authentication schemes. The solution could be generalization also for the java client as well. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-896) Improve C client to support dynamic authentication schemes
[ https://issues.apache.org/jira/browse/ZOOKEEPER-896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12932607#action_12932607 ] Patrick Hunt commented on ZOOKEEPER-896: The security docs (both client side and server (plugin arch is totally undoc'd)) is sorely in need of improvement. If you are in the area and would like to help out adding docs would be huge. Thanks. Improve C client to support dynamic authentication schemes -- Key: ZOOKEEPER-896 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-896 Project: Zookeeper Issue Type: Improvement Components: c client Affects Versions: 3.3.1 Reporter: Botond Hejj Assignee: Botond Hejj Fix For: 3.4.0 Attachments: ZOOKEEPER-896.patch When we started exploring zookeeper for our requirements we found the authentication mechanism is not flexible enough. We want to use kerberos for authentication but using the current API we ran into a few problems. The idea is that we get a kerberos token on the client side and than send that token to the server with a kerberos scheme. A server side authentication plugin can use that token to authenticate the client and also use the token for authorization. We ran into two problems with this approach: 1. A different kerberos token is needed for each different server that client can connect to since kerberos uses mutual authentication. That means when the client acquires this kerberos token it has to know which server it connects to and generate the token according to that. The client currently can't generate a token for a specific server. The token stored in the auth_info is used for all the servers. 2. The kerberos token might have an expiry time so if the client loses the connection to the server and than it tries to reconnect it should acquire a new token. That is not possible currently since the token is stored in auth_info and reused for every connection. The problem can be solved if we allow the client to register a callback for authentication instead a static token. This can be a callback with an argument which passes the current host string. The zookeeper client code could call this callback before it sends the authentication info to the server to get a fresh server specific token. This would solve our problem with the kerberos authentication and also could be used for other more dynamic authentication schemes. The solution could be generalization also for the java client as well. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-931) Documentation for Hedwig
[ https://issues.apache.org/jira/browse/ZOOKEEPER-931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mahadev konar updated ZOOKEEPER-931: Fix Version/s: 3.4.0 Documentation for Hedwig Key: ZOOKEEPER-931 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-931 Project: Zookeeper Issue Type: Task Components: contrib-hedwig Reporter: Erwin Tam Fix For: 3.4.0 This is a tracking jira for providing documentation to Hedwig on the Apache site. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-756) some cleanup and improvements for zooinspector
[ https://issues.apache.org/jira/browse/ZOOKEEPER-756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Goodheart-Smithe updated ZOOKEEPER-756: - Attachment: (was: zooInspectorChanges.patch) some cleanup and improvements for zooinspector -- Key: ZOOKEEPER-756 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-756 Project: Zookeeper Issue Type: Improvement Components: contrib Affects Versions: 3.3.0 Reporter: Thomas Koch Assignee: Colin Goodheart-Smithe Fix For: 3.4.0 Attachments: zooInspectorChanges.patch Copied from the already closed ZOOKEEPER-678: * specify the exact URL, where the icons are from. It's best to include the link also in the NOTICE.txt file. It seems, that zooinspector finds it's icons only if the icons folder is in the current path. But when I install zooinspector as part of the Zookeeper Debian package, I want to be able to call it regardless of the current path. Could you use getRessources or something so that I can point to the icons location from the wrapper shell script? Can I place the zooinspector config files in /etc/zookeeper/zooinspector/ ? Could I give zooinspector a property to point to the config file location? There are several places, where viewers is missspelled as Veiwers. Please do a case insensitive search for veiw to correct these. Even the config file defaultNodeVeiwers.cfg is missspelled like this. This has the potential to confuse the hell out of people when debugging something! -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-756) some cleanup and improvements for zooinspector
[ https://issues.apache.org/jira/browse/ZOOKEEPER-756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Goodheart-Smithe updated ZOOKEEPER-756: - Attachment: zooInspectorChanges.patch I have recreated the patch file. I have done this a few times now and I'm not sure what I'm doing wrong so I've explained what I'm doing with TortoiseSVN below, if any of this is wrong could you let me know so that I can recreate it correctly: # Navigated to the repository folder in Windows Explorer # Right-clicked the repository folder and selected TortoiseSVN -- Create Patch... # Selected the files I have changed in the dialog and clicked OK # selected a location and name for the patch file and clicked Save. The repository is set to check out from the following URL: http://svn.apache.org/repos/asf/hadoop/zookeeper/trunk some cleanup and improvements for zooinspector -- Key: ZOOKEEPER-756 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-756 Project: Zookeeper Issue Type: Improvement Components: contrib Affects Versions: 3.3.0 Reporter: Thomas Koch Assignee: Colin Goodheart-Smithe Fix For: 3.4.0 Attachments: zooInspectorChanges.patch Copied from the already closed ZOOKEEPER-678: * specify the exact URL, where the icons are from. It's best to include the link also in the NOTICE.txt file. It seems, that zooinspector finds it's icons only if the icons folder is in the current path. But when I install zooinspector as part of the Zookeeper Debian package, I want to be able to call it regardless of the current path. Could you use getRessources or something so that I can point to the icons location from the wrapper shell script? Can I place the zooinspector config files in /etc/zookeeper/zooinspector/ ? Could I give zooinspector a property to point to the config file location? There are several places, where viewers is missspelled as Veiwers. Please do a case insensitive search for veiw to correct these. Even the config file defaultNodeVeiwers.cfg is missspelled like this. This has the potential to confuse the hell out of people when debugging something! -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-756) some cleanup and improvements for zooinspector
[ https://issues.apache.org/jira/browse/ZOOKEEPER-756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Goodheart-Smithe updated ZOOKEEPER-756: - Status: Patch Available (was: Open) Patch submitted - see previous comment some cleanup and improvements for zooinspector -- Key: ZOOKEEPER-756 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-756 Project: Zookeeper Issue Type: Improvement Components: contrib Affects Versions: 3.3.0 Reporter: Thomas Koch Assignee: Colin Goodheart-Smithe Fix For: 3.4.0 Attachments: zooInspectorChanges.patch Copied from the already closed ZOOKEEPER-678: * specify the exact URL, where the icons are from. It's best to include the link also in the NOTICE.txt file. It seems, that zooinspector finds it's icons only if the icons folder is in the current path. But when I install zooinspector as part of the Zookeeper Debian package, I want to be able to call it regardless of the current path. Could you use getRessources or something so that I can point to the icons location from the wrapper shell script? Can I place the zooinspector config files in /etc/zookeeper/zooinspector/ ? Could I give zooinspector a property to point to the config file location? There are several places, where viewers is missspelled as Veiwers. Please do a case insensitive search for veiw to correct these. Even the config file defaultNodeVeiwers.cfg is missspelled like this. This has the potential to confuse the hell out of people when debugging something! -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-756) some cleanup and improvements for zooinspector
[ https://issues.apache.org/jira/browse/ZOOKEEPER-756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12932637#action_12932637 ] Hadoop QA commented on ZOOKEEPER-756: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12459725/zooInspectorChanges.patch against trunk revision 1035727. +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://hudson.apache.org/hudson/job/PreCommit-ZOOKEEPER-Build/33//console This message is automatically generated. some cleanup and improvements for zooinspector -- Key: ZOOKEEPER-756 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-756 Project: Zookeeper Issue Type: Improvement Components: contrib Affects Versions: 3.3.0 Reporter: Thomas Koch Assignee: Colin Goodheart-Smithe Fix For: 3.4.0 Attachments: zooInspectorChanges.patch Copied from the already closed ZOOKEEPER-678: * specify the exact URL, where the icons are from. It's best to include the link also in the NOTICE.txt file. It seems, that zooinspector finds it's icons only if the icons folder is in the current path. But when I install zooinspector as part of the Zookeeper Debian package, I want to be able to call it regardless of the current path. Could you use getRessources or something so that I can point to the icons location from the wrapper shell script? Can I place the zooinspector config files in /etc/zookeeper/zooinspector/ ? Could I give zooinspector a property to point to the config file location? There are several places, where viewers is missspelled as Veiwers. Please do a case insensitive search for veiw to correct these. Even the config file defaultNodeVeiwers.cfg is missspelled like this. This has the potential to confuse the hell out of people when debugging something! -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-922) enable faster timeout of sessions in case of unexpected socket disconnect
[ https://issues.apache.org/jira/browse/ZOOKEEPER-922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12932639#action_12932639 ] Benjamin Reed commented on ZOOKEEPER-922: - if we had a foolproof way to tell that a client is down, we could do this fast expire. the methods you are proposing are not foolproof and will lead to problems exactly when you most want them not to. the timeout interactions you are talking about are problematic. it's really hard to get them right. one way that i can see this working is to not allow clients to reconnect to other servers. in that can a socket reset would indicate an expired session. is this acceptable to you? enable faster timeout of sessions in case of unexpected socket disconnect - Key: ZOOKEEPER-922 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-922 Project: Zookeeper Issue Type: Improvement Components: server Reporter: Camille Fournier Assignee: Camille Fournier Fix For: 3.4.0 Attachments: ZOOKEEPER-922.patch In the case when a client connection is closed due to socket error instead of the client calling close explicitly, it would be nice to enable the session associated with that client to time out faster than the negotiated session timeout. This would enable a zookeeper ensemble that is acting as a dynamic discovery provider to remove ephemeral nodes for crashed clients quickly, while allowing for a longer heartbeat-based timeout for java clients that need to do long stop-the-world GC. I propose doing this by setting the timeout associated with the crashed session to minSessionTimeout. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-922) enable faster timeout of sessions in case of unexpected socket disconnect
[ https://issues.apache.org/jira/browse/ZOOKEEPER-922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12932649#action_12932649 ] Camille Fournier commented on ZOOKEEPER-922: I'm interested in hearing the problems that you believe it would lead to in more detail. To me, this feels like a reasonable compromise solution to a tough problem. If the problem you foresee is a client and server getting disconnected from each other but both staying alive, and this causing weirdness leading to a session expiration for the client on reconnecting to another server, for my particular scenario that is fine. I have a wrapped ZK client that is highly tolerant to all sorts of failures and has no problem resetting its state. I realize that may not be acceptable for other users, and I would not propose this solution without either community agreement that this risk, if well-documented, is ok, or a fix for that problem. But I don't know what other problems you are seeing and while I might be able to solve them if you help me see what they are, I can't do anything on vague suppositions of problematic circumstances. Don't get me wrong, I'm not married to this solution, but I am interested in some solution if possible. It seems to me that not allowing clients to reconnect to other servers causes a host of other problems and is a worse solution for people that would not want this fast expiration forced on them. In what scenarios can a client not reconnect to another server? All? Obviously that won't fly because even I would not want to have all of my sessions expire in the case of an ensemble member dying and clients failing over. If we only want to do this where my code is doing the touchAndClose (ie, when the server the client was connected to sees a failure-based disconnect), then we see exactly the same potential problem outlined above where the client could still be alive but have a switch go down and disconnect it from the server. Now it tries to fail over and its session is always dead. I'm not convinced off the bat that that is any better than letting it try to fail over and risking a potential session timeout race, which I think could possibly be fixed by associating the client session with the server currently maintaining it (already done but not passed through on ticks). What did you mean in the earlier comment about this causing leadership election issues? Does this actually interact with that at all? This is the kind of thing I could use guidance on. Or we can let this whole idea drop, but it does seem that more people than me are interested so might be worth hashing it out. enable faster timeout of sessions in case of unexpected socket disconnect - Key: ZOOKEEPER-922 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-922 Project: Zookeeper Issue Type: Improvement Components: server Reporter: Camille Fournier Assignee: Camille Fournier Fix For: 3.4.0 Attachments: ZOOKEEPER-922.patch In the case when a client connection is closed due to socket error instead of the client calling close explicitly, it would be nice to enable the session associated with that client to time out faster than the negotiated session timeout. This would enable a zookeeper ensemble that is acting as a dynamic discovery provider to remove ephemeral nodes for crashed clients quickly, while allowing for a longer heartbeat-based timeout for java clients that need to do long stop-the-world GC. I propose doing this by setting the timeout associated with the crashed session to minSessionTimeout. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-366) Session timeout detection can go wrong if the leader system time changes
[ https://issues.apache.org/jira/browse/ZOOKEEPER-366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12932708#action_12932708 ] Patrick Hunt commented on ZOOKEEPER-366: FYI, this came up again today on hbase list: 14:59 _hp_ man this system time update on a bunch of machines causing zookeeper session timeouts causing hr's to die is really taking its toll, count on a table now hangs, i disabled and enabled the table, tried count again, and it hangs at the same place still. Arg. Ben any progress on this? Should we try to get it into 3.3.3? Session timeout detection can go wrong if the leader system time changes Key: ZOOKEEPER-366 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-366 Project: Zookeeper Issue Type: Bug Reporter: Benjamin Reed Assignee: Benjamin Reed Attachments: ZOOKEEPER-366.patch the leader tracks session expirations by calculating when a session will timeout and then periodically checking to see what needs to be timed out based on the current time. this works great as long as the leaders clock progresses at a steady pace. the problem comes when there are big (session size) changes in clock, by ntp for example. if time gets adjusted forward, all the sessions could timeout immediately. if time goes backward sessions that should timeout may take a lot longer to actually expire. this is really just a leader issue. the easiest way to deal with this is to have the leader relinquish leadership if it detects a big jump forward in time. when a new leader gets elected, it will recalculate timeouts of active sessions. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-366) Session timeout detection can go wrong if the leader system time changes
[ https://issues.apache.org/jira/browse/ZOOKEEPER-366?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Hunt updated ZOOKEEPER-366: --- Component/s: server quorum Fix Version/s: 3.4.0 3.3.3 Session timeout detection can go wrong if the leader system time changes Key: ZOOKEEPER-366 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-366 Project: Zookeeper Issue Type: Bug Components: quorum, server Reporter: Benjamin Reed Assignee: Benjamin Reed Fix For: 3.3.3, 3.4.0 Attachments: ZOOKEEPER-366.patch the leader tracks session expirations by calculating when a session will timeout and then periodically checking to see what needs to be timed out based on the current time. this works great as long as the leaders clock progresses at a steady pace. the problem comes when there are big (session size) changes in clock, by ntp for example. if time gets adjusted forward, all the sessions could timeout immediately. if time goes backward sessions that should timeout may take a lot longer to actually expire. this is really just a leader issue. the easiest way to deal with this is to have the leader relinquish leadership if it detects a big jump forward in time. when a new leader gets elected, it will recalculate timeouts of active sessions. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-900) FLE implementation should be improved to use non-blocking sockets
[ https://issues.apache.org/jira/browse/ZOOKEEPER-900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12932724#action_12932724 ] Vishal K commented on ZOOKEEPER-900: Hi Flavio, Currently the QCM keeps the TCP connections to other peers alive even after finishing leader election. How about we shutdown these threads after a node decides to be a leader or a follower?The threads get created on fly. Do you see any problem with that? Thanks. -Vishal FLE implementation should be improved to use non-blocking sockets - Key: ZOOKEEPER-900 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-900 Project: Zookeeper Issue Type: Bug Reporter: Vishal K Assignee: Vishal K Priority: Critical Fix For: 3.4.0 Attachments: ZOOKEEPER-900.patch, ZOOKEEPER-900.patch1, ZOOKEEPER-900.patch2 From earlier email exchanges: 1. Blocking connects and accepts: a) The first problem is in manager.toSend(). This invokes connectOne(), which does a blocking connect. While testing, I changed the code so that connectOne() starts a new thread called AsyncConnct(). AsyncConnect.run() does a socketChannel.connect(). After starting AsyncConnect, connectOne starts a timer. connectOne continues with normal operations if the connection is established before the timer expires, otherwise, when the timer expires it interrupts AsyncConnect() thread and returns. In this way, I can have an upper bound on the amount of time we need to wait for connect to succeed. Of course, this was a quick fix for my testing. Ideally, we should use Selector to do non-blocking connects/accepts. I am planning to do that later once we at least have a quick fix for the problem and consensus from others for the real fix (this problem is big blocker for us). Note that it is OK to do blocking IO in SenderWorker and RecvWorker threads since they block IO to the respective ! peer. b) The blocking IO problem is not just restricted to connectOne(), but also in receiveConnection(). The Listener thread calls receiveConnection() for each incoming connection request. receiveConnection does blocking IO to get peer's info (s.read(msgBuffer)). Worse, it invokes connectOne() back to the peer that had sent the connection request. All of this is happening from the Listener. In short, if a peer fails after initiating a connection, the Listener thread won't be able to accept connections from other peers, because it would be stuck in read() or connetOne(). Also the code has an inherent cycle. initiateConnection() and receiveConnection() will have to be very carefully synchronized otherwise, we could run into deadlocks. This code is going to be difficult to maintain/modify. Also see: https://issues.apache.org/jira/browse/ZOOKEEPER-822 -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-896) Improve C client to support dynamic authentication schemes
[ https://issues.apache.org/jira/browse/ZOOKEEPER-896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koontz updated ZOOKEEPER-896: Hi Patrick, I would love to help with the documentation and have some notes already for myself that I will use as source material. Improve C client to support dynamic authentication schemes -- Key: ZOOKEEPER-896 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-896 Project: Zookeeper Issue Type: Improvement Components: c client Affects Versions: 3.3.1 Reporter: Botond Hejj Assignee: Botond Hejj Fix For: 3.4.0 Attachments: ZOOKEEPER-896.patch When we started exploring zookeeper for our requirements we found the authentication mechanism is not flexible enough. We want to use kerberos for authentication but using the current API we ran into a few problems. The idea is that we get a kerberos token on the client side and than send that token to the server with a kerberos scheme. A server side authentication plugin can use that token to authenticate the client and also use the token for authorization. We ran into two problems with this approach: 1. A different kerberos token is needed for each different server that client can connect to since kerberos uses mutual authentication. That means when the client acquires this kerberos token it has to know which server it connects to and generate the token according to that. The client currently can't generate a token for a specific server. The token stored in the auth_info is used for all the servers. 2. The kerberos token might have an expiry time so if the client loses the connection to the server and than it tries to reconnect it should acquire a new token. That is not possible currently since the token is stored in auth_info and reused for every connection. The problem can be solved if we allow the client to register a callback for authentication instead a static token. This can be a callback with an argument which passes the current host string. The zookeeper client code could call this callback before it sends the authentication info to the server to get a fresh server specific token. This would solve our problem with the kerberos authentication and also could be used for other more dynamic authentication schemes. The solution could be generalization also for the java client as well. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-925) Consider maven site generation to replace our forrest site and documentation generation
[ https://issues.apache.org/jira/browse/ZOOKEEPER-925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12932799#action_12932799 ] Benjamin Reed commented on ZOOKEEPER-925: - i cannot figure out how to convert forrest to anything. actually, i can't figure out how we have forrest working at all! after burning the afternoon trying to figure out how to convert forrest to confluence, i'm officially declaring defeat. it should be an easy thing to do for an xml/xsl master, but that is not me. the most promising thing appears to be the doxia converter that will go from a bunch of formats to a bunch more formats, including from docbook or xdoc to confluence. unfortunately, forrest seems close to both of those, but not close enough... Consider maven site generation to replace our forrest site and documentation generation --- Key: ZOOKEEPER-925 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-925 Project: Zookeeper Issue Type: Wish Components: documentation Reporter: Patrick Hunt Assignee: Patrick Hunt Attachments: ZOOKEEPER-925.patch See WHIRR-19 for some background. In whirr we looked at a number of site/doc generation facilities. In the end Maven site generation plugin turned out to be by far the best option. You can see our nascent site here (no attempt at styling,etc so far): http://incubator.apache.org/whirr/ In particular take a look at the quick start: http://incubator.apache.org/whirr/quick-start-guide.html which was generated from http://svn.apache.org/repos/asf/incubator/whirr/trunk/src/site/confluence/quick-start-guide.confluence notice this was standard wiki markup (confluence wiki markup, same as available from apache) You can read more about mvn site plugin here: http://maven.apache.org/guides/mini/guide-site.html Notice that other formats are available, not just confluence markup, also note that you can use different markup formats if you like in the same site (although probably not a great idea, but in some cases might be handy, for example whirr uses the confluence wiki, so we can pretty much copy/paste source docs from wiki to our site (svn) if we like) Re maven vs our current ant based build. It's probably a good idea for us to move the build to maven at some point. We could initially move just the doc generation, and then incrementally move functionality from build.xml to mvn over a longer time period. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-366) Session timeout detection can go wrong if the leader system time changes
[ https://issues.apache.org/jira/browse/ZOOKEEPER-366?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12932809#action_12932809 ] Benjamin Reed commented on ZOOKEEPER-366: - i haven't had a chance to get back to this. we really need to convert all the currentTimeMillis() to nanoTime(). we need to do a similar change in the C client. i don't think we can do a test for this. Session timeout detection can go wrong if the leader system time changes Key: ZOOKEEPER-366 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-366 Project: Zookeeper Issue Type: Bug Components: quorum, server Reporter: Benjamin Reed Assignee: Benjamin Reed Fix For: 3.3.3, 3.4.0 Attachments: ZOOKEEPER-366.patch the leader tracks session expirations by calculating when a session will timeout and then periodically checking to see what needs to be timed out based on the current time. this works great as long as the leaders clock progresses at a steady pace. the problem comes when there are big (session size) changes in clock, by ntp for example. if time gets adjusted forward, all the sessions could timeout immediately. if time goes backward sessions that should timeout may take a lot longer to actually expire. this is really just a leader issue. the easiest way to deal with this is to have the leader relinquish leadership if it detects a big jump forward in time. when a new leader gets elected, it will recalculate timeouts of active sessions. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-756) some cleanup and improvements for zooinspector
[ https://issues.apache.org/jira/browse/ZOOKEEPER-756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Hunt updated ZOOKEEPER-756: --- Status: Open (was: Patch Available) some cleanup and improvements for zooinspector -- Key: ZOOKEEPER-756 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-756 Project: Zookeeper Issue Type: Improvement Components: contrib Affects Versions: 3.3.0 Reporter: Thomas Koch Assignee: Colin Goodheart-Smithe Fix For: 3.4.0 Attachments: zooInspectorChanges.patch, ZOOKEEPER-756.patch Copied from the already closed ZOOKEEPER-678: * specify the exact URL, where the icons are from. It's best to include the link also in the NOTICE.txt file. It seems, that zooinspector finds it's icons only if the icons folder is in the current path. But when I install zooinspector as part of the Zookeeper Debian package, I want to be able to call it regardless of the current path. Could you use getRessources or something so that I can point to the icons location from the wrapper shell script? Can I place the zooinspector config files in /etc/zookeeper/zooinspector/ ? Could I give zooinspector a property to point to the config file location? There are several places, where viewers is missspelled as Veiwers. Please do a case insensitive search for veiw to correct these. Even the config file defaultNodeVeiwers.cfg is missspelled like this. This has the potential to confuse the hell out of people when debugging something! -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-756) some cleanup and improvements for zooinspector
[ https://issues.apache.org/jira/browse/ZOOKEEPER-756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Hunt updated ZOOKEEPER-756: --- Status: Patch Available (was: Open) some cleanup and improvements for zooinspector -- Key: ZOOKEEPER-756 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-756 Project: Zookeeper Issue Type: Improvement Components: contrib Affects Versions: 3.3.0 Reporter: Thomas Koch Assignee: Colin Goodheart-Smithe Fix For: 3.4.0 Attachments: zooInspectorChanges.patch, ZOOKEEPER-756.patch Copied from the already closed ZOOKEEPER-678: * specify the exact URL, where the icons are from. It's best to include the link also in the NOTICE.txt file. It seems, that zooinspector finds it's icons only if the icons folder is in the current path. But when I install zooinspector as part of the Zookeeper Debian package, I want to be able to call it regardless of the current path. Could you use getRessources or something so that I can point to the icons location from the wrapper shell script? Can I place the zooinspector config files in /etc/zookeeper/zooinspector/ ? Could I give zooinspector a property to point to the config file location? There are several places, where viewers is missspelled as Veiwers. Please do a case insensitive search for veiw to correct these. Even the config file defaultNodeVeiwers.cfg is missspelled like this. This has the potential to confuse the hell out of people when debugging something! -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-756) some cleanup and improvements for zooinspector
[ https://issues.apache.org/jira/browse/ZOOKEEPER-756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Hunt updated ZOOKEEPER-756: --- Attachment: ZOOKEEPER-756.patch Looks like line endings are the problem. The current source in svn has some lines with ^M line endings and this seems to be messing up the patch application (at least under unix). I tried it myself and had similar problems (I'm on unix). I then used dos2unix to convert all the zooinspector files to unix line endings, then reapplied Colin's patch and it patched fine. I created a new patch file based on this, which I'm now attaching. Please review this updated patch and make sure I didn't miss anything some cleanup and improvements for zooinspector -- Key: ZOOKEEPER-756 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-756 Project: Zookeeper Issue Type: Improvement Components: contrib Affects Versions: 3.3.0 Reporter: Thomas Koch Assignee: Colin Goodheart-Smithe Fix For: 3.4.0 Attachments: zooInspectorChanges.patch, ZOOKEEPER-756.patch Copied from the already closed ZOOKEEPER-678: * specify the exact URL, where the icons are from. It's best to include the link also in the NOTICE.txt file. It seems, that zooinspector finds it's icons only if the icons folder is in the current path. But when I install zooinspector as part of the Zookeeper Debian package, I want to be able to call it regardless of the current path. Could you use getRessources or something so that I can point to the icons location from the wrapper shell script? Can I place the zooinspector config files in /etc/zookeeper/zooinspector/ ? Could I give zooinspector a property to point to the config file location? There are several places, where viewers is missspelled as Veiwers. Please do a case insensitive search for veiw to correct these. Even the config file defaultNodeVeiwers.cfg is missspelled like this. This has the potential to confuse the hell out of people when debugging something! -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (ZOOKEEPER-756) some cleanup and improvements for zooinspector
[ https://issues.apache.org/jira/browse/ZOOKEEPER-756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12932839#action_12932839 ] Hadoop QA commented on ZOOKEEPER-756: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12459771/ZOOKEEPER-756.patch against trunk revision 1035727. -1 @author. The patch appears to contain 4 @author tags which the Zookeeper community has agreed to not allow in code contributions. +1 tests included. The patch appears to include 6 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 1 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. +1 contrib tests. The patch passed contrib unit tests. Test results: https://hudson.apache.org/hudson/job/PreCommit-ZOOKEEPER-Build/34//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-ZOOKEEPER-Build/34//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-ZOOKEEPER-Build/34//console This message is automatically generated. some cleanup and improvements for zooinspector -- Key: ZOOKEEPER-756 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-756 Project: Zookeeper Issue Type: Improvement Components: contrib Affects Versions: 3.3.0 Reporter: Thomas Koch Assignee: Colin Goodheart-Smithe Fix For: 3.4.0 Attachments: zooInspectorChanges.patch, ZOOKEEPER-756.patch Copied from the already closed ZOOKEEPER-678: * specify the exact URL, where the icons are from. It's best to include the link also in the NOTICE.txt file. It seems, that zooinspector finds it's icons only if the icons folder is in the current path. But when I install zooinspector as part of the Zookeeper Debian package, I want to be able to call it regardless of the current path. Could you use getRessources or something so that I can point to the icons location from the wrapper shell script? Can I place the zooinspector config files in /etc/zookeeper/zooinspector/ ? Could I give zooinspector a property to point to the config file location? There are several places, where viewers is missspelled as Veiwers. Please do a case insensitive search for veiw to correct these. Even the config file defaultNodeVeiwers.cfg is missspelled like this. This has the potential to confuse the hell out of people when debugging something! -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (ZOOKEEPER-756) some cleanup and improvements for zooinspector
[ https://issues.apache.org/jira/browse/ZOOKEEPER-756?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Patrick Hunt updated ZOOKEEPER-756: --- Status: Open (was: Patch Available) cancelling patch - please address the @author issues (remove them), the javadoc warning seems to be some issue with the link being accessed (not with the change itself). some cleanup and improvements for zooinspector -- Key: ZOOKEEPER-756 URL: https://issues.apache.org/jira/browse/ZOOKEEPER-756 Project: Zookeeper Issue Type: Improvement Components: contrib Affects Versions: 3.3.0 Reporter: Thomas Koch Assignee: Colin Goodheart-Smithe Fix For: 3.4.0 Attachments: zooInspectorChanges.patch, ZOOKEEPER-756.patch Copied from the already closed ZOOKEEPER-678: * specify the exact URL, where the icons are from. It's best to include the link also in the NOTICE.txt file. It seems, that zooinspector finds it's icons only if the icons folder is in the current path. But when I install zooinspector as part of the Zookeeper Debian package, I want to be able to call it regardless of the current path. Could you use getRessources or something so that I can point to the icons location from the wrapper shell script? Can I place the zooinspector config files in /etc/zookeeper/zooinspector/ ? Could I give zooinspector a property to point to the config file location? There are several places, where viewers is missspelled as Veiwers. Please do a case insensitive search for veiw to correct these. Even the config file defaultNodeVeiwers.cfg is missspelled like this. This has the potential to confuse the hell out of people when debugging something! -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
