[jira] [Commented] (ZOOKEEPER-2845) Data inconsistency issue due to retain database in leader election
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124813#comment-16124813 ] Michael Han commented on ZOOKEEPER-2845: [~lvfangmin] Any plan to submit your retain db implementation? This is an important bug to fix. > Data inconsistency issue due to retain database in leader election > -- > > Key: ZOOKEEPER-2845 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2845 > Project: ZooKeeper > Issue Type: Bug > Components: quorum >Affects Versions: 3.4.10, 3.5.3, 3.6.0 >Reporter: Fangmin Lv >Assignee: Fangmin Lv >Priority: Critical > > In ZOOKEEPER-2678, the ZKDatabase is retained to reduce the unavailable time > during leader election. In ZooKeeper ensemble, it's possible that the > snapshot is ahead of txn file (due to slow disk on the server, etc), or the > txn file is ahead of snapshot due to no commit message being received yet. > If snapshot is ahead of txn file, since the SyncRequestProcessor queue will > be drained during shutdown, the snapshot and txn file will keep consistent > before leader election happening, so this is not an issue. > But if txn is ahead of snapshot, it's possible that the ensemble will have > data inconsistent issue, here is the simplified scenario to show the issue: > Let's say we have a 3 servers in the ensemble, server A and B are followers, > and C is leader, and all the snapshot and txn are up to T0: > 1. A new request reached to leader C to create Node N, and it's converted to > txn T1 > 2. Txn T1 was synced to disk in C, but just before the proposal reaching out > to the followers, A and B restarted, so the T1 didn't exist in A and B > 3. A and B formed a new quorum after restart, let's say B is the leader > 4. C changed to looking state due to no enough followers, it will sync with > leader B with last Zxid T0, which will have an empty diff sync > 5. Before C take snapshot it restarted, it replayed the txns on disk which > includes T1, now it will have Node N, but A and B doesn't have it. > Also I included the a test case to reproduce this issue consistently. > We have a totally different RetainDB version which will avoid this issue by > doing consensus between snapshot and txn files before leader election, will > submit for review. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ZOOKEEPER-2846) Leader follower sync with on disk txns can possibly leads to data inconsistency
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124812#comment-16124812 ] Michael Han commented on ZOOKEEPER-2846: [~lvfangmin] Do you have a plan to upload a patch that fixes the issue? The existing pull request contains a test only. > Leader follower sync with on disk txns can possibly leads to data > inconsistency > --- > > Key: ZOOKEEPER-2846 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2846 > Project: ZooKeeper > Issue Type: Bug > Components: quorum >Affects Versions: 3.4.10, 3.5.3, 3.6.0 >Reporter: Fangmin Lv >Priority: Critical > > On disk txn sync could cause data inconsistency if the current leader just > had a snap sync before it became leader, and then having diff sync with its > followers may synced the txns gap on disk. Here is scenario: > Let's say S0 - S3 are followers, and S4 is leader at the beginning: > 1. Stop S2 and send one more request > 2. Stop S3 and send more requests to the quorum to let S3 have a snap sync > with S4 when it started up > 3. Stop S4 and S3 became the new leader > 4. Start S2 and had a diff sync with S3, now there are gaps in S2 > Attached the test case to verify the issue. Currently, there is no efficient > way to check the gap in txn files is a real gap or due to Epoch change. We > need to add that support, but before that, it would be safer to disable the > on disk txn leader-follower sync. > Another two scenarios which could cause the same issue: > (Scenario 1) Server A, B, C, A is leader, the others are followers: > 1). A synced to disk, but the other 2 restarted before receiving the > proposal > 2). B and C formed quorum, B is leader, and committed some requests > 3). A looking again, and sync with B, B won't able to trunc A but send snap > instead, and leaves the extra txn in A's txn file > 4). A became new leader, and someone else has a diff sync with A it will > have the extra txn > (Scenario 2) Diff sync with committed txn, will only apply to data tree but > not on disk txn file, which will also leave hole in it and lead to data > inconsistency issue when syncing with learners. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] zookeeper issue #325: Fix typos in zookeeperAdmin.html
Github user hanm commented on the issue: https://github.com/apache/zookeeper/pull/325 merged to master. In the future, please create a JIRA issue before sending a pull request. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] zookeeper pull request #325: Fix typos in zookeeperAdmin.html
Github user asfgit closed the pull request at: https://github.com/apache/zookeeper/pull/325 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (ZOOKEEPER-2859) CMake build doesn't support OS X
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124808#comment-16124808 ] ASF GitHub Bot commented on ZOOKEEPER-2859: --- Github user hanm commented on the issue: https://github.com/apache/zookeeper/pull/319 The precommit job failured was fixed separately, and irrelevant to this patch. LGTM, merged to master: https://github.com/apache/zookeeper/commit/5bcffe9bc242d825ad97ff4f9d2dea6476983a9a 3.5: https://github.com/apache/zookeeper/commit/1bcf57dc775c5cec00f1f54414cf5a0e5f18e2f4 3.4: https://github.com/apache/zookeeper/commit/1f811a6281090e1b24152dc51507aa6a2bdeafe3 > CMake build doesn't support OS X > > > Key: ZOOKEEPER-2859 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2859 > Project: ZooKeeper > Issue Type: Bug > Environment: OS X 10.12.6 >Reporter: Andrew Schwartzmeyer >Assignee: Andrew Schwartzmeyer > Fix For: 3.5.4, 3.6.0, 3.4.11 > > > Couple problems: > libm, librt, and libdl are all Linux specific, and provided "for free" on OS X > CppUnit (at least on OS X) needs `-std=c++11` > clang's ld doesn't understand --wrap > I can post an easy patch that at least lets you build the client (but not the > tests). The tests use that `--wrap` and it's non trivial to fix that on OS X. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] zookeeper issue #319: ZOOKEEPER-2859: Fix CMake build on OS X.
Github user hanm commented on the issue: https://github.com/apache/zookeeper/pull/319 The precommit job failured was fixed separately, and irrelevant to this patch. LGTM, merged to master: https://github.com/apache/zookeeper/commit/5bcffe9bc242d825ad97ff4f9d2dea6476983a9a 3.5: https://github.com/apache/zookeeper/commit/1bcf57dc775c5cec00f1f54414cf5a0e5f18e2f4 3.4: https://github.com/apache/zookeeper/commit/1f811a6281090e1b24152dc51507aa6a2bdeafe3 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Resolved] (ZOOKEEPER-2859) CMake build doesn't support OS X
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2859?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Han resolved ZOOKEEPER-2859. Resolution: Fixed Fix Version/s: 3.5.4 3.6.0 3.4.11 Issue resolved by pull request 319 [https://github.com/apache/zookeeper/pull/319] > CMake build doesn't support OS X > > > Key: ZOOKEEPER-2859 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2859 > Project: ZooKeeper > Issue Type: Bug > Environment: OS X 10.12.6 >Reporter: Andrew Schwartzmeyer >Assignee: Andrew Schwartzmeyer > Fix For: 3.4.11, 3.6.0, 3.5.4 > > > Couple problems: > libm, librt, and libdl are all Linux specific, and provided "for free" on OS X > CppUnit (at least on OS X) needs `-std=c++11` > clang's ld doesn't understand --wrap > I can post an easy patch that at least lets you build the client (but not the > tests). The tests use that `--wrap` and it's non trivial to fix that on OS X. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[GitHub] zookeeper pull request #319: ZOOKEEPER-2859: Fix CMake build on OS X.
Github user asfgit closed the pull request at: https://github.com/apache/zookeeper/pull/319 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (ZOOKEEPER-2859) CMake build doesn't support OS X
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2859?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124806#comment-16124806 ] ASF GitHub Bot commented on ZOOKEEPER-2859: --- Github user asfgit closed the pull request at: https://github.com/apache/zookeeper/pull/319 > CMake build doesn't support OS X > > > Key: ZOOKEEPER-2859 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2859 > Project: ZooKeeper > Issue Type: Bug > Environment: OS X 10.12.6 >Reporter: Andrew Schwartzmeyer >Assignee: Andrew Schwartzmeyer > > Couple problems: > libm, librt, and libdl are all Linux specific, and provided "for free" on OS X > CppUnit (at least on OS X) needs `-std=c++11` > clang's ld doesn't understand --wrap > I can post an easy patch that at least lets you build the client (but not the > tests). The tests use that `--wrap` and it's non trivial to fix that on OS X. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ZOOKEEPER-2872) Interrupted snapshot sync causes data loss
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124803#comment-16124803
]
ASF GitHub Bot commented on ZOOKEEPER-2872:
---
Github user hanm commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/333#discussion_r132832594
--- Diff: src/java/main/org/apache/zookeeper/server/quorum/Learner.java ---
@@ -364,6 +364,7 @@ protected void syncWithLeader(long newLeaderZxid)
throws Exception{
readPacket(qp);
LinkedList packetsCommitted = new LinkedList();
LinkedList packetsNotCommitted = new
LinkedList();
+boolean syncSnapshot = false;
--- End diff --
We can level this variable definition up so it's clustered with
`snapshotNeed` boolean.
Another possibility is to get ride of this variable and use existing
`snapshotNeeded` - but that will do fysnc snapshot for TRUNC sync, which the
existing patch will not.
> Interrupted snapshot sync causes data loss
> --
>
> Key: ZOOKEEPER-2872
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2872
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
>Affects Versions: 3.4.10, 3.5.3, 3.6.0
>Reporter: Brian Nixon
>
> There is a way for observers to permanently lose data from their local data
> tree while remaining members of good standing with the ensemble and
> continuing to serve client traffic when the following chain of events occurs.
> 1. The observer dies in epoch N from machine failure.
> 2. The observer comes back up in epoch N+1 and requests a snapshot sync to
> catch up.
> 3. The machine powers off before the snapshot is synced to disc and after
> some txn's have been logged (depending on the OS, this can happen!).
> 4. The observer comes back a second time and replays its most recent snapshot
> (epoch <= N) as well as the txn logs (epoch N+1).
> 5. A diff sync is requested from the leader and the observer broadcasts
> availability.
> In this scenario, any commits from epoch N that the observer did not receive
> before it died the first time will never be exposed to the observer and no
> part of the ensemble will complain.
> This situation is not unique to observers and can happen to any learner. As a
> simple fix, fsync-ing the snapshots received from the leader will avoid the
> case of missing snapshots causing data loss.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (ZOOKEEPER-2872) Interrupted snapshot sync causes data loss
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124805#comment-16124805
]
ASF GitHub Bot commented on ZOOKEEPER-2872:
---
Github user hanm commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/333#discussion_r132832602
--- Diff: src/java/main/org/apache/zookeeper/server/quorum/Learner.java ---
@@ -364,6 +364,7 @@ protected void syncWithLeader(long newLeaderZxid)
throws Exception{
readPacket(qp);
LinkedList packetsCommitted = new LinkedList();
LinkedList packetsNotCommitted = new
LinkedList();
+boolean syncSnapshot = false;
--- End diff --
Another possibility as I just commented is to get rid of this variable and
always Fsync snapshot serialization.
> Interrupted snapshot sync causes data loss
> --
>
> Key: ZOOKEEPER-2872
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2872
> Project: ZooKeeper
> Issue Type: Bug
> Components: server
>Affects Versions: 3.4.10, 3.5.3, 3.6.0
>Reporter: Brian Nixon
>
> There is a way for observers to permanently lose data from their local data
> tree while remaining members of good standing with the ensemble and
> continuing to serve client traffic when the following chain of events occurs.
> 1. The observer dies in epoch N from machine failure.
> 2. The observer comes back up in epoch N+1 and requests a snapshot sync to
> catch up.
> 3. The machine powers off before the snapshot is synced to disc and after
> some txn's have been logged (depending on the OS, this can happen!).
> 4. The observer comes back a second time and replays its most recent snapshot
> (epoch <= N) as well as the txn logs (epoch N+1).
> 5. A diff sync is requested from the leader and the observer broadcasts
> availability.
> In this scenario, any commits from epoch N that the observer did not receive
> before it died the first time will never be exposed to the observer and no
> part of the ensemble will complain.
> This situation is not unique to observers and can happen to any learner. As a
> simple fix, fsync-ing the snapshots received from the leader will avoid the
> case of missing snapshots causing data loss.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[GitHub] zookeeper pull request #333: ZOOKEEPER-2872: Interrupted snapshot sync cause...
Github user hanm commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/333#discussion_r132832602
--- Diff: src/java/main/org/apache/zookeeper/server/quorum/Learner.java ---
@@ -364,6 +364,7 @@ protected void syncWithLeader(long newLeaderZxid)
throws Exception{
readPacket(qp);
LinkedList packetsCommitted = new LinkedList();
LinkedList packetsNotCommitted = new
LinkedList();
+boolean syncSnapshot = false;
--- End diff --
Another possibility as I just commented is to get rid of this variable and
always Fsync snapshot serialization.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zookeeper pull request #333: ZOOKEEPER-2872: Interrupted snapshot sync cause...
Github user hanm commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/333#discussion_r132832594
--- Diff: src/java/main/org/apache/zookeeper/server/quorum/Learner.java ---
@@ -364,6 +364,7 @@ protected void syncWithLeader(long newLeaderZxid)
throws Exception{
readPacket(qp);
LinkedList packetsCommitted = new LinkedList();
LinkedList packetsNotCommitted = new
LinkedList();
+boolean syncSnapshot = false;
--- End diff --
We can level this variable definition up so it's clustered with
`snapshotNeed` boolean.
Another possibility is to get ride of this variable and use existing
`snapshotNeeded` - but that will do fysnc snapshot for TRUNC sync, which the
existing patch will not.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] zookeeper issue #333: ZOOKEEPER-2872: Interrupted snapshot sync causes data ...
Github user hanm commented on the issue: https://github.com/apache/zookeeper/pull/333 I am now wondering why we should not fsync snapshot taking at all cases. It seems to be a useful property to have for snapshot serialization, and will make code simpler. Any performance considerations that lead to the conclusion of only applying fsync snapshot when it's a SNAP sync? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (ZOOKEEPER-2872) Interrupted snapshot sync causes data loss
[ https://issues.apache.org/jira/browse/ZOOKEEPER-2872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124802#comment-16124802 ] ASF GitHub Bot commented on ZOOKEEPER-2872: --- Github user hanm commented on the issue: https://github.com/apache/zookeeper/pull/333 I am now wondering why we should not fsync snapshot taking at all cases. It seems to be a useful property to have for snapshot serialization, and will make code simpler. Any performance considerations that lead to the conclusion of only applying fsync snapshot when it's a SNAP sync? > Interrupted snapshot sync causes data loss > -- > > Key: ZOOKEEPER-2872 > URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2872 > Project: ZooKeeper > Issue Type: Bug > Components: server >Affects Versions: 3.4.10, 3.5.3, 3.6.0 >Reporter: Brian Nixon > > There is a way for observers to permanently lose data from their local data > tree while remaining members of good standing with the ensemble and > continuing to serve client traffic when the following chain of events occurs. > 1. The observer dies in epoch N from machine failure. > 2. The observer comes back up in epoch N+1 and requests a snapshot sync to > catch up. > 3. The machine powers off before the snapshot is synced to disc and after > some txn's have been logged (depending on the OS, this can happen!). > 4. The observer comes back a second time and replays its most recent snapshot > (epoch <= N) as well as the txn logs (epoch N+1). > 5. A diff sync is requested from the leader and the observer broadcasts > availability. > In this scenario, any commits from epoch N that the observer did not receive > before it died the first time will never be exposed to the observer and no > part of the ensemble will complain. > This situation is not unique to observers and can happen to any learner. As a > simple fix, fsync-ing the snapshots received from the leader will avoid the > case of missing snapshots causing data loss. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ZOOKEEPER-2836) QuorumCnxManager.Listener Thread Better handling of SocketTimeoutException
[
https://issues.apache.org/jira/browse/ZOOKEEPER-2836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16124606#comment-16124606
]
ASF GitHub Bot commented on ZOOKEEPER-2836:
---
Github user maoling commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/334#discussion_r132820615
--- Diff:
src/java/main/org/apache/zookeeper/server/quorum/QuorumCnxManager.java ---
@@ -647,11 +648,11 @@ public void run() {
numRetries = 0;
}
} catch (IOException e) {
-if (shutdown) {
-break;
-}
LOG.error("Exception while listening", e);
-numRetries++;
+if (!(e instanceof SocketTimeoutException)) {
+numRetries++;
+}
+}finally {
--- End diff --
1. add a space between **}** and **finally**
2. why we need to move some codes about closing **ServerSocket** in
**catch** code block to **finally** code block? this will
make codes in **Line677-Line685** redundant
3. If **numRetries** don't count **SocketTimeoutExceptions**.is this code
facing an endless loop if SocketTimeoutExceptions always happen for a long
time? this way of handling SocketTimeoutException is appropriate?
> QuorumCnxManager.Listener Thread Better handling of SocketTimeoutException
> --
>
> Key: ZOOKEEPER-2836
> URL: https://issues.apache.org/jira/browse/ZOOKEEPER-2836
> Project: ZooKeeper
> Issue Type: Bug
> Components: leaderElection, quorum
>Affects Versions: 3.4.6
> Environment: Machine: Linux 3.2.0-4-amd64 #1 SMP Debian 3.2.78-1
> x86_64 GNU/Linux
> Java Version: jdk64/jdk1.8.0_40
> zookeeper version: 3.4.6.2.3.2.0-2950
>Reporter: Amarjeet Singh
>Priority: Critical
>
> QuorumCnxManager Listener thread blocks SocketServer on accept but we are
> getting SocketTimeoutException on our boxes after 49days 17 hours . As per
> current code there is a 3 times retry and after that it says "_As I'm leaving
> the listener thread, I won't be able to participate in leader election any
> longer: $/$:3888__" , Once server nodes reache this state and
> we restart or add a new node ,it fails to join cluster and logs 'WARN
>
[GitHub] zookeeper pull request #334: ZOOKEEPER-2836 SocketTimeoutException
Github user maoling commented on a diff in the pull request:
https://github.com/apache/zookeeper/pull/334#discussion_r132820615
--- Diff:
src/java/main/org/apache/zookeeper/server/quorum/QuorumCnxManager.java ---
@@ -647,11 +648,11 @@ public void run() {
numRetries = 0;
}
} catch (IOException e) {
-if (shutdown) {
-break;
-}
LOG.error("Exception while listening", e);
-numRetries++;
+if (!(e instanceof SocketTimeoutException)) {
+numRetries++;
+}
+}finally {
--- End diff --
1. add a space between **}** and **finally**
2. why we need to move some codes about closing **ServerSocket** in
**catch** code block to **finally** code block? this will
make codes in **Line677-Line685** redundant
3. If **numRetries** don't count **SocketTimeoutExceptions**.is this code
facing an endless loop if SocketTimeoutExceptions always happen for a long
time? this way of handling SocketTimeoutException is appropriate?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
ZooKeeper_branch35_jdk8 - Build # 632 - Failure
See https://builds.apache.org/job/ZooKeeper_branch35_jdk8/632/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 70.33 MB...] [junit] 2017-08-12 12:17:39,362 [myid:127.0.0.1:13918] - WARN [main-SendThread(127.0.0.1:13918):ClientCnxn$SendThread@1235] - Session 0x1071c2262270001 for server 127.0.0.1/127.0.0.1:13918, unexpected error, closing socket connection and attempting reconnect [junit] java.net.ConnectException: Connection refused [junit] at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [junit] at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) [junit] at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357) [junit] at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214) [junit] 2017-08-12 12:17:39,715 [myid:127.0.0.1:13915] - INFO [main-SendThread(127.0.0.1:13915):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:13915. Will not attempt to authenticate using SASL (unknown error) [junit] 2017-08-12 12:17:39,716 [myid:127.0.0.1:13915] - WARN [main-SendThread(127.0.0.1:13915):ClientCnxn$SendThread@1235] - Session 0x71c225f57 for server 127.0.0.1/127.0.0.1:13915, unexpected error, closing socket connection and attempting reconnect [junit] java.net.ConnectException: Connection refused [junit] at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [junit] at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) [junit] at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357) [junit] at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214) [junit] 2017-08-12 12:17:40,373 [myid:127.0.0.1:13927] - INFO [main-SendThread(127.0.0.1:13927):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:13927. Will not attempt to authenticate using SASL (unknown error) [junit] 2017-08-12 12:17:40,373 [myid:127.0.0.1:13927] - INFO [main-SendThread(127.0.0.1:13927):ClientCnxn$SendThread@946] - Socket connection established, initiating session, client: /127.0.0.1:59454, server: 127.0.0.1/127.0.0.1:13927 [junit] 2017-08-12 12:17:40,374 [myid:4] - INFO [NIOServerCxnFactory.AcceptThread:localhost/127.0.0.1:13927:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:59454 [junit] 2017-08-12 12:17:40,728 [myid:127.0.0.1:13918] - INFO [main-SendThread(127.0.0.1:13918):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:13918. Will not attempt to authenticate using SASL (unknown error) [junit] 2017-08-12 12:17:40,728 [myid:127.0.0.1:13918] - WARN [main-SendThread(127.0.0.1:13918):ClientCnxn$SendThread@1235] - Session 0x1071c226227 for server 127.0.0.1/127.0.0.1:13918, unexpected error, closing socket connection and attempting reconnect [junit] java.net.ConnectException: Connection refused [junit] at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [junit] at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) [junit] at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357) [junit] at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214) [junit] 2017-08-12 12:17:40,882 [myid:127.0.0.1:13915] - INFO [main-SendThread(127.0.0.1:13915):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:13915. Will not attempt to authenticate using SASL (unknown error) [junit] 2017-08-12 12:17:40,883 [myid:127.0.0.1:13915] - WARN [main-SendThread(127.0.0.1:13915):ClientCnxn$SendThread@1235] - Session 0x71c225f570001 for server 127.0.0.1/127.0.0.1:13915, unexpected error, closing socket connection and attempting reconnect [junit] java.net.ConnectException: Connection refused [junit] at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [junit] at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717) [junit] at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357) [junit] at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214) [junit] 2017-08-12 12:17:41,034 [myid:127.0.0.1:13918] - INFO [main-SendThread(127.0.0.1:13918):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:13918. Will not attempt to authenticate using SASL (unknown error) [junit] 2017-08-12 12:17:41,035 [myid:127.0.0.1:13918] - WARN [main-SendThread(127.0.0.1:13918):ClientCnxn$SendThread@1235] - Session 0x1071c2262270001 for server 127.0.0.1/127.0.0.1:13918, unexpected error, closing socket connection and attempting reconnect
ZooKeeper-trunk-jdk8 - Build # 1160 - Failure
See https://builds.apache.org/job/ZooKeeper-trunk-jdk8/1160/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 62.35 MB...] [junit] 2017-08-12 11:58:43,452 [myid:127.0.0.1:27386] - INFO [main-SendThread(127.0.0.1:27386):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:27386. Will not attempt to authenticate using SASL (unknown error) [junit] 2017-08-12 11:58:43,453 [myid:2] - INFO [NIOServerCxnFactory.AcceptThread:localhost/127.0.0.1:27386:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:38452 [junit] 2017-08-12 11:58:43,453 [myid:127.0.0.1:27386] - INFO [main-SendThread(127.0.0.1:27386):ClientCnxn$SendThread@946] - Socket connection established, initiating session, client: /127.0.0.1:38452, server: 127.0.0.1/127.0.0.1:27386 [junit] 2017-08-12 11:59:04,873 [myid:127.0.0.1:27386] - WARN [main-SendThread(127.0.0.1:27386):ClientCnxn$SendThread@1181] - Client session timed out, have not heard from server in 30030ms for sessionid 0x0 [junit] 2017-08-12 11:59:04,873 [myid:127.0.0.1:27386] - INFO [main-SendThread(127.0.0.1:27386):ClientCnxn$SendThread@1229] - Client session timed out, have not heard from server in 30030ms for sessionid 0x0, closing socket connection and attempting reconnect [junit] 2017-08-12 11:59:06,142 [myid:127.0.0.1:27386] - INFO [main-SendThread(127.0.0.1:27386):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:27386. Will not attempt to authenticate using SASL (unknown error) [junit] 2017-08-12 11:59:06,144 [myid:127.0.0.1:27386] - INFO [main-SendThread(127.0.0.1:27386):ClientCnxn$SendThread@946] - Socket connection established, initiating session, client: /127.0.0.1:39030, server: 127.0.0.1/127.0.0.1:27386 [junit] 2017-08-12 11:59:06,144 [myid:2] - INFO [NIOServerCxnFactory.AcceptThread:localhost/127.0.0.1:27386:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:39030 [junit] 2017-08-12 11:59:13,481 [myid:127.0.0.1:27386] - WARN [main-SendThread(127.0.0.1:27386):ClientCnxn$SendThread@1181] - Client session timed out, have not heard from server in 30029ms for sessionid 0x2054fb559f8 [junit] 2017-08-12 11:59:13,482 [myid:127.0.0.1:27386] - INFO [main-SendThread(127.0.0.1:27386):ClientCnxn$SendThread@1229] - Client session timed out, have not heard from server in 30029ms for sessionid 0x2054fb559f8, closing socket connection and attempting reconnect [junit] 2017-08-12 11:59:14,884 [myid:127.0.0.1:27386] - INFO [main-SendThread(127.0.0.1:27386):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:27386. Will not attempt to authenticate using SASL (unknown error) [junit] 2017-08-12 11:59:14,885 [myid:127.0.0.1:27386] - INFO [main-SendThread(127.0.0.1:27386):ClientCnxn$SendThread@946] - Socket connection established, initiating session, client: /127.0.0.1:39262, server: 127.0.0.1/127.0.0.1:27386 [junit] 2017-08-12 11:59:14,885 [myid:2] - INFO [NIOServerCxnFactory.AcceptThread:localhost/127.0.0.1:27386:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:39262 [junit] 2017-08-12 11:59:36,173 [myid:127.0.0.1:27386] - WARN [main-SendThread(127.0.0.1:27386):ClientCnxn$SendThread@1181] - Client session timed out, have not heard from server in 30030ms for sessionid 0x0 [junit] 2017-08-12 11:59:36,173 [myid:127.0.0.1:27386] - INFO [main-SendThread(127.0.0.1:27386):ClientCnxn$SendThread@1229] - Client session timed out, have not heard from server in 30030ms for sessionid 0x0, closing socket connection and attempting reconnect [junit] 2017-08-12 11:59:37,560 [myid:127.0.0.1:27386] - INFO [main-SendThread(127.0.0.1:27386):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:27386. Will not attempt to authenticate using SASL (unknown error) [junit] 2017-08-12 11:59:37,560 [myid:127.0.0.1:27386] - INFO [main-SendThread(127.0.0.1:27386):ClientCnxn$SendThread@946] - Socket connection established, initiating session, client: /127.0.0.1:39844, server: 127.0.0.1/127.0.0.1:27386 [junit] 2017-08-12 11:59:37,561 [myid:2] - INFO [NIOServerCxnFactory.AcceptThread:localhost/127.0.0.1:27386:NIOServerCnxnFactory$AcceptThread@296] - Accepted socket connection from /127.0.0.1:39844 [junit] 2017-08-12 11:59:44,913 [myid:127.0.0.1:27386] - WARN [main-SendThread(127.0.0.1:27386):ClientCnxn$SendThread@1181] - Client session timed out, have not heard from server in 30028ms for sessionid 0x2054fb559f8 [junit] 2017-08-12 11:59:44,913 [myid:127.0.0.1:27386] - INFO [main-SendThread(127.0.0.1:27386):ClientCnxn$SendThread@1229] - Client session timed out, have not heard from server in 30028ms for sessionid
ZooKeeper_branch35_openjdk7 - Build # 631 - Still Failing
See https://builds.apache.org/job/ZooKeeper_branch35_openjdk7/631/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 165.77 MB...] [junit] 2017-08-12 10:06:23,145 [myid:127.0.0.1:16734] - WARN [main-SendThread(127.0.0.1:16734):ClientCnxn$SendThread@1235] - Session 0x205ffc9743e for server 127.0.0.1/127.0.0.1:16734, unexpected error, closing socket connection and attempting reconnect [junit] java.net.ConnectException: Connection refused [junit] at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [junit] at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [junit] at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357) [junit] at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214) [junit] 2017-08-12 10:06:23,157 [myid:127.0.0.1:16608] - INFO [main-SendThread(127.0.0.1:16608):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16608. Will not attempt to authenticate using SASL (unknown error) [junit] 2017-08-12 10:06:23,158 [myid:127.0.0.1:16608] - WARN [main-SendThread(127.0.0.1:16608):ClientCnxn$SendThread@1235] - Session 0x105ffc6b3f7 for server 127.0.0.1/127.0.0.1:16608, unexpected error, closing socket connection and attempting reconnect [junit] java.net.ConnectException: Connection refused [junit] at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [junit] at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [junit] at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357) [junit] at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214) [junit] 2017-08-12 10:06:23,570 [myid:127.0.0.1:16657] - INFO [main-SendThread(127.0.0.1:16657):ClientCnxn$SendThread@1113] - Opening socket connection to server 127.0.0.1/127.0.0.1:16657. Will not attempt to authenticate using SASL (unknown error) [junit] 2017-08-12 10:06:23,571 [myid:127.0.0.1:16657] - WARN [main-SendThread(127.0.0.1:16657):ClientCnxn$SendThread@1235] - Session 0x105ffc7002c0001 for server 127.0.0.1/127.0.0.1:16657, unexpected error, closing socket connection and attempting reconnect [junit] java.net.ConnectException: Connection refused [junit] at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) [junit] at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) [junit] at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:357) [junit] at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1214) [junit] 2017-08-12 10:06:23,863 [myid:] - INFO [ProcessThread(sid:0 cport:16854)::PrepRequestProcessor@611] - Processed session termination for sessionid: 0x105ffccd38a [junit] 2017-08-12 10:06:23,865 [myid:] - INFO [SyncThread:0:MBeanRegistry@128] - Unregister MBean [org.apache.ZooKeeperService:name0=StandaloneServer_port16854,name1=Connections,name2=127.0.0.1,name3=0x105ffccd38a] [junit] 2017-08-12 10:06:23,865 [myid:] - INFO [main:ZooKeeper@1334] - Session: 0x105ffccd38a closed [junit] 2017-08-12 10:06:23,865 [myid:] - INFO [main-EventThread:ClientCnxn$EventThread@513] - EventThread shut down for session: 0x105ffccd38a [junit] 2017-08-12 10:06:23,865 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@82] - Memory used 220227 [junit] 2017-08-12 10:06:23,866 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@87] - Number of threads 2498 [junit] 2017-08-12 10:06:23,866 [myid:] - INFO [main:JUnit4ZKTestRunner$LoggedInvokeMethod@102] - FINISHED TEST METHOD testWatcherAutoResetWithLocal [junit] 2017-08-12 10:06:23,866 [myid:] - INFO [main:ClientBase@586] - tearDown starting [junit] 2017-08-12 10:06:23,866 [myid:] - INFO [main:ClientBase@556] - STOPPING server [junit] 2017-08-12 10:06:23,866 [myid:] - INFO [main:NettyServerCnxnFactory@464] - shutdown called 0.0.0.0/0.0.0.0:16854 [junit] 2017-08-12 10:06:23,874 [myid:] - INFO [main:ZooKeeperServer@541] - shutting down [junit] 2017-08-12 10:06:23,874 [myid:] - ERROR [main:ZooKeeperServer@505] - ZKShutdownHandler is not registered, so ZooKeeper server won't take any action on ERROR or SHUTDOWN server state changes [junit] 2017-08-12 10:06:23,874 [myid:] - INFO [main:SessionTrackerImpl@232] - Shutting down [junit] 2017-08-12 10:06:23,874 [myid:] - INFO [main:PrepRequestProcessor@1005] - Shutting down [junit] 2017-08-12 10:06:23,875 [myid:] - INFO [main:SyncRequestProcessor@191] - Shutting down [junit] 2017-08-12 10:06:23,875 [myid:] - INFO [ProcessThread(sid:0 cport:16854)::PrepRequestProcessor@155] - PrepRequestProcessor exited loop! [junit] 2017-08-12
ZooKeeper_branch35_jdk7 - Build # 1073 - Still Failing
See https://builds.apache.org/job/ZooKeeper_branch35_jdk7/1073/
###
## LAST 60 LINES OF THE CONSOLE
###
[...truncated 64.81 MB...]
[junit] 2017-08-12 08:52:01,198 [myid:] - WARN [New I/O boss
#5236:ClientCnxnSocketNetty$ZKClientHandler@439] - Exception caught: [id:
0x0a70d224] EXCEPTION: java.net.ConnectException: Connection refused:
127.0.0.1/127.0.0.1:30199
[junit] java.net.ConnectException: Connection refused:
127.0.0.1/127.0.0.1:30199
[junit] at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
[junit] at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
[junit] at
org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
[junit] at
org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
[junit] at
org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
[junit] at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
[junit] at
org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
[junit] at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
[junit] at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
[junit] at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit] at java.lang.Thread.run(Thread.java:745)
[junit] 2017-08-12 08:52:01,199 [myid:] - INFO [New I/O boss
#5236:ClientCnxnSocketNetty@208] - channel is told closing
[junit] 2017-08-12 08:52:01,199 [myid:127.0.0.1:30199] - INFO
[main-SendThread(127.0.0.1:30199):ClientCnxn$SendThread@1231] - channel for
sessionid 0x2054f129541 is lost, closing socket connection and attempting
reconnect
[junit] 2017-08-12 08:52:01,266 [myid:127.0.0.1:30202] - INFO
[main-SendThread(127.0.0.1:30202):ClientCnxn$SendThread@1113] - Opening socket
connection to server 127.0.0.1/127.0.0.1:30202. Will not attempt to
authenticate using SASL (unknown error)
[junit] 2017-08-12 08:52:01,267 [myid:] - INFO [New I/O boss
#5253:ClientCnxnSocketNetty$1@127] - future isn't success, cause: {}
[junit] java.net.ConnectException: Connection refused:
127.0.0.1/127.0.0.1:30202
[junit] at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
[junit] at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
[junit] at
org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
[junit] at
org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
[junit] at
org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
[junit] at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
[junit] at
org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
[junit] at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
[junit] at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
[junit] at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[junit] at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[junit] at java.lang.Thread.run(Thread.java:745)
[junit] 2017-08-12 08:52:01,274 [myid:] - WARN [New I/O boss
#5253:ClientCnxnSocketNetty$ZKClientHandler@439] - Exception caught: [id:
0x70ac23e5] EXCEPTION: java.net.ConnectException: Connection refused:
127.0.0.1/127.0.0.1:30202
[junit] java.net.ConnectException: Connection refused:
127.0.0.1/127.0.0.1:30202
[junit] at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
[junit] at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:744)
[junit] at
org.jboss.netty.channel.socket.nio.NioClientBoss.connect(NioClientBoss.java:152)
[junit] at
org.jboss.netty.channel.socket.nio.NioClientBoss.processSelectedKeys(NioClientBoss.java:105)
[junit] at
org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.java:79)
[junit] at
org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:337)
[junit] at
org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:42)
[junit] at
org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
[junit] at
org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
[junit] at
