Paulo, I can confirm that the problem is as you stated. Some or all of the other nodes are keeping a connection in CLOSE_WAIT state. Those nodes are seen as DN from the point of the node I have restarted the Cassandra service on. But nodetool disablegossip did not fix the problem.
This sounds like an issue that can potentially affect many users. Is it not the case? Do we have a solution for this? -------- Here is netstat and nodetool status output 1. right after stopping cassandra service on 10.4.68.222: -------------------------------------- ip-10-4-54-176 tcp 0 0 10.4.54.176:51268 10.4.68.222:7000 TIME_WAIT tcp 0 0 10.4.54.176:56135 10.4.68.222:7000 TIME_WAIT tcp 1 0 10.4.54.176:43697 10.4.68.222:7000 CLOSE_WAIT tcp 0 0 10.4.54.176:52372 10.4.68.222:7000 TIME_WAIT -------------------------------------- -------------------------------------- ip-10-4-54-177 tcp 0 0 10.4.54.177:56960 10.4.68.222:7000 TIME_WAIT tcp 0 0 10.4.54.177:54539 10.4.68.222:7000 TIME_WAIT tcp 0 0 10.4.54.177:32823 10.4.68.222:7000 TIME_WAIT tcp 1 0 10.4.54.177:48985 10.4.68.222:7000 CLOSE_WAIT -------------------------------------- -------------------------------------- ip-10-4-68-222 tcp 0 0 10.4.68.222:7000 10.4.54.176:43697 FIN_WAIT2 tcp 0 0 10.4.68.222:7000 10.4.54.177:48985 FIN_WAIT2 tcp 0 0 10.4.68.222:7000 10.4.68.222:54419 TIME_WAIT tcp 0 0 10.4.68.222:7000 10.4.43.65:43197 FIN_WAIT2 tcp 0 0 10.4.68.222:7000 10.4.68.221:44149 FIN_WAIT2 tcp 0 0 10.4.68.222:7000 10.4.68.222:41302 TIME_WAIT tcp 0 0 10.4.68.222:7000 10.4.43.66:54321 FIN_WAIT2 -------------------------------------- -------------------------------------- ip-10-4-68-221 tcp 0 0 10.4.68.221:49599 10.4.68.222:7000 TIME_WAIT tcp 0 0 10.4.68.221:55033 10.4.68.222:7000 TIME_WAIT tcp 0 0 10.4.68.221:51628 10.4.68.222:7000 TIME_WAIT tcp 1 0 10.4.68.221:44149 10.4.68.222:7000 CLOSE_WAIT -------------------------------------- -------------------------------------- ip-10-4-43-66 tcp 0 0 10.4.43.66:55930 10.4.68.222:7000 TIME_WAIT tcp 1 0 10.4.43.66:54321 10.4.68.222:7000 CLOSE_WAIT tcp 0 0 10.4.43.66:60968 10.4.68.222:7000 TIME_WAIT tcp 0 0 10.4.43.66:49087 10.4.68.222:7000 TIME_WAIT -------------------------------------- -------------------------------------- ip-10-4-43-65 tcp 1 0 10.4.43.65:43197 10.4.68.222:7000 CLOSE_WAIT tcp 0 0 10.4.43.65:36467 10.4.68.222:7000 TIME_WAIT tcp 0 0 10.4.43.65:53317 10.4.68.222:7000 TIME_WAIT tcp 0 0 10.4.43.65:54897 10.4.68.222:7000 TIME_WAIT -------------------------------------- 2. a bit after stopping cassandra service on 10.4.68.222: -------------------------------------- ip-10-4-54-176 tcp 1 0 10.4.54.176:43697 10.4.68.222:7000 CLOSE_WAIT -------------------------------------- -------------------------------------- ip-10-4-54-177 -------------------------------------- -------------------------------------- ip-10-4-68-222 -------------------------------------- -------------------------------------- ip-10-4-68-221 tcp 1 0 10.4.68.221:44149 10.4.68.222:7000 CLOSE_WAIT -------------------------------------- -------------------------------------- ip-10-4-43-66 tcp 1 0 10.4.43.66:54321 10.4.68.222:7000 CLOSE_WAIT -------------------------------------- -------------------------------------- ip-10-4-43-65 tcp 1 0 10.4.43.65:43197 10.4.68.222:7000 CLOSE_WAIT -------------------------------------- 3. after starting cassandra service on 10.4.68.222: -------------------------------------- ip-10-4-54-176 tcp 0 0 10.4.54.176:42460 10.4.68.222:7000 ESTABLISHED tcp 1 303403 10.4.54.176:43697 10.4.68.222:7000 CLOSE_WAIT tcp 0 0 10.4.54.176:42109 10.4.68.222:7000 ESTABLISHED -------------------------------------- -------------------------------------- ip-10-4-54-177 tcp 0 0 10.4.54.177:43687 10.4.68.222:7000 ESTABLISHED tcp 0 0 10.4.54.177:56107 10.4.68.222:7000 ESTABLISHED tcp 0 0 10.4.54.177:39426 10.4.68.222:7000 ESTABLISHED -------------------------------------- -------------------------------------- ip-10-4-68-222 tcp 0 0 10.4.68.222:7000 0.0.0.0:* LISTEN tcp 0 0 10.4.68.222:7000 10.4.54.176:42109 ESTABLISHED tcp 0 0 10.4.68.222:7000 10.4.54.177:43687 ESTABLISHED tcp 0 0 10.4.68.222:7000 10.4.54.176:42460 ESTABLISHED tcp 0 0 10.4.68.222:7000 10.4.43.66:55168 ESTABLISHED tcp 0 0 10.4.68.222:7000 10.4.43.65:60239 ESTABLISHED tcp 0 0 10.4.68.222:7000 10.4.54.177:39426 ESTABLISHED tcp 0 0 10.4.68.222:7000 10.4.43.65:43480 ESTABLISHED tcp 0 0 10.4.68.222:7000 10.4.68.221:54490 ESTABLISHED tcp 0 0 10.4.68.222:7000 10.4.68.221:59771 ESTABLISHED tcp 0 0 10.4.68.222:7000 10.4.54.177:56107 ESTABLISHED tcp 0 0 10.4.68.222:7000 10.4.43.66:55581 ESTABLISHED -------------------------------------- -------------------------------------- ip-10-4-68-221 tcp 0 0 10.4.68.221:54490 10.4.68.222:7000 ESTABLISHED tcp 0 0 10.4.68.221:59771 10.4.68.222:7000 ESTABLISHED tcp 1 304316 10.4.68.221:44149 10.4.68.222:7000 CLOSE_WAIT -------------------------------------- -------------------------------------- ip-10-4-43-66 tcp 1 322344 10.4.43.66:54321 10.4.68.222:7000 CLOSE_WAIT tcp 0 0 10.4.43.66:55581 10.4.68.222:7000 ESTABLISHED tcp 0 0 10.4.43.66:55168 10.4.68.222:7000 ESTABLISHED -------------------------------------- -------------------------------------- ip-10-4-43-65 tcp 1 376331 10.4.43.65:43197 10.4.68.222:7000 CLOSE_WAIT tcp 0 0 10.4.43.65:43480 10.4.68.222:7000 ESTABLISHED tcp 0 0 10.4.43.65:60239 10.4.68.222:7000 ESTABLISHED -------------------------------------- 4. nodetool status on all nodes after starting cassandra service on 10.4.68.222: ip-10-4-54-176 Datacenter: us-east =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 10.4.54.176 127.67 GB 256 47.5% 7163bf77-2fef-4e33-81c1-0e61038dece1 1b UN 10.4.43.65 124.19 GB 256 46.2% 80265afb-8beb-4887-a696-fc9b75956894 1a UN 10.4.54.177 136.06 GB 256 50.7% b9010e24-4e92-4212-8a17-65892ea9ff66 1b UN 10.4.43.66 141.94 GB 256 52.3% b00fdf10-1075-4953-8a96-caf375221684 1a UN 10.4.68.221 137.12 GB 256 50.7% 37479ec3-7b6d-4537-975c-f9d95e92ee1d 1d UN 10.4.68.222 141.89 GB 256 52.7% 8df87657-c39b-405a-ba54-d60b577c1429 1d -------------------------------------- -------------------------------------- ip-10-4-54-177 Datacenter: us-east =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 10.4.54.176 127.63 GB 256 47.5% 7163bf77-2fef-4e33-81c1-0e61038dece1 1b UN 10.4.43.65 124.19 GB 256 46.2% 80265afb-8beb-4887-a696-fc9b75956894 1a UN 10.4.54.177 136.06 GB 256 50.7% b9010e24-4e92-4212-8a17-65892ea9ff66 1b UN 10.4.43.66 141.94 GB 256 52.3% b00fdf10-1075-4953-8a96-caf375221684 1a UN 10.4.68.221 137.12 GB 256 50.7% 37479ec3-7b6d-4537-975c-f9d95e92ee1d 1d UN 10.4.68.222 141.89 GB 256 52.7% 8df87657-c39b-405a-ba54-d60b577c1429 1d ------------------------ -------------------------------------- ip-10-4-68-222 Datacenter: us-east =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack DN 10.4.54.176 127.63 GB 256 47.5% 7163bf77-2fef-4e33-81c1-0e61038dece1 1b DN 10.4.43.65 124.19 GB 256 46.2% 80265afb-8beb-4887-a696-fc9b75956894 1a UN 10.4.54.177 136.06 GB 256 50.7% b9010e24-4e92-4212-8a17-65892ea9ff66 1b DN 10.4.43.66 141.94 GB 256 52.3% b00fdf10-1075-4953-8a96-caf375221684 1a DN 10.4.68.221 137.12 GB 256 50.7% 37479ec3-7b6d-4537-975c-f9d95e92ee1d 1d UN 10.4.68.222 141.89 GB 256 52.7% 8df87657-c39b-405a-ba54-d60b577c1429 1d -------------------------------------- -------------------------------------- ip-10-4-68-221 Datacenter: us-east =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 10.4.54.176 127.63 GB 256 47.5% 7163bf77-2fef-4e33-81c1-0e61038dece1 1b UN 10.4.43.65 124.19 GB 256 46.2% 80265afb-8beb-4887-a696-fc9b75956894 1a UN 10.4.54.177 136.06 GB 256 50.7% b9010e24-4e92-4212-8a17-65892ea9ff66 1b UN 10.4.43.66 141.94 GB 256 52.3% b00fdf10-1075-4953-8a96-caf375221684 1a UN 10.4.68.221 137.12 GB 256 50.7% 37479ec3-7b6d-4537-975c-f9d95e92ee1d 1d UN 10.4.68.222 141.89 GB 256 52.7% 8df87657-c39b-405a-ba54-d60b577c1429 1d -------------------------------------- -------------------------------------- ip-10-4-43-66 Datacenter: us-east =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 10.4.54.176 127.67 GB 256 47.5% 7163bf77-2fef-4e33-81c1-0e61038dece1 1b UN 10.4.43.65 124.19 GB 256 46.2% 80265afb-8beb-4887-a696-fc9b75956894 1a UN 10.4.54.177 136.06 GB 256 50.7% b9010e24-4e92-4212-8a17-65892ea9ff66 1b UN 10.4.43.66 141.95 GB 256 52.3% b00fdf10-1075-4953-8a96-caf375221684 1a UN 10.4.68.221 137.12 GB 256 50.7% 37479ec3-7b6d-4537-975c-f9d95e92ee1d 1d UN 10.4.68.222 141.89 GB 256 52.7% 8df87657-c39b-405a-ba54-d60b577c1429 1d -------------------------------------- -------------------------------------- ip-10-4-43-65 Datacenter: us-east =================== Status=Up/Down |/ State=Normal/Leaving/Joining/Moving -- Address Load Tokens Owns (effective) Host ID Rack UN 10.4.54.176 127.67 GB 256 47.5% 7163bf77-2fef-4e33-81c1-0e61038dece1 1b UN 10.4.43.65 124.19 GB 256 46.2% 80265afb-8beb-4887-a696-fc9b75956894 1a UN 10.4.54.177 136.06 GB 256 50.7% b9010e24-4e92-4212-8a17-65892ea9ff66 1b UN 10.4.43.66 141.94 GB 256 52.3% b00fdf10-1075-4953-8a96-caf375221684 1a UN 10.4.68.221 137.12 GB 256 50.7% 37479ec3-7b6d-4537-975c-f9d95e92ee1d 1d UN 10.4.68.222 141.89 GB 256 52.7% 8df87657-c39b-405a-ba54-d60b577c1429 1d -------------------------------------- On Wed, Jul 27, 2016 at 2:04 PM, Farzad Panahi <farzad.pan...@gmail.com> wrote: > Thanks Paulo for the reply. > > Cassandra version is 3.0.8. I will test what you said and share the > results. > > On Wed, Jul 27, 2016 at 2:01 PM, Paulo Motta <pauloricard...@gmail.com> > wrote: > >> This looks somewhat related to CASSANDRA-9630. What is the C* version? >> >> Can you check with netstats if other nodes keep connections with the >> stopped node in the CLOSE_WAIT state? And also if the problem disappears if >> you run nodetool disablegossip before stopping the node? >> >> 2016-07-26 16:54 GMT-03:00 Farzad Panahi <farzad.pan...@gmail.com>: >> >>> I am new to Cassandra and trying to figure out how the cluster behaves >>> when things go south. >>> >>> I have a 6-node cluster, RF=3. >>> >>> I stop Cassandra service on a node for a while. All nodes see the node >>> as DN. After a while I start the Cassandra service on DN. Interesting point >>> is that all other nodes see the node now as UN but the node itself sees 4 >>> nodes as DN and only one node as UN. After about 10 minutes the node sees >>> other nodes as up as well. >>> >>> I am trying to figure out where this delay is coming from. >>> >>> I have attached part of system.log that looks interesting. Looks like >>> after Gossiper logs InetAddress <ip> is now UP the node is actually seeing >>> that node as up even though the node has already handshaked with that node >>> before. >>> >>> Any ideas? >>> >>> Cheers >>> >>> Farzad >>> >>> -------------------------------------- >>> INFO [main] 2016-07-25 21:58:46,044 StorageService.java:533 - Cassandra >>> version: 3.0.8 >>> INFO [main] 2016-07-25 21:58:46,098 StorageService.java:534 - Thrift >>> API version: 20.1.0 >>> INFO [main] 2016-07-25 21:58:46,150 StorageService.java:535 - CQL >>> supported versions: 3.4.0 (default: 3.4.0) >>> INFO [main] 2016-07-25 21:58:46,284 IndexSummaryManager.java:85 - >>> Initializing index summary manager with a memory pool size of 198 MB and a >>> resize interval of 60 minutes >>> INFO [main] 2016-07-25 21:58:46,343 StorageService.java:554 - Loading >>> persisted ring state >>> INFO [main] 2016-07-25 21:58:46,418 StorageService.java:743 - Starting >>> up server gossip >>> INFO [main] 2016-07-25 21:58:46,680 TokenMetadata.java:429 - Updating >>> topology for ip-10-4-43-66.ec2.internal/10.4.43.66 >>> INFO [main] 2016-07-25 21:58:46,707 TokenMetadata.java:429 - Updating >>> topology for ip-10-4-43-66.ec2.internal/10.4.43.66 >>> INFO [main] 2016-07-25 21:58:46,792 MessagingService.java:557 - >>> Starting Messaging Service on ip-10-4-43-66.ec2.internal/10.4.43.66:7000 >>> (eth0) >>> >>> INFO [HANDSHAKE-/10.4.68.222] 2016-07-25 21:58:46,920 >>> OutboundTcpConnection.java:515 - Handshaking version with /10.4.68.222 >>> INFO [GossipStage:1] 2016-07-25 21:58:47,011 Gossiper.java:1028 - Node / >>> 10.4.68.221 has restarted, now UP >>> INFO [HANDSHAKE-/10.4.68.222] 2016-07-25 21:58:47,007 >>> OutboundTcpConnection.java:515 - Handshaking version with /10.4.68.222 >>> INFO [main] 2016-07-25 21:58:47,030 StorageService.java:1902 - Node >>> ip-10-4-43-66.ec2.internal/10.4.43.66 state jump to NORMAL >>> INFO [main] 2016-07-25 21:58:47,096 CassandraDaemon.java:644 - Waiting >>> for gossip to settle before accepting client requests... >>> INFO [GossipStage:1] 2016-07-25 21:58:47,134 StorageService.java:1902 - >>> Node /10.4.68.221 state jump to NORMAL >>> INFO [HANDSHAKE-/10.4.68.221] 2016-07-25 21:58:47,137 >>> OutboundTcpConnection.java:515 - Handshaking version with /10.4.68.221 >>> INFO [GossipStage:1] 2016-07-25 21:58:47,211 TokenMetadata.java:429 - >>> Updating topology for /10.4.68.221 >>> INFO [GossipStage:1] 2016-07-25 21:58:47,261 TokenMetadata.java:429 - >>> Updating topology for /10.4.68.221 >>> INFO [GossipStage:1] 2016-07-25 21:58:47,295 Gossiper.java:1028 - Node / >>> 10.4.68.222 has restarted, now UP >>> INFO [GossipStage:1] 2016-07-25 21:58:47,337 StorageService.java:1902 - >>> Node /10.4.68.222 state jump to NORMAL >>> INFO [GossipStage:1] 2016-07-25 21:58:47,385 TokenMetadata.java:429 - >>> Updating topology for /10.4.68.222 >>> INFO [GossipStage:1] 2016-07-25 21:58:47,452 TokenMetadata.java:429 - >>> Updating topology for /10.4.68.222 >>> INFO [GossipStage:1] 2016-07-25 21:58:47,497 Gossiper.java:1028 - Node / >>> 10.4.54.176 has restarted, now UP >>> INFO [GossipStage:1] 2016-07-25 21:58:47,544 StorageService.java:1902 - >>> Node /10.4.54.176 state jump to NORMAL >>> INFO [HANDSHAKE-/10.4.54.176] 2016-07-25 21:58:47,548 >>> OutboundTcpConnection.java:515 - Handshaking version with /10.4.54.176 >>> INFO [GossipStage:1] 2016-07-25 21:58:47,594 TokenMetadata.java:429 - >>> Updating topology for /10.4.54.176 >>> INFO [GossipStage:1] 2016-07-25 21:58:47,639 TokenMetadata.java:429 - >>> Updating topology for /10.4.54.176 >>> WARN [GossipTasks:1] 2016-07-25 21:58:47,678 FailureDetector.java:287 - >>> Not marking nodes down due to local pause of 43226235115 > 5000000000 >>> INFO [HANDSHAKE-/10.4.43.65] 2016-07-25 21:58:47,679 >>> OutboundTcpConnection.java:515 - Handshaking version with /10.4.43.65 >>> INFO [GossipStage:1] 2016-07-25 21:58:47,757 Gossiper.java:1028 - Node / >>> 10.4.54.177 has restarted, now UP >>> INFO [GossipStage:1] 2016-07-25 21:58:47,788 StorageService.java:1902 - >>> Node /10.4.54.177 state jump to NORMAL >>> INFO [HANDSHAKE-/10.4.54.177] 2016-07-25 21:58:47,789 >>> OutboundTcpConnection.java:515 - Handshaking version with /10.4.54.177 >>> INFO [GossipStage:1] 2016-07-25 21:58:47,836 TokenMetadata.java:429 - >>> Updating topology for /10.4.54.177 >>> INFO [GossipStage:1] 2016-07-25 21:58:47,887 TokenMetadata.java:429 - >>> Updating topology for /10.4.54.177 >>> INFO [GossipStage:1] 2016-07-25 21:58:47,926 Gossiper.java:1028 - Node / >>> 10.4.43.65 has restarted, now UP >>> INFO [GossipStage:1] 2016-07-25 21:58:47,976 StorageService.java:1902 - >>> Node /10.4.43.65 state jump to NORMAL >>> INFO [GossipStage:1] 2016-07-25 21:58:48,036 TokenMetadata.java:429 - >>> Updating topology for /10.4.43.65 >>> INFO [GossipStage:1] 2016-07-25 21:58:48,076 TokenMetadata.java:429 - >>> Updating topology for /10.4.43.65 >>> INFO [GossipStage:1] 2016-07-25 21:58:48,122 StorageService.java:1902 - >>> Node /10.4.43.65 state jump to NORMAL >>> INFO [HANDSHAKE-/10.4.54.176] 2016-07-25 21:58:48,123 >>> OutboundTcpConnection.java:515 - Handshaking version with /10.4.54.176 >>> INFO [HANDSHAKE-/10.4.68.221] 2016-07-25 21:58:48,123 >>> OutboundTcpConnection.java:515 - Handshaking version with /10.4.68.221 >>> INFO [GossipStage:1] 2016-07-25 21:58:48,165 TokenMetadata.java:429 - >>> Updating topology for /10.4.43.65 >>> INFO [SharedPool-Worker-1] 2016-07-25 21:58:48,170 Gossiper.java:994 - >>> InetAddress /10.4.54.177 is now UP >>> INFO [GossipStage:1] 2016-07-25 21:58:48,198 TokenMetadata.java:429 - >>> Updating topology for /10.4.43.65 >>> INFO [GossipStage:1] 2016-07-25 21:58:48,278 StorageService.java:1902 - >>> Node /10.4.54.177 state jump to NORMAL >>> INFO [GossipStage:1] 2016-07-25 21:58:48,345 TokenMetadata.java:429 - >>> Updating topology for /10.4.54.177 >>> INFO [GossipStage:1] 2016-07-25 21:58:48,388 TokenMetadata.java:429 - >>> Updating topology for /10.4.54.177 >>> INFO [HANDSHAKE-/10.4.43.65] 2016-07-25 21:58:48,444 >>> OutboundTcpConnection.java:515 - Handshaking version with /10.4.43.65 >>> INFO [HANDSHAKE-/10.4.54.177] 2016-07-25 21:58:48,444 >>> OutboundTcpConnection.java:515 - Handshaking version with /10.4.54.177 >>> INFO [main] 2016-07-25 21:58:55,155 CassandraDaemon.java:675 - No >>> gossip backlog; proceeding >>> INFO [main] 2016-07-25 21:58:55,436 NativeTransportService.java:70 - >>> Netty using native Epoll event loop >>> INFO [main] 2016-07-25 21:58:55,594 Server.java:159 - Using Netty >>> Version: [netty-buffer=netty-buffer-4.0.23.Final.208198c, >>> netty-codec=netty-codec-4.0.23.Final.208198c, >>> netty-codec-http=netty-codec-http-4.0.23.Final.208198c, >>> netty-codec-socks=netty-codec-socks-4.0.23.Final.208198c, >>> netty-common=netty-common-4.0.23.Final.208198c, >>> netty-handler=netty-handler-4.0.23.Final.208198c, >>> netty-transport=netty-transport-4.0.23.Final.208198c, >>> netty-transport-rxtx=netty-transport-rxtx-4.0.23.Final.208198c, >>> netty-transport-sctp=netty-transport-sctp-4.0.23.Final.208198c, >>> netty-transport-udt=netty-transport-udt-4.0.23.Final.208198c] >>> INFO [main] 2016-07-25 21:58:55,691 Server.java:160 - Starting >>> listening for CQL clients on /0.0.0.0:9042 (unencrypted)... >>> INFO [main] 2016-07-25 21:58:55,842 ThriftServer.java:119 - Binding >>> thrift service to /0.0.0.0:9160 >>> INFO [Thread-3] 2016-07-25 21:58:55,896 ThriftServer.java:136 - >>> Listening for thrift clients... >>> INFO [HANDSHAKE-/10.4.54.177] 2016-07-25 21:59:24,753 >>> OutboundTcpConnection.java:515 - Handshaking version with /10.4.54.177 >>> INFO [HANDSHAKE-ip-10-4-43-66.ec2.internal/10.4.43.66] 2016-07-25 >>> 21:59:33,375 OutboundTcpConnection.java:515 - Handshaking version with >>> ip-10-4-43-66.ec2.internal/10.4.43.66 >>> INFO [SharedPool-Worker-4] 2016-07-25 22:14:19,273 Gossiper.java:994 - >>> InetAddress /10.4.43.65 is now UP >>> INFO [SharedPool-Worker-2] 2016-07-25 22:14:19,274 Gossiper.java:994 - >>> InetAddress /10.4.43.65 is now UP >>> INFO [SharedPool-Worker-7] 2016-07-25 22:14:19,307 Gossiper.java:994 - >>> InetAddress /10.4.43.65 is now UP >>> INFO [SharedPool-Worker-36] 2016-07-25 22:14:19,313 Gossiper.java:994 - >>> InetAddress /10.4.43.65 is now UP >>> INFO [SharedPool-Worker-4] 2016-07-25 22:14:19,347 Gossiper.java:994 - >>> InetAddress /10.4.43.65 is now UP >>> INFO [SharedPool-Worker-2] 2016-07-25 22:14:19,356 Gossiper.java:994 - >>> InetAddress /10.4.43.65 is now UP >>> INFO [SharedPool-Worker-7] 2016-07-25 22:14:19,369 Gossiper.java:994 - >>> InetAddress /10.4.43.65 is now UP >>> INFO [SharedPool-Worker-36] 2016-07-25 22:14:19,372 Gossiper.java:994 - >>> InetAddress /10.4.43.65 is now UP >>> INFO [SharedPool-Worker-2] 2016-07-25 22:14:19,426 Gossiper.java:994 - >>> InetAddress /10.4.43.65 is now UP >>> INFO [SharedPool-Worker-36] 2016-07-25 22:14:19,426 Gossiper.java:994 - >>> InetAddress /10.4.43.65 is now UP >>> INFO [SharedPool-Worker-4] 2016-07-25 22:14:19,434 Gossiper.java:994 - >>> InetAddress /10.4.43.65 is now UP >>> INFO [SharedPool-Worker-7] 2016-07-25 22:14:19,442 Gossiper.java:994 - >>> InetAddress /10.4.43.65 is now UP >>> INFO [SharedPool-Worker-4] 2016-07-25 22:14:19,486 Gossiper.java:994 - >>> InetAddress /10.4.43.65 is now UP >>> INFO [SharedPool-Worker-2] 2016-07-25 22:14:19,500 Gossiper.java:994 - >>> InetAddress /10.4.43.65 is now UP >>> INFO [SharedPool-Worker-36] 2016-07-25 22:14:19,508 Gossiper.java:994 - >>> InetAddress /10.4.43.65 is now UP >>> INFO [SharedPool-Worker-7] 2016-07-25 22:14:19,514 Gossiper.java:994 - >>> InetAddress /10.4.43.65 is now UP >>> INFO [SharedPool-Worker-1] 2016-07-25 22:14:19,687 Gossiper.java:994 - >>> InetAddress /10.4.54.176 is now UP >>> INFO [SharedPool-Worker-5] 2016-07-25 22:14:19,699 Gossiper.java:994 - >>> InetAddress /10.4.54.176 is now UP >>> INFO [SharedPool-Worker-33] 2016-07-25 22:14:19,719 Gossiper.java:994 - >>> InetAddress /10.4.54.176 is now UP >>> INFO [SharedPool-Worker-7] 2016-07-25 22:14:19,721 Gossiper.java:994 - >>> InetAddress /10.4.54.176 is now UP >>> INFO [SharedPool-Worker-5] 2016-07-25 22:14:19,756 Gossiper.java:994 - >>> InetAddress /10.4.54.176 is now UP >>> INFO [SharedPool-Worker-1] 2016-07-25 22:14:19,757 Gossiper.java:994 - >>> InetAddress /10.4.54.176 is now UP >>> INFO [SharedPool-Worker-33] 2016-07-25 22:14:19,762 Gossiper.java:994 - >>> InetAddress /10.4.54.176 is now UP >>> INFO [SharedPool-Worker-7] 2016-07-25 22:14:19,778 Gossiper.java:994 - >>> InetAddress /10.4.54.176 is now UP >>> INFO [SharedPool-Worker-1] 2016-07-25 22:14:19,809 Gossiper.java:994 - >>> InetAddress /10.4.54.176 is now UP >>> INFO [SharedPool-Worker-5] 2016-07-25 22:14:19,814 Gossiper.java:994 - >>> InetAddress /10.4.54.176 is now UP >>> INFO [SharedPool-Worker-7] 2016-07-25 22:14:19,819 Gossiper.java:994 - >>> InetAddress /10.4.54.176 is now UP >>> INFO [SharedPool-Worker-33] 2016-07-25 22:14:19,822 Gossiper.java:994 - >>> InetAddress /10.4.54.176 is now UP >>> INFO [SharedPool-Worker-1] 2016-07-25 22:14:19,872 Gossiper.java:994 - >>> InetAddress /10.4.54.176 is now UP >>> INFO [SharedPool-Worker-5] 2016-07-25 22:14:19,874 Gossiper.java:994 - >>> InetAddress /10.4.54.176 is now UP >>> INFO [SharedPool-Worker-7] 2016-07-25 22:14:19,879 Gossiper.java:994 - >>> InetAddress /10.4.54.176 is now UP >>> INFO [SharedPool-Worker-33] 2016-07-25 22:14:22,489 Gossiper.java:994 - >>> InetAddress /10.4.68.222 is now UP >>> INFO [SharedPool-Worker-36] 2016-07-25 22:14:22,491 Gossiper.java:994 - >>> InetAddress /10.4.68.222 is now UP >>> INFO [SharedPool-Worker-34] 2016-07-25 22:14:22,491 Gossiper.java:994 - >>> InetAddress /10.4.68.222 is now UP >>> INFO [SharedPool-Worker-35] 2016-07-25 22:14:22,491 Gossiper.java:994 - >>> InetAddress /10.4.68.222 is now UP >>> INFO [SharedPool-Worker-36] 2016-07-25 22:14:22,533 Gossiper.java:994 - >>> InetAddress /10.4.68.222 is now UP >>> INFO [SharedPool-Worker-34] 2016-07-25 22:14:22,549 Gossiper.java:994 - >>> InetAddress /10.4.68.222 is now UP >>> INFO [SharedPool-Worker-33] 2016-07-25 22:14:22,570 Gossiper.java:994 - >>> InetAddress /10.4.68.222 is now UP >>> INFO [SharedPool-Worker-35] 2016-07-25 22:14:22,574 Gossiper.java:994 - >>> InetAddress /10.4.68.222 is now UP >>> INFO [SharedPool-Worker-36] 2016-07-25 22:14:22,637 Gossiper.java:994 - >>> InetAddress /10.4.68.222 is now UP >>> INFO [SharedPool-Worker-34] 2016-07-25 22:14:22,655 Gossiper.java:994 - >>> InetAddress /10.4.68.222 is now UP >>> INFO [SharedPool-Worker-33] 2016-07-25 22:14:22,672 Gossiper.java:994 - >>> InetAddress /10.4.68.222 is now UP >>> INFO [SharedPool-Worker-35] 2016-07-25 22:14:22,685 Gossiper.java:994 - >>> InetAddress /10.4.68.222 is now UP >>> INFO [SharedPool-Worker-36] 2016-07-25 22:14:22,703 Gossiper.java:994 - >>> InetAddress /10.4.68.222 is now UP >>> INFO [SharedPool-Worker-33] 2016-07-25 22:14:22,710 Gossiper.java:994 - >>> InetAddress /10.4.68.222 is now UP >>> INFO [SharedPool-Worker-34] 2016-07-25 22:14:22,725 Gossiper.java:994 - >>> InetAddress /10.4.68.222 is now UP >>> INFO [SharedPool-Worker-35] 2016-07-25 22:14:22,740 Gossiper.java:994 - >>> InetAddress /10.4.68.221 is now UP >>> INFO [SharedPool-Worker-36] 2016-07-25 22:14:22,748 Gossiper.java:994 - >>> InetAddress /10.4.68.221 is now UP >>> INFO [SharedPool-Worker-33] 2016-07-25 22:14:22,758 Gossiper.java:994 - >>> InetAddress /10.4.68.221 is now UP >>> INFO [SharedPool-Worker-34] 2016-07-25 22:14:22,764 Gossiper.java:994 - >>> InetAddress /10.4.68.221 is now UP >>> INFO [SharedPool-Worker-36] 2016-07-25 22:14:22,788 Gossiper.java:994 - >>> InetAddress /10.4.68.221 is now UP >>> INFO [SharedPool-Worker-35] 2016-07-25 22:14:22,806 Gossiper.java:994 - >>> InetAddress /10.4.68.221 is now UP >>> INFO [SharedPool-Worker-33] 2016-07-25 22:14:22,808 Gossiper.java:994 - >>> InetAddress /10.4.68.221 is now UP >>> INFO [SharedPool-Worker-34] 2016-07-25 22:14:22,836 Gossiper.java:994 - >>> InetAddress /10.4.68.221 is now UP >>> INFO [SharedPool-Worker-36] 2016-07-25 22:14:22,843 Gossiper.java:994 - >>> InetAddress /10.4.68.221 is now UP >>> INFO [SharedPool-Worker-33] 2016-07-25 22:14:22,849 Gossiper.java:994 - >>> InetAddress /10.4.68.221 is now UP >>> INFO [SharedPool-Worker-35] 2016-07-25 22:14:22,862 Gossiper.java:994 - >>> InetAddress /10.4.68.221 is now UP >>> INFO [SharedPool-Worker-33] 2016-07-25 22:14:22,892 Gossiper.java:994 - >>> InetAddress /10.4.68.221 is now UP >>> INFO [SharedPool-Worker-34] 2016-07-25 22:14:22,900 Gossiper.java:994 - >>> InetAddress /10.4.68.221 is now UP >>> INFO [SharedPool-Worker-36] 2016-07-25 22:14:22,904 Gossiper.java:994 - >>> InetAddress /10.4.68.221 is now UP >>> INFO [SharedPool-Worker-35] 2016-07-25 22:14:22,944 Gossiper.java:994 - >>> InetAddress /10.4.68.221 is now UP >>> INFO [SharedPool-Worker-33] 2016-07-25 22:14:22,945 Gossiper.java:994 - >>> InetAddress /10.4.68.221 is now UP >>> >>> >> >> >