[ https://issues.apache.org/jira/browse/HBASE-19753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16321345#comment-16321345 ]
stack commented on HBASE-19753: ------------------------------- Thanks for taking a look [~Apache9] Here is info. Here is failure: [INFO] ------------------------------------------------------- [INFO] T E S T S [INFO] ------------------------------------------------------- [INFO] Running org.apache.hadoop.hbase.zookeeper.TestReadOnlyZKClient [ERROR] Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 250.187 s <<< FAILURE! - in org.apache.hadoop.hbase.zookeeper.TestReadOnlyZKClient [ERROR] testSessionExpire(org.apache.hadoop.hbase.zookeeper.TestReadOnlyZKClient) Time elapsed: 156.005 s <<< ERROR! java.util.concurrent.ExecutionException: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /test at org.apache.hadoop.hbase.zookeeper.TestReadOnlyZKClient.testSessionExpire(TestReadOnlyZKClient.java:145) Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /test {code} 119 2018-01-10 15:22:41,604 INFO [main] hbase.ResourceChecker(148): before: zookeeper.TestReadOnlyZKClient#testSessionExpire Thread=13, OpenFileDescriptor=112, MaxFileDescriptor=32000, SystemLoadAverage=67, ProcessCount=833, AvailableMemoryMB=12065 120 2018-01-10 15:22:41,605 INFO [ReadOnlyZKClient] zookeeper.ZooKeeper(438): Initiating client connection, connectString=localhost:51573 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$11/1499902826@57a14636 121 2018-01-10 15:22:41,607 INFO [ReadOnlyZKClient-SendThread(user-164-127-0-0.play-internet.pl:51573)] zookeeper.ClientCnxn$SendThread(1032): Opening socket connection to server user-164-127-0-0.play-internet.pl/164.127.0.0:51573. Will not attempt to authenticate using SASL (unknown error) 122 2018-01-10 15:22:44,606 INFO [ReadOnlyZKClient] zookeeper.ReadOnlyZKClient(305): 0x57f23557 no activities for 3000 ms, close active connection. Will reconnect next time when there are new requests. 123 2018-01-10 15:22:44,606 DEBUG [ReadOnlyZKClient] zookeeper.ZooKeeper(673): Closing session: 0x0 124 2018-01-10 15:22:44,606 DEBUG [ReadOnlyZKClient] zookeeper.ClientCnxn(1370): Closing client for session: 0x0 125 2018-01-10 15:23:26,648 WARN [ReadOnlyZKClient-SendThread(user-164-127-0-0.play-internet.pl:51573)] zookeeper.ClientCnxn$SendThread(1108): Client session timed out, have not heard from server in 45043ms for sessionid 0x0 126 2018-01-10 15:23:26,648 DEBUG [ReadOnlyZKClient-SendThread(user-164-127-0-0.play-internet.pl:51573)] zookeeper.ClientCnxn$SendThread(1146): An exception was thrown while closing send thread for session 0x0 : Client session timed out, have not heard from server in 45043ms for sessionid 0x0 127 2018-01-10 15:23:26,649 DEBUG [ReadOnlyZKClient-SendThread(user-164-127-0-0.play-internet.pl:51573)] zookeeper.ClientCnxnSocketNIO(203): Ignoring exception during shutdown input 128 java.net.SocketException: Socket is not connected 129 at sun.nio.ch.Net.translateToSocketException(Net.java:123) 130 at sun.nio.ch.Net.translateException(Net.java:157) 131 at sun.nio.ch.Net.translateException(Net.java:163) 132 at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:401) 133 at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:200) 134 at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1246) 135 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1182) 136 Caused by: java.nio.channels.NotYetConnectedException 137 at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:782) 138 at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:399) 139 ... 3 more 140 2018-01-10 15:23:26,653 DEBUG [ReadOnlyZKClient-SendThread(user-164-127-0-0.play-internet.pl:51573)] zookeeper.ClientCnxnSocketNIO(210): Ignoring exception during shutdown output 141 java.net.SocketException: Socket is not connected 142 at sun.nio.ch.Net.translateToSocketException(Net.java:123) 143 at sun.nio.ch.Net.translateException(Net.java:157) 144 at sun.nio.ch.Net.translateException(Net.java:163) 145 at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:409) 146 at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:207) 147 at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1246) 148 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1182) 149 Caused by: java.nio.channels.NotYetConnectedException 150 at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:799) 151 at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:407) 152 ... 3 more 153 2018-01-10 15:23:26,754 DEBUG [ReadOnlyZKClient] zookeeper.ClientCnxn(1354): Disconnecting client for session: 0x0 154 2018-01-10 15:23:26,755 INFO [ReadOnlyZKClient] zookeeper.ZooKeeper(684): Session: 0x0 closed 155 2018-01-10 15:23:26,755 INFO [ReadOnlyZKClient-EventThread] zookeeper.ClientCnxn$EventThread(519): EventThread shut down for session: 0x0 156 2018-01-10 15:23:26,755 WARN [ReadOnlyZKClient] zookeeper.ReadOnlyZKClient$ZKTask$1(178): 0x57f23557 failed for get of /test, code = CONNECTIONLOSS, retries = 1 157 2018-01-10 15:23:26,855 INFO [ReadOnlyZKClient] zookeeper.ZooKeeper(438): Initiating client connection, connectString=localhost:51573 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$11/1499902826@57a14636 158 2018-01-10 15:23:26,857 INFO [ReadOnlyZKClient-SendThread(localhost:51573)] zookeeper.ClientCnxn$SendThread(1032): Opening socket connection to server localhost/127.0.0.1:51573. Will not attempt to authenticate using SASL (unknown error) 159 2018-01-10 15:23:26,858 INFO [ReadOnlyZKClient-SendThread(localhost:51573)] zookeeper.ClientCnxn$SendThread(876): Socket connection established to localhost/127.0.0.1:51573, initiating session 160 2018-01-10 15:23:26,858 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:51573] server.NIOServerCnxnFactory(192): Accepted socket connection from /127.0.0.1:34294 161 2018-01-10 15:23:26,858 DEBUG [ReadOnlyZKClient-SendThread(localhost:51573)] zookeeper.ClientCnxn$SendThread(949): Session establishment request sent on localhost/127.0.0.1:51573 162 2018-01-10 15:23:26,859 DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:51573] server.ZooKeeperServer(884): Session establishment request from client /127.0.0.1:34294 client's lastZxid is 0x0 163 2018-01-10 15:23:26,859 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:51573] server.ZooKeeperServer(942): Client attempting to establish new session at /127.0.0.1:34294 164 2018-01-10 15:23:26,860 DEBUG [SyncThread:0] server.FinalRequestProcessor(88): Processing request:: sessionid:0x160e261f37a0002 type:createSession cxid:0x0 zxid:0xb txntype:-10 reqpath:n/a 165 2018-01-10 15:23:26,861 DEBUG [SyncThread:0] server.FinalRequestProcessor(160): sessionid:0x160e261f37a0002 type:createSession cxid:0x0 zxid:0xb txntype:-10 reqpath:n/a 166 2018-01-10 15:23:26,862 INFO [SyncThread:0] server.ZooKeeperServer(687): Established session 0x160e261f37a0002 with negotiated timeout 40000 for client /127.0.0.1:34294 167 2018-01-10 15:23:26,862 INFO [ReadOnlyZKClient-SendThread(localhost:51573)] zookeeper.ClientCnxn$SendThread(1299): Session establishment complete on server localhost/127.0.0.1:51573, sessionid = 0x160e261f37a0002, negotiated timeout = 40000 168 2018-01-10 15:23:26,863 DEBUG [SyncThread:0] server.FinalRequestProcessor(88): Processing request:: sessionid:0x160e261f37a0002 type:getData cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:/test 169 2018-01-10 15:23:26,863 DEBUG [SyncThread:0] server.FinalRequestProcessor(160): sessionid:0x160e261f37a0002 type:getData cxid:0x1 zxid:0xfffffffffffffffe txntype:unknown reqpath:/test 170 2018-01-10 15:23:26,865 DEBUG [ReadOnlyZKClient-SendThread(localhost:51573)] zookeeper.ClientCnxn$SendThread(843): Reading reply sessionid:0x160e261f37a0002, packet:: clientPath:/test serverPath:/test finished:false header:: 1,4 replyHeader:: 1,11,0 request:: '/test,F response:: #ffffffc77867ffffffcbffffffa7ffffffd12dffffffa5ffffffb46f,s{2,2,1515626558454,15 2018-01-10 15:23:26,865 INFO [main] server.ZooKeeperServer(337): Closing session 0x160e261f37a0002 172 2018-01-10 15:23:26,866 INFO [ProcessThread(sid:0 cport:51573):] server.PrepRequestProcessor(486): Processed session termination for sessionid: 0x160e261f37a0002 173 2018-01-10 15:23:26,866 DEBUG [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:51573] server.ZooKeeperServer(754): Dropping request: No session with sessionid 0x160e261f37a0002 exists, probably expired and removed 174 2018-01-10 15:23:26,866 DEBUG [SyncThread:0] server.FinalRequestProcessor(88): Processing request:: sessionid:0x160e261f37a0002 type:closeSession cxid:0x0 zxid:0xc txntype:-11 reqpath:n/a 175 2018-01-10 15:23:26,867 INFO [SyncThread:0] server.NIOServerCnxn(1044): Closed socket connection for client /127.0.0.1:34294 which had sessionid 0x160e261f37a0002 176 2018-01-10 15:23:26,868 INFO [ReadOnlyZKClient-SendThread(localhost:51573)] zookeeper.ClientCnxn$SendThread(1158): Unable to read additional data from server sessionid 0x160e261f37a0002, likely server has closed socket, closing socket connection and attempting reconnect 177 2018-01-10 15:23:26,968 WARN [ReadOnlyZKClient] zookeeper.ReadOnlyZKClient$ZKTask$1(178): 0x57f23557 failed for get of /test, code = CONNECTIONLOSS, retries = 1 178 2018-01-10 15:23:27,190 INFO [ReadOnlyZKClient-SendThread(user-164-127-0-0.play-internet.pl:51573)] zookeeper.ClientCnxn$SendThread(1032): Opening socket connection to server user-164-127-0-0.play-internet.pl/164.127.0.0:51573. Will not attempt to authenticate using SASL (unknown error) 179 2018-01-10 15:23:30,069 INFO [ReadOnlyZKClient] zookeeper.ReadOnlyZKClient(305): 0x57f23557 no activities for 3000 ms, close active connection. Will reconnect next time when there are new requests. 180 2018-01-10 15:23:30,069 DEBUG [ReadOnlyZKClient] zookeeper.ZooKeeper(673): Closing session: 0x160e261f37a0002 181 2018-01-10 15:23:30,069 DEBUG [ReadOnlyZKClient] zookeeper.ClientCnxn(1370): Closing client for session: 0x160e261f37a0002 182 2018-01-10 15:23:46,976 WARN [ReadOnlyZKClient-SendThread(user-164-127-0-0.play-internet.pl:51573)] zookeeper.ClientCnxn$SendThread(1108): Client session timed out, have not heard from server in 20008ms for sessionid 0x160e261f37a0002 183 2018-01-10 15:23:46,977 DEBUG [ReadOnlyZKClient-SendThread(user-164-127-0-0.play-internet.pl:51573)] zookeeper.ClientCnxn$SendThread(1146): An exception was thrown while closing send thread for session 0x160e261f37a0002 : Client session timed out, have not heard from server in 20008ms for sessionid 0x160e261f37a0002 184 2018-01-10 15:23:46,977 DEBUG [ReadOnlyZKClient-SendThread(user-164-127-0-0.play-internet.pl:51573)] zookeeper.ClientCnxnSocketNIO(203): Ignoring exception during shutdown input 185 java.net.SocketException: Socket is not connected 186 at sun.nio.ch.Net.translateToSocketException(Net.java:123) 187 at sun.nio.ch.Net.translateException(Net.java:157) 188 at sun.nio.ch.Net.translateException(Net.java:163) 189 at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:401) 190 at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:200) 191 at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1246) 192 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1182) 193 Caused by: java.nio.channels.NotYetConnectedException 194 at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:782) 195 at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:399) 196 ... 3 more 197 2018-01-10 15:23:46,977 DEBUG [ReadOnlyZKClient-SendThread(user-164-127-0-0.play-internet.pl:51573)] zookeeper.ClientCnxnSocketNIO(210): Ignoring exception during shutdown output 198 java.net.SocketException: Socket is not connected 199 at sun.nio.ch.Net.translateToSocketException(Net.java:123) 200 at sun.nio.ch.Net.translateException(Net.java:157) 201 at sun.nio.ch.Net.translateException(Net.java:163) 202 at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:409) 203 at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:207) 204 at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1246) 205 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1182) 206 Caused by: java.nio.channels.NotYetConnectedException 207 at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:799) 208 at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:407) 209 ... 3 more 210 2018-01-10 15:23:47,078 DEBUG [ReadOnlyZKClient] zookeeper.ClientCnxn(1354): Disconnecting client for session: 0x160e261f37a0002 211 2018-01-10 15:23:47,079 INFO [ReadOnlyZKClient] zookeeper.ZooKeeper(684): Session: 0x160e261f37a0002 closed 212 2018-01-10 15:23:47,079 INFO [ReadOnlyZKClient-EventThread] zookeeper.ClientCnxn$EventThread(519): EventThread shut down for session: 0x160e261f37a0002 213 2018-01-10 15:23:47,079 WARN [ReadOnlyZKClient] zookeeper.ReadOnlyZKClient$ZKTask$1(178): 0x57f23557 failed for get of /test, code = CONNECTIONLOSS, retries = 2 214 2018-01-10 15:23:47,179 INFO [ReadOnlyZKClient] zookeeper.ZooKeeper(438): Initiating client connection, connectString=localhost:51573 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$11/1499902826@57a14636 215 2018-01-10 15:23:47,180 INFO [ReadOnlyZKClient-SendThread(user-164-127-0-0.play-internet.pl:51573)] zookeeper.ClientCnxn$SendThread(1032): Opening socket connection to server user-164-127-0-0.play-internet.pl/164.127.0.0:51573. Will not attempt to authenticate using SASL (unknown error) 216 2018-01-10 15:23:50,180 INFO [ReadOnlyZKClient] zookeeper.ReadOnlyZKClient(305): 0x57f23557 no activities for 3000 ms, close active connection. Will reconnect next time when there are new requests. 217 2018-01-10 15:23:50,181 DEBUG [ReadOnlyZKClient] zookeeper.ZooKeeper(673): Closing session: 0x0 218 2018-01-10 15:23:50,181 DEBUG [ReadOnlyZKClient] zookeeper.ClientCnxn(1370): Closing client for session: 0x0 219 2018-01-10 15:24:32,222 WARN [ReadOnlyZKClient-SendThread(user-164-127-0-0.play-internet.pl:51573)] zookeeper.ClientCnxn$SendThread(1108): Client session timed out, have not heard from server in 45042ms for sessionid 0x0 220 2018-01-10 15:24:32,223 DEBUG [ReadOnlyZKClient-SendThread(user-164-127-0-0.play-internet.pl:51573)] zookeeper.ClientCnxn$SendThread(1146): An exception was thrown while closing send thread for session 0x0 : Client session timed out, have not heard from server in 45042ms for sessionid 0x0 221 2018-01-10 15:24:32,223 DEBUG [ReadOnlyZKClient-SendThread(user-164-127-0-0.play-internet.pl:51573)] zookeeper.ClientCnxnSocketNIO(203): Ignoring exception during shutdown input 222 java.net.SocketException: Socket is not connected 223 at sun.nio.ch.Net.translateToSocketException(Net.java:123) 224 at sun.nio.ch.Net.translateException(Net.java:157) 225 at sun.nio.ch.Net.translateException(Net.java:163) 226 at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:401) 227 at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:200) 228 at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1246) 229 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1182) 230 Caused by: java.nio.channels.NotYetConnectedException 231 at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:782) 232 at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:399) 233 ... 3 more 234 2018-01-10 15:24:32,223 DEBUG [ReadOnlyZKClient-SendThread(user-164-127-0-0.play-internet.pl:51573)] zookeeper.ClientCnxnSocketNIO(210): Ignoring exception during shutdown output 235 java.net.SocketException: Socket is not connected 236 at sun.nio.ch.Net.translateToSocketException(Net.java:123) 237 at sun.nio.ch.Net.translateException(Net.java:157) 238 at sun.nio.ch.Net.translateException(Net.java:163) 239 at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:409) 240 at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:207) 241 at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1246) 242 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1182) 243 Caused by: java.nio.channels.NotYetConnectedException 244 at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:799) 245 at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:407) 246 ... 3 more 247 2018-01-10 15:24:32,324 DEBUG [ReadOnlyZKClient] zookeeper.ClientCnxn(1354): Disconnecting client for session: 0x0 248 2018-01-10 15:24:32,325 INFO [ReadOnlyZKClient] zookeeper.ZooKeeper(684): Session: 0x0 closed 249 2018-01-10 15:24:32,325 INFO [ReadOnlyZKClient-EventThread] zookeeper.ClientCnxn$EventThread(519): EventThread shut down for session: 0x0 250 2018-01-10 15:24:32,325 WARN [ReadOnlyZKClient] zookeeper.ReadOnlyZKClient$ZKTask$1(178): 0x57f23557 failed for get of /test, code = CONNECTIONLOSS, retries = 3 251 2018-01-10 15:24:32,425 INFO [ReadOnlyZKClient] zookeeper.ZooKeeper(438): Initiating client connection, connectString=localhost:51573 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$11/1499902826@57a14636 252 2018-01-10 15:24:32,427 INFO [ReadOnlyZKClient-SendThread(user-164-127-0-0.play-internet.pl:51573)] zookeeper.ClientCnxn$SendThread(1032): Opening socket connection to server user-164-127-0-0.play-internet.pl/164.127.0.0:51573. Will not attempt to authenticate using SASL (unknown error) 253 2018-01-10 15:24:35,427 INFO [ReadOnlyZKClient] zookeeper.ReadOnlyZKClient(305): 0x57f23557 no activities for 3000 ms, close active connection. Will reconnect next time when there are new requests. 254 2018-01-10 15:24:35,427 DEBUG [ReadOnlyZKClient] zookeeper.ZooKeeper(673): Closing session: 0x0 255 2018-01-10 15:24:35,427 DEBUG [ReadOnlyZKClient] zookeeper.ClientCnxn(1370): Closing client for session: 0x0 256 2018-01-10 15:25:17,468 WARN [ReadOnlyZKClient-SendThread(user-164-127-0-0.play-internet.pl:51573)] zookeeper.ClientCnxn$SendThread(1108): Client session timed out, have not heard from server in 45042ms for sessionid 0x0 257 2018-01-10 15:25:17,469 DEBUG [ReadOnlyZKClient-SendThread(user-164-127-0-0.play-internet.pl:51573)] zookeeper.ClientCnxn$SendThread(1146): An exception was thrown while closing send thread for session 0x0 : Client session timed out, have not heard from server in 45042ms for sessionid 0x0 258 2018-01-10 15:25:17,469 DEBUG [ReadOnlyZKClient-SendThread(user-164-127-0-0.play-internet.pl:51573)] zookeeper.ClientCnxnSocketNIO(203): Ignoring exception during shutdown input 259 java.net.SocketException: Socket is not connected 260 at sun.nio.ch.Net.translateToSocketException(Net.java:123) 261 at sun.nio.ch.Net.translateException(Net.java:157) 262 at sun.nio.ch.Net.translateException(Net.java:163) 263 at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:401) 264 at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:200) 265 at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1246) 266 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1182) 267 Caused by: java.nio.channels.NotYetConnectedException 268 at sun.nio.ch.SocketChannelImpl.shutdownInput(SocketChannelImpl.java:782) 269 at sun.nio.ch.SocketAdaptor.shutdownInput(SocketAdaptor.java:399) 270 ... 3 more 271 2018-01-10 15:25:17,469 DEBUG [ReadOnlyZKClient-SendThread(user-164-127-0-0.play-internet.pl:51573)] zookeeper.ClientCnxnSocketNIO(210): Ignoring exception during shutdown output 272 java.net.SocketException: Socket is not connected 273 at sun.nio.ch.Net.translateToSocketException(Net.java:123) 274 at sun.nio.ch.Net.translateException(Net.java:157) 275 at sun.nio.ch.Net.translateException(Net.java:163) 276 at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:409) 277 at org.apache.zookeeper.ClientCnxnSocketNIO.cleanup(ClientCnxnSocketNIO.java:207) 278 at org.apache.zookeeper.ClientCnxn$SendThread.cleanup(ClientCnxn.java:1246) 279 at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1182) 280 Caused by: java.nio.channels.NotYetConnectedException 281 at sun.nio.ch.SocketChannelImpl.shutdownOutput(SocketChannelImpl.java:799) 282 at sun.nio.ch.SocketAdaptor.shutdownOutput(SocketAdaptor.java:407) 283 ... 3 more 284 2018-01-10 15:25:17,570 DEBUG [ReadOnlyZKClient] zookeeper.ClientCnxn(1354): Disconnecting client for session: 0x0 285 2018-01-10 15:25:17,571 INFO [ReadOnlyZKClient] zookeeper.ZooKeeper(684): Session: 0x0 closed 286 2018-01-10 15:25:17,571 INFO [ReadOnlyZKClient-EventThread] zookeeper.ClientCnxn$EventThread(519): EventThread shut down for session: 0x0 287 2018-01-10 15:25:17,571 WARN [ReadOnlyZKClient] zookeeper.ReadOnlyZKClient$ZKTask$1(182): 0x57f23557 failed for get of /test, code = CONNECTIONLOSS, retries = 3, give up 288 2018-01-10 15:25:17,609 INFO [main] hbase.ResourceChecker(172): after: zookeeper.TestReadOnlyZKClient#testSessionExpire Thread=12 (was 13), OpenFileDescriptor=108 (was 112), MaxFileDescriptor=32000 (was 32000), SystemLoadAverage=6 (was 67), ProcessCount=833 (was 833), AvailableMemoryMB=12090 (was 12065) - AvailableMemoryMB LEAK? - {code} > Miscellany of fixes for hbase-zookeeper tests to make them more robust > ---------------------------------------------------------------------- > > Key: HBASE-19753 > URL: https://issues.apache.org/jira/browse/HBASE-19753 > Project: HBase > Issue Type: Bug > Reporter: stack > Assignee: stack > Attachments: HBASE-19753.branch-2.001.patch > > > On my cluster which slows zk, tests hbase-zookeeper rarely all pass. -- This message was sent by Atlassian JIRA (v6.4.14#64029)