Hello, Since this morning I get a strange Timeout on the connection to the cluster even with only one node. It used to work just fine until yesterday evening. At that point I destroyed the clustered and recreated it this morning.
I do so with a batch so it is supposed to be the same all the time. That's what I do not understand, I cannot see what change to cause that problem. Here is some logs any idea?? 2020-07-28 09:55:44,105 DEBUG [org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] (ServerService Thread Pool -- 11) Node version to set: 2.7.5#20190603-sha1:be4f2a15 2020-07-28 09:55:44,117 DEBUG [org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] (ServerService Thread Pool -- 11) Using parameter [localHost=0.0.0.0] 2020-07-28 09:55:44,117 DEBUG [org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] (ServerService Thread Pool -- 11) Using parameter [localPort=47500] 2020-07-28 09:55:44,117 DEBUG [org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] (ServerService Thread Pool -- 11) Using parameter [localPortRange=0] 2020-07-28 09:55:44,118 DEBUG [org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] (ServerService Thread Pool -- 11) Using parameter [threadPri=10] 2020-07-28 09:55:44,118 DEBUG [org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] (ServerService Thread Pool -- 11) Using parameter [failureDetectionTimeout=10000] 2020-07-28 09:55:44,118 DEBUG [org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] (ServerService Thread Pool -- 11) Using parameter [ipFinder=TcpDiscoveryIpFinderAdapter [shared=true]] 2020-07-28 09:55:44,118 DEBUG [org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] (ServerService Thread Pool -- 11) Using parameter [ipFinderCleanFreq=60000] 2020-07-28 09:55:44,118 DEBUG [org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] (ServerService Thread Pool -- 11) Using parameter [metricsUpdateFreq=2000] 2020-07-28 09:55:44,118 DEBUG [org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] (ServerService Thread Pool -- 11) Using parameter [statsPrintFreq=0] 2020-07-28 09:55:44,122 DEBUG [org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] (ServerService Thread Pool -- 11) Registered SPI MBean: org.apache:clsLdr=407b70da,igniteInstanceName=com.wolterskluwer.processing,group=SPIs,name=TcpDiscoverySpi 2020-07-28 09:55:44,133 INFO [org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] (ServerService Thread Pool -- 11) Connection check threshold is calculated: 10000 2020-07-28 09:55:44,134 DEBUG [org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] (tcp-disco-msg-worker-#2%com.wolterskluwer.processing%) Grid runnable started: tcp-disco-msg-worker 2020-07-28 09:55:44,135 DEBUG [org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] (tcp-disco-msg-worker-#2%com.wolterskluwer.processing%) Message worker started [locNodeId=32c22aa5-430c-4352-a8d3-55e0a7b2671e] 2020-07-28 09:55:44,135 INFO [org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] (ServerService Thread Pool -- 11) Successfully bound to TCP port [port=47500, localHost=0.0.0.0/0.0.0.0, locNodeId=32c22aa5-430c-4352-a8d3-55e0a7b2671e] 2020-07-28 09:55:44,137 DEBUG [org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] (ServerService Thread Pool -- 11) Local node initialized: TcpDiscoveryNode [id=32c22aa5-430c-4352-a8d3-55e0a7b2671e, addrs=[10.244.0.8, 127.0.0.1], sockAddrs=[processing-engine-pe-v1.master-864fb658c6-sdw56/10.244.0.8:47500, /127.0.0.1:47500], discPort=47500, order=0, intOrder=0, lastExchangeTime=1595930144127, loc=true, ver=2.7.5#20190603-sha1:be4f2a15, isClient=false] 2020-07-28 09:55:44,138 DEBUG [org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] (tcp-disco-srvr-#3%com.wolterskluwer.processing%) Grid runnable started: tcp-disco-srvr 2020-07-28 09:55:44,149 DEBUG [org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder] (ServerService Thread Pool -- 11) Getting Apache Ignite endpoints from: https://kubernetes.default.svc.cluster.local:443/api/v1/namespaces/default/endpoints/processing-engine-pe-v1-ignite 2020-07-28 09:55:50,187 DEBUG [org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] (tcp-disco-msg-worker-#2%com.wolterskluwer.processing%) Message has been added to queue: TcpDiscoveryStatusCheckMessage [creatorNode=TcpDiscoveryNode [id=32c22aa5-430c-4352-a8d3-55e0a7b2671e, addrs=[10.244.0.8, 127.0.0.1], sockAddrs=[processing-engine-pe-v1.master-864fb658c6-sdw56/10.244.0.8:47500, /127.0.0.1:47500], discPort=47500, order=0, intOrder=0, lastExchangeTime=1595930144127, loc=true, ver=2.7.5#20190603-sha1:be4f2a15, isClient=false], failedNodeId=null, status=0, super=TcpDiscoveryAbstractMessage [sndNodeId=null, id=44e59d49371-32c22aa5-430c-4352-a8d3-55e0a7b2671e, verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null, isClient=false]] 2020-07-28 09:55:50,187 DEBUG [org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] (tcp-disco-msg-worker-#2%com.wolterskluwer.processing%) Processing message [cls=TcpDiscoveryStatusCheckMessage, id=44e59d49371-32c22aa5-430c-4352-a8d3-55e0a7b2671e] 2020-07-28 09:55:50,187 DEBUG [org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] (tcp-disco-msg-worker-#2%com.wolterskluwer.processing%) Ignore message, local node order is not initialized [msg=TcpDiscoveryStatusCheckMessage [creatorNode=TcpDiscoveryNode [id=32c22aa5-430c-4352-a8d3-55e0a7b2671e, addrs=[10.244.0.8, 127.0.0.1], sockAddrs=[processing-engine-pe-v1.master-864fb658c6-sdw56/10.244.0.8:47500, /127.0.0.1:47500], discPort=47500, order=0, intOrder=0, lastExchangeTime=1595930144127, loc=true, ver=2.7.5#20190603-sha1:be4f2a15, isClient=false], failedNodeId=null, status=0, super=TcpDiscoveryAbstractMessage [sndNodeId=null, id=44e59d49371-32c22aa5-430c-4352-a8d3-55e0a7b2671e, verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null, isClient=false]], locNode=TcpDiscoveryNode [id=32c22aa5-430c-4352-a8d3-55e0a7b2671e, addrs=[10.244.0.8, 127.0.0.1], sockAddrs=[processing-engine-pe-v1.master-864fb658c6-sdw56/10.244.0.8:47500, /127.0.0.1:47500], discPort=47500, order=0, intOrder=0, lastExchangeTime=1595930144127, loc=true, ver=2.7.5#20190603-sha1:be4f2a15, isClient=false]] 2020-07-28 09:55:56,239 DEBUG [org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] (tcp-disco-msg-worker-#2%com.wolterskluwer.processing%) Message has been added to queue: TcpDiscoveryStatusCheckMessage [creatorNode=TcpDiscoveryNode [id=32c22aa5-430c-4352-a8d3-55e0a7b2671e, addrs=[10.244.0.8, 127.0.0.1], sockAddrs=[processing-engine-pe-v1.master-864fb658c6-sdw56/10.244.0.8:47500, /127.0.0.1:47500], discPort=47500, order=0, intOrder=0, lastExchangeTime=1595930144127, loc=true, ver=2.7.5#20190603-sha1:be4f2a15, isClient=false], failedNodeId=null, status=0, super=TcpDiscoveryAbstractMessage [sndNodeId=null, id=54e59d49371-32c22aa5-430c-4352-a8d3-55e0a7b2671e, verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null, isClient=false]] 2020-07-28 09:55:56,239 DEBUG [org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] (tcp-disco-msg-worker-#2%com.wolterskluwer.processing%) Processing message [cls=TcpDiscoveryStatusCheckMessage, id=54e59d49371-32c22aa5-430c-4352-a8d3-55e0a7b2671e] 2020-07-28 09:55:56,240 DEBUG [org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] (tcp-disco-msg-worker-#2%com.wolterskluwer.processing%) Ignore message, local node order is not initialized [msg=TcpDiscoveryStatusCheckMessage [creatorNode=TcpDiscoveryNode [id=32c22aa5-430c-4352-a8d3-55e0a7b2671e, addrs=[10.244.0.8, 127.0.0.1], sockAddrs=[processing-engine-pe-v1.master-864fb658c6-sdw56/10.244.0.8:47500, /127.0.0.1:47500], discPort=47500, order=0, intOrder=0, lastExchangeTime=1595930144127, loc=true, ver=2.7.5#20190603-sha1:be4f2a15, isClient=false], failedNodeId=null, status=0, super=TcpDiscoveryAbstractMessage [sndNodeId=null, id=54e59d49371-32c22aa5-430c-4352-a8d3-55e0a7b2671e, verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null, isClient=false]], locNode=TcpDiscoveryNode [id=32c22aa5-430c-4352-a8d3-55e0a7b2671e, addrs=[10.244.0.8, 127.0.0.1], sockAddrs=[processing-engine-pe-v1.master-864fb658c6-sdw56/10.244.0.8:47500, /127.0.0.1:47500], discPort=47500, order=0, intOrder=0, lastExchangeTime=1595930144127, loc=true, ver=2.7.5#20190603-sha1:be4f2a15, isClient=false]] 2020-07-28 09:56:02,294 DEBUG [org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] (tcp-disco-msg-worker-#2%com.wolterskluwer.processing%) Message has been added to queue: TcpDiscoveryStatusCheckMessage [creatorNode=TcpDiscoveryNode [id=32c22aa5-430c-4352-a8d3-55e0a7b2671e, addrs=[10.244.0.8, 127.0.0.1], sockAddrs=[processing-engine-pe-v1.master-864fb658c6-sdw56/10.244.0.8:47500, /127.0.0.1:47500], discPort=47500, order=0, intOrder=0, lastExchangeTime=1595930144127, loc=true, ver=2.7.5#20190603-sha1:be4f2a15, isClient=false], failedNodeId=null, status=0, super=TcpDiscoveryAbstractMessage [sndNodeId=null, id=64e59d49371-32c22aa5-430c-4352-a8d3-55e0a7b2671e, verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null, isClient=false]] 2020-07-28 09:56:02,294 DEBUG [org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] (tcp-disco-msg-worker-#2%com.wolterskluwer.processing%) Processing message [cls=TcpDiscoveryStatusCheckMessage, id=64e59d49371-32c22aa5-430c-4352-a8d3-55e0a7b2671e] 2020-07-28 09:56:02,295 DEBUG [org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] (tcp-disco-msg-worker-#2%com.wolterskluwer.processing%) Ignore message, local node order is not initialized [msg=TcpDiscoveryStatusCheckMessage [creatorNode=TcpDiscoveryNode [id=32c22aa5-430c-4352-a8d3-55e0a7b2671e, addrs=[10.244.0.8, 127.0.0.1], sockAddrs=[processing-engine-pe-v1.master-864fb658c6-sdw56/10.244.0.8:47500, /127.0.0.1:47500], discPort=47500, order=0, intOrder=0, lastExchangeTime=1595930144127, loc=true, ver=2.7.5#20190603-sha1:be4f2a15, isClient=false], failedNodeId=null, status=0, super=TcpDiscoveryAbstractMessage [sndNodeId=null, id=64e59d49371-32c22aa5-430c-4352-a8d3-55e0a7b2671e, verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null, isClient=false]], locNode=TcpDiscoveryNode [id=32c22aa5-430c-4352-a8d3-55e0a7b2671e, addrs=[10.244.0.8, 127.0.0.1], sockAddrs=[processing-engine-pe-v1.master-864fb658c6-sdw56/10.244.0.8:47500, /127.0.0.1:47500], discPort=47500, order=0, intOrder=0, lastExchangeTime=1595930144127, loc=true, ver=2.7.5#20190603-sha1:be4f2a15, isClient=false]] and here the exception 2020-07-28 09:57:51,279 DEBUG [org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] (tcp-disco-msg-worker-#2%com.wolterskluwer.processing%) Message has been added to queue: TcpDiscoveryStatusCheckMessage [creatorNode=TcpDiscoveryNode [id=32c22aa5-430c-4352-a8d3-55e0a7b2671e, addrs=[10.244.0.8, 127.0.0.1], sockAddrs=[processing-engine-pe-v1.master-864fb658c6-sdw56/10.244.0.8:47500, /127.0.0.1:47500], discPort=47500, order=0, intOrder=0, lastExchangeTime=1595930144127, loc=true, ver=2.7.5#20190603-sha1:be4f2a15, isClient=false], failedNodeId=null, status=0, super=TcpDiscoveryAbstractMessage [sndNodeId=null, id=85e59d49371-32c22aa5-430c-4352-a8d3-55e0a7b2671e, verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null, isClient=false]] 2020-07-28 09:57:51,279 DEBUG [org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] (tcp-disco-msg-worker-#2%com.wolterskluwer.processing%) Processing message [cls=TcpDiscoveryStatusCheckMessage, id=85e59d49371-32c22aa5-430c-4352-a8d3-55e0a7b2671e] 2020-07-28 09:57:51,280 DEBUG [org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] (tcp-disco-msg-worker-#2%com.wolterskluwer.processing%) Ignore message, local node order is not initialized [msg=TcpDiscoveryStatusCheckMessage [creatorNode=TcpDiscoveryNode [id=32c22aa5-430c-4352-a8d3-55e0a7b2671e, addrs=[10.244.0.8, 127.0.0.1], sockAddrs=[processing-engine-pe-v1.master-864fb658c6-sdw56/10.244.0.8:47500, /127.0.0.1:47500], discPort=47500, order=0, intOrder=0, lastExchangeTime=1595930144127, loc=true, ver=2.7.5#20190603-sha1:be4f2a15, isClient=false], failedNodeId=null, status=0, super=TcpDiscoveryAbstractMessage [sndNodeId=null, id=85e59d49371-32c22aa5-430c-4352-a8d3-55e0a7b2671e, verifierNodeId=null, topVer=0, pendingIdx=0, failedNodes=null, isClient=false]], locNode=TcpDiscoveryNode [id=32c22aa5-430c-4352-a8d3-55e0a7b2671e, addrs=[10.244.0.8, 127.0.0.1], sockAddrs=[processing-engine-pe-v1.master-864fb658c6-sdw56/10.244.0.8:47500, /127.0.0.1:47500], discPort=47500, order=0, intOrder=0, lastExchangeTime=1595930144127, loc=true, ver=2.7.5#20190603-sha1:be4f2a15, isClient=false]] 2020-07-28 09:57:53,999 ERROR [org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi] (ServerService Thread Pool -- 11) Failed to get registered addresses from IP finder on start (retrying every 2000ms; change 'reconnectDelay' to configure the frequency of retries).: class org.apache.ignite.spi.IgniteSpiException: Failed to retrieve Ignite pods IP addresses. at org.apache.ignite.spi.discovery.tcp.ipfinder.kubernetes.TcpDiscoveryKubernetesIpFinder.getRegisteredAddresses(TcpDiscoveryKubernetesIpFinder.java:172) at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.registeredAddresses(TcpDiscoverySpi.java:1900) at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.resolvedAddresses(TcpDiscoverySpi.java:1848) at org.apache.ignite.spi.discovery.tcp.ServerImpl.sendJoinRequestMessage(ServerImpl.java:1049) at org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:910) at org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:391) Caused by: java.net.ConnectException: Connection timed out (Connection timed out) at java.net.PlainSocketImpl.socketConnect(Native Method) at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
