Re: Cannot start ignite nodes with shared memory - Ignite version 2.10.0
Also, as I know, shmem is not recommended to use. Shmem will not be used in Ignite 3. пн, 21 июн. 2021 г. в 15:30, Ilya Kazakov : > Hello, as I can see no one from the community wants to take this ticket. > Try to ask on dev-list: d...@ignite.apache.org > > ср, 2 июн. 2021 г. в 05:11, sarahsamji : > >> Hi, is there a fix expected for this issue in the next release? >> >> >> >> -- >> Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >> >
Re: Cannot start ignite nodes with shared memory - Ignite version 2.10.0
Hello, as I can see no one from the community wants to take this ticket. Try to ask on dev-list: d...@ignite.apache.org ср, 2 июн. 2021 г. в 05:11, sarahsamji : > Hi, is there a fix expected for this issue in the next release? > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
Re: Cannot start ignite nodes with shared memory - Ignite version 2.10.0
Hi, is there a fix expected for this issue in the next release? -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Cannot start ignite nodes with shared memory - Ignite version 2.10.0
Thankyou Ilya. -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: Cannot start ignite nodes with shared memory - Ignite version 2.10.0
Hello. It looks like a bug. I have reported it: https://issues.apache.org/jira/browse/IGNITE-14634 Ilya пт, 16 апр. 2021 г. в 05:09, sarahsamji : > Sample > > import org.apache.ignite.Ignite; > import org.apache.ignite.Ignition; > import org.apache.ignite.configuration.*; > import org.apache.ignite.events.EventType; > import org.apache.ignite.failure.StopNodeOrHaltFailureHandler; > import org.apache.ignite.internal.util.IgniteUtils; > import org.apache.ignite.spi.communication.CommunicationSpi; > import org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi; > import org.apache.ignite.spi.discovery.DiscoverySpi; > import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi; > import org.apache.ignite.spi.discovery.tcp.ipfinder.TcpDiscoveryIpFinder; > import > org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder; > import org.apache.logging.log4j.ThreadContext; > import org.junit.jupiter.api.BeforeAll; > import org.junit.jupiter.api.Test; > > import java.nio.file.Paths; > import java.util.List; > import java.util.Random; > import static org.junit.jupiter.api.Assertions.*; > > public class SampleTest { > static String localhost; > private static Random random; > final static int discPort = 36101; > final static int commPort = 37601; > final static int sharedMemPort = 37235; > final static int clusterSize = 2; > private static final int messageQueueLimit = 8192; > private static final long tcpCommTimeout = 10_000L; > private static final int igniteCommPortRange = 5; > final static int igniteDiscPortRange = clusterSize * 2; > final static int igniteDiscMaxPort = discPort + igniteDiscPortRange; > private static final long discoveryNetworkTimeout = 20_000L; > private static final int default_threadpool_size = 8; > private static final int igniteLongRunningThreadPoolSize = 8; > private static final int igniteStreamerThreadPoolSize = 8; > > private static final String igniteDiscHost = "127.0.0.1"; > final static String tmpdir = System.getProperty("java.io.tmpdir"); > public static final int[] EVENT_TYPES = { > EventType.EVT_CACHE_OBJECT_EXPIRED, > EventType.EVT_NODE_JOINED, > EventType.EVT_NODE_LEFT, > EventType.EVT_NODE_SEGMENTED, > EventType.EVT_CLIENT_NODE_DISCONNECTED, > EventType.EVT_CLIENT_NODE_RECONNECTED, > EventType.EVT_NODE_FAILED, > EventType.EVT_CACHE_REBALANCE_PART_DATA_LOST}; > > > @BeforeAll > static void beforeAll() { > IgniteUtils.setCurrentIgniteName(""); > ThreadContext.clearAll(); > System.setProperty("ignite.cluster.encryption.disabled", "true"); > } > > @Test > void test() { > int httpPort = 37500; > localhost = "localhost:" + httpPort; > try { > startCluster(); > final Ignite ignite = Ignition.ignite("NODE_I1"); > assertNotNull(ignite); > } finally { > IgniteUtils.setCurrentIgniteName(null); > ThreadContext.clearAll(); > } > } > > private static void startCluster() { > random = new Random(); > Thread.interrupted(); > > startNode("NODE_1"); > > for (int i = 2; i <= clusterSize; i++) { > startNode("NODE" + i); > } > > Ignite i1 = Ignition.ignite("NODE_1"); > assertEquals(clusterSize, i1.cluster().nodes().size()); > } > > private static void startNode(String instanceName) { > IgniteConfiguration igniteConfig = new > IgniteConfiguration().setIgniteInstanceName(instanceName) > .setPeerClassLoadingEnabled(false) > .setWorkDirectory(Paths.get(tmpdir,"ignite", "work", > String.valueOf(random.nextInt(100))).toString()) > .setFailureDetectionTimeout(30_000L) > .setMetricsLogFrequency(30L) > > .setDataStorageConfiguration(createDataStorageConfiguration()) > .setIncludeEventTypes(EVENT_TYPES) > .setCommunicationSpi(createCommunicationSpi()) > .setDiscoverySpi(createDiscoverySpi()) > .setPublicThreadPoolSize(default_threadpool_size) > > .setDataStreamerThreadPoolSize(igniteStreamerThreadPoolSize) > .setSystemThreadPoolSize(default_threadpool_size) > .setServiceThreadPoolSize(default_threadpool_size) > .setStripedPoolSize(default_threadpool_size) > > .setExecutorConfiguration(createExecutorConfig("IgniteLongRunning", > igniteLongRunningThreadPoolSize)) > .setConnectorConfiguration(null) > .setClientConnectorConfiguration(null) > .setBinaryConfiguration(createBinaryConfiguration()) > .setFailureHandler(new StopNodeOrHaltFailureHandler(false, > 0));; > Ignite ignite = Ignition.getOrStart(igniteConfig); > } > > static DataStora
Re: Cannot start ignite nodes with shared memory - Ignite version 2.10.0
Sample import org.apache.ignite.Ignite; import org.apache.ignite.Ignition; import org.apache.ignite.configuration.*; import org.apache.ignite.events.EventType; import org.apache.ignite.failure.StopNodeOrHaltFailureHandler; import org.apache.ignite.internal.util.IgniteUtils; import org.apache.ignite.spi.communication.CommunicationSpi; import org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi; import org.apache.ignite.spi.discovery.DiscoverySpi; import org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi; import org.apache.ignite.spi.discovery.tcp.ipfinder.TcpDiscoveryIpFinder; import org.apache.ignite.spi.discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder; import org.apache.logging.log4j.ThreadContext; import org.junit.jupiter.api.BeforeAll; import org.junit.jupiter.api.Test; import java.nio.file.Paths; import java.util.List; import java.util.Random; import static org.junit.jupiter.api.Assertions.*; public class SampleTest { static String localhost; private static Random random; final static int discPort = 36101; final static int commPort = 37601; final static int sharedMemPort = 37235; final static int clusterSize = 2; private static final int messageQueueLimit = 8192; private static final long tcpCommTimeout = 10_000L; private static final int igniteCommPortRange = 5; final static int igniteDiscPortRange = clusterSize * 2; final static int igniteDiscMaxPort = discPort + igniteDiscPortRange; private static final long discoveryNetworkTimeout = 20_000L; private static final int default_threadpool_size = 8; private static final int igniteLongRunningThreadPoolSize = 8; private static final int igniteStreamerThreadPoolSize = 8; private static final String igniteDiscHost = "127.0.0.1"; final static String tmpdir = System.getProperty("java.io.tmpdir"); public static final int[] EVENT_TYPES = { EventType.EVT_CACHE_OBJECT_EXPIRED, EventType.EVT_NODE_JOINED, EventType.EVT_NODE_LEFT, EventType.EVT_NODE_SEGMENTED, EventType.EVT_CLIENT_NODE_DISCONNECTED, EventType.EVT_CLIENT_NODE_RECONNECTED, EventType.EVT_NODE_FAILED, EventType.EVT_CACHE_REBALANCE_PART_DATA_LOST}; @BeforeAll static void beforeAll() { IgniteUtils.setCurrentIgniteName(""); ThreadContext.clearAll(); System.setProperty("ignite.cluster.encryption.disabled", "true"); } @Test void test() { int httpPort = 37500; localhost = "localhost:" + httpPort; try { startCluster(); final Ignite ignite = Ignition.ignite("NODE_I1"); assertNotNull(ignite); } finally { IgniteUtils.setCurrentIgniteName(null); ThreadContext.clearAll(); } } private static void startCluster() { random = new Random(); Thread.interrupted(); startNode("NODE_1"); for (int i = 2; i <= clusterSize; i++) { startNode("NODE" + i); } Ignite i1 = Ignition.ignite("NODE_1"); assertEquals(clusterSize, i1.cluster().nodes().size()); } private static void startNode(String instanceName) { IgniteConfiguration igniteConfig = new IgniteConfiguration().setIgniteInstanceName(instanceName) .setPeerClassLoadingEnabled(false) .setWorkDirectory(Paths.get(tmpdir,"ignite", "work", String.valueOf(random.nextInt(100))).toString()) .setFailureDetectionTimeout(30_000L) .setMetricsLogFrequency(30L) .setDataStorageConfiguration(createDataStorageConfiguration()) .setIncludeEventTypes(EVENT_TYPES) .setCommunicationSpi(createCommunicationSpi()) .setDiscoverySpi(createDiscoverySpi()) .setPublicThreadPoolSize(default_threadpool_size) .setDataStreamerThreadPoolSize(igniteStreamerThreadPoolSize) .setSystemThreadPoolSize(default_threadpool_size) .setServiceThreadPoolSize(default_threadpool_size) .setStripedPoolSize(default_threadpool_size) .setExecutorConfiguration(createExecutorConfig("IgniteLongRunning", igniteLongRunningThreadPoolSize)) .setConnectorConfiguration(null) .setClientConnectorConfiguration(null) .setBinaryConfiguration(createBinaryConfiguration()) .setFailureHandler(new StopNodeOrHaltFailureHandler(false, 0));; Ignite ignite = Ignition.getOrStart(igniteConfig); } static DataStorageConfiguration createDataStorageConfiguration() { int evictionThreshold = 90; String DEFAULT_MEMORY_REGION = "Default_Region"; long memSize = 200 * 1024 * 1024;; DataRegionConfiguration regionConfig = new DataRegionConfiguration() .setName(DEFAULT_MEMORY_REGION)
Re: Cannot start ignite nodes with shared memory - Ignite version 2.10.0
public class SampleTest { static String localhost; private static Random random; final static int discPort = 36101; final static int commPort = 37601; final static int sharedMemPort = 37235; final static int clusterSize = 2; private static final int messageQueueLimit = 8192; private static final long tcpCommTimeout = 10_000L; private static final int igniteCommPortRange = 5; final static int igniteDiscPortRange = clusterSize * 2; final static int igniteDiscMaxPort = discPort + igniteDiscPortRange; private static final long discoveryNetworkTimeout = 20_000L; private static final int default_threadpool_size = 8; private static final int igniteLongRunningThreadPoolSize = 8; private static final int igniteStreamerThreadPoolSize = 8; private static final String igniteDiscHost = "127.0.0.1"; final static String tmpdir = System.getProperty("java.io.tmpdir"); public static final int[] EVENT_TYPES = { EventType.EVT_CACHE_OBJECT_EXPIRED, EventType.EVT_NODE_JOINED, EventType.EVT_NODE_LEFT, EventType.EVT_NODE_SEGMENTED, EventType.EVT_CLIENT_NODE_DISCONNECTED, EventType.EVT_CLIENT_NODE_RECONNECTED, EventType.EVT_NODE_FAILED, EventType.EVT_CACHE_REBALANCE_PART_DATA_LOST}; @BeforeAll static void beforeAll() { IgniteUtils.setCurrentIgniteName(""); ThreadContext.clearAll(); System.setProperty("ignite.cluster.encryption.disabled", "true"); } @Test void test() { int httpPort = PortFinder.getRandomPort(37000, 37500); localhost = "localhost:" + httpPort; try { startCluster(); final Ignite ignite = Ignition.ignite("NODE_I1"); assertNotNull(ignite); TestDomainSvcImpl.resetCounters(); } finally { IgniteUtils.setCurrentIgniteName(null); ThreadContext.clearAll(); } } private static void startCluster() { random = new Random(); Thread.interrupted(); startNode("NODE_1"); for (int i = 2; i <= clusterSize; i++) { startNode("NODE" + i); } Ignite i1 = Ignition.ignite("NODE_1"); assertEquals(clusterSize, i1.cluster().nodes().size()); } private static void startNode(String instanceName) { IgniteConfiguration igniteConfig = buildIgniteConfig(instanceName); Ignite ignite = Ignition.start(igniteConfig); } private static IgniteConfiguration buildIgniteConfig(String instanceName) { IgniteConfiguration cfg = new IgniteConfigBuilder().build(); return cfg .setIgniteInstanceName(instanceName) .setPeerClassLoadingEnabled(false) .setWorkDirectory(Paths.get(tmpdir, "firecracker", "ignite", "work", String.valueOf(random.nextInt(100))).toString()) .setFailureDetectionTimeout(30_000L) .setMetricsLogFrequency(30L) .setDataStorageConfiguration(createDataStorageConfiguration()) .setIncludeEventTypes(EVENT_TYPES) .setCommunicationSpi(createCommunicationSpi()) .setDiscoverySpi(createDiscoverySpi()) .setPublicThreadPoolSize(default_threadpool_size) .setDataStreamerThreadPoolSize(igniteStreamerThreadPoolSize) .setSystemThreadPoolSize(default_threadpool_size) .setServiceThreadPoolSize(default_threadpool_size) .setStripedPoolSize(default_threadpool_size) .setExecutorConfiguration(createExecutorConfig("IgniteLongRunning", igniteLongRunningThreadPoolSize)) .setConnectorConfiguration(null) .setClientConnectorConfiguration(null) .setBinaryConfiguration(createBinaryConfiguration()) .setFailureHandler(new StopNodeOrHaltFailureHandler(false, 0)); } static DataStorageConfiguration createDataStorageConfiguration() { int evictionThreshold = 90; String DEFAULT_MEMORY_REGION = "Default_Region"; long memSize = 200 * 1024 * 1024;; DataRegionConfiguration regionConfig = new DataRegionConfiguration() .setName(DEFAULT_MEMORY_REGION) .setInitialSize(memSize) .setMaxSize(memSize) .setMetricsEnabled(true) .setPageEvictionMode(DataPageEvictionMode.DISABLED) // Only enable eviction if we can find a way to control it .setEvictionThreshold(evictionThreshold / 100); DataStorageConfiguration dataStorageConfiguration = new DataStorageConfiguration() .setDefaultDataRegionConfiguration(regionConfig); dataStorageConfiguration.setMetricsEnabled(true); return dataStorageConfiguration; } static CommunicationSpi createCommunicationSpi() {
Re: Cannot start ignite nodes with shared memory - Ignite version 2.10.0
Hello. It will be great if you can share some simple reproducer. ср, 14 апр. 2021 г. в 03:39, sarahsamji : > Hi, > > * > Tested with:* > - Ignite version - 2.10.0 > - Multinode cluster on single/different JVM - localhost > - First node successfully starts, shared memory endpoint also starts but > cluster formation fails moving ahead and is indefinitely stuck at > GridCachePartitionExchangeManager - Failed to wait for initial partition > map > exchange. Possible reasons are: > ^-- Transactions in deadlock. > ^-- Long running transactions (ignore if this is the case). > ^-- Unreleased explicit locks. > WARN 2021-04-13T10:53:13,531 : [main] GridCachePartitionExchangeManager > - > Still waiting for initial partition map exchange. > > *Logs:* > > INFO 2021-04-13T12:21:20,628 : > [tcp-disco-srvr-[:36830]-#3%NODE_I1%-#61%NODE_I1%] TcpDiscoverySpi - TCP > discovery accepted incoming connection [rmtAddr=/0:0:0:0:0:0:0:1, > rmtPort=54827] > INFO 2021-04-13T12:21:20,628 : > [tcp-disco-srvr-[:36830]-#3%NODE_I1%-#61%NODE_I1%] TcpDiscoverySpi - TCP > discovery spawning a new thread for connection [rmtAddr=/0:0:0:0:0:0:0:1, > rmtPort=54827] > INFO 2021-04-13T12:21:20,628 : > [tcp-disco-sock-reader-[]-#6%NODE_I1%-#95%NODE_I1%] TcpDiscoverySpi - > Started serving remote node connection [rmtAddr=/0:0:0:0:0:0:0:1:54827, > rmtPort=54827] > INFO 2021-04-13T12:21:20,633 : > [tcp-disco-sock-reader-[]-#6%NODE_I1%-#95%NODE_I1%] TcpDiscoverySpi - > Received ping request from the remote node > [rmtNodeId=6d8863ec-7499-40d4-ad47-48a075adfed9, > rmtAddr=/0:0:0:0:0:0:0:1:54827, rmtPort=54827] > INFO 2021-04-13T12:21:20,633 : > [tcp-disco-sock-reader-[]-#6%NODE_I1%-#95%NODE_I1%] TcpDiscoverySpi - > Finished writing ping response > [rmtNodeId=6d8863ec-7499-40d4-ad47-48a075adfed9, > rmtAddr=/0:0:0:0:0:0:0:1:54827, rmtPort=54827] > INFO 2021-04-13T12:21:20,633 : > [tcp-disco-sock-reader-[]-#6%NODE_I1%-#95%NODE_I1%] TcpDiscoverySpi - > Finished serving remote node connection [rmtAddr=/0:0:0:0:0:0:0:1:54827, > rmtPort=54827 > INFO 2021-04-13T12:21:21,642 : > [tcp-disco-srvr-[:36830]-#3%NODE_I1%-#61%NODE_I1%] TcpDiscoverySpi - TCP > discovery accepted incoming connection [rmtAddr=/0:0:0:0:0:0:0:1, > rmtPort=54828] > INFO 2021-04-13T12:21:21,642 : > [tcp-disco-srvr-[:36830]-#3%NODE_I1%-#61%NODE_I1%] TcpDiscoverySpi - TCP > discovery spawning a new thread for connection [rmtAddr=/0:0:0:0:0:0:0:1, > rmtPort=54828] > INFO 2021-04-13T12:21:21,642 : > [tcp-disco-sock-reader-[]-#7%NODE_I1%-#96%NODE_I1%] TcpDiscoverySpi - > Started serving remote node connection [rmtAddr=/0:0:0:0:0:0:0:1:54828, > rmtPort=54828] > INFO 2021-04-13T12:21:21,646 : > [tcp-disco-sock-reader-[]-#7%NODE_I1%-#96%NODE_I1%] TcpDiscoverySpi - > Received ping request from the remote node > [rmtNodeId=6d8863ec-7499-40d4-ad47-48a075adfed9, > rmtAddr=/0:0:0:0:0:0:0:1:54828, rmtPort=54828] > INFO 2021-04-13T12:21:21,646 : > [tcp-disco-sock-reader-[]-#7%NODE_I1%-#96%NODE_I1%] TcpDiscoverySpi - > Finished writing ping response > [rmtNodeId=6d8863ec-7499-40d4-ad47-48a075adfed9, > rmtAddr=/0:0:0:0:0:0:0:1:54828, rmtPort=54828] > INFO 2021-04-13T12:21:21,646 : > [tcp-disco-sock-reader-[]-#7%NODE_I1%-#96%NODE_I1%] TcpDiscoverySpi - > Finished serving remote node connection [rmtAddr=/0:0:0:0:0:0:0:1:54828, > rmtPort=54828 > INFO 2021-04-13T12:21:22,655 : > [tcp-disco-srvr-[:36830]-#3%NODE_I1%-#61%NODE_I1%] TcpDiscoverySpi - TCP > discovery accepted incoming connection [rmtAddr=/0:0:0:0:0:0:0:1, > rmtPort=54829] > INFO 2021-04-13T12:21:22,655 : > [tcp-disco-srvr-[:36830]-#3%NODE_I1%-#61%NODE_I1%] TcpDiscoverySpi - TCP > discovery spawning a new thread for connection [rmtAddr=/0:0:0:0:0:0:0:1, > rmtPort=54829] > INFO 2021-04-13T12:21:22,656 : > [tcp-disco-sock-reader-[]-#8%NODE_I1%-#97%NODE_I1%] TcpDiscoverySpi - > Started serving remote node connection [rmtAddr=/0:0:0:0:0:0:0:1:54829, > rmtPort=54829] > INFO 2021-04-13T12:21:22,659 : > [tcp-disco-sock-reader-[]-#8%NODE_I1%-#97%NODE_I1%] TcpDiscoverySpi - > Received ping request from the remote node > [rmtNodeId=6d8863ec-7499-40d4-ad47-48a075adfed9, > rmtAddr=/0:0:0:0:0:0:0:1:54829, rmtPort=54829] > INFO 2021-04-13T12:21:22,659 : > [tcp-disco-sock-reader-[]-#8%NODE_I1%-#97%NODE_I1%] TcpDiscoverySpi - > Finished writing ping response > [rmtNodeId=6d8863ec-7499-40d4-ad47-48a075adfed9, > rmtAddr=/0:0:0:0:0:0:0:1:54829, rmtPort=54829] > INFO 2021-04-13T12:21:22,659 : > [tcp-disco-sock-reader-[]-#8%NODE_I1%-#97%NODE_I1%] TcpDiscoverySpi - > Finished serving remote node connection [rmtAddr=/0:0:0:0:0:0:0:1:54829, > rmtPort=54829 > WARN 2021-04-13T12:21:30,044 : [services-deployment-worker-#76%NODE_I1%] > ServiceDeploymentManager - Failed to wait service deployment process or > timeout had been reached, timeout=1, > taskDepId=ServiceDeploymentProcessId [topVer=AffinityTopologyVersion > [topVer=2, minorTopVer=0], reqId=null] > WARN 2021-04-13T12:21:30,049 : [exchange-worker-#6
Cannot start ignite nodes with shared memory - Ignite version 2.10.0
Hi, * Tested with:* - Ignite version - 2.10.0 - Multinode cluster on single/different JVM - localhost - First node successfully starts, shared memory endpoint also starts but cluster formation fails moving ahead and is indefinitely stuck at GridCachePartitionExchangeManager - Failed to wait for initial partition map exchange. Possible reasons are: ^-- Transactions in deadlock. ^-- Long running transactions (ignore if this is the case). ^-- Unreleased explicit locks. WARN 2021-04-13T10:53:13,531 : [main] GridCachePartitionExchangeManager - Still waiting for initial partition map exchange. *Logs:* INFO 2021-04-13T12:21:20,628 : [tcp-disco-srvr-[:36830]-#3%NODE_I1%-#61%NODE_I1%] TcpDiscoverySpi - TCP discovery accepted incoming connection [rmtAddr=/0:0:0:0:0:0:0:1, rmtPort=54827] INFO 2021-04-13T12:21:20,628 : [tcp-disco-srvr-[:36830]-#3%NODE_I1%-#61%NODE_I1%] TcpDiscoverySpi - TCP discovery spawning a new thread for connection [rmtAddr=/0:0:0:0:0:0:0:1, rmtPort=54827] INFO 2021-04-13T12:21:20,628 : [tcp-disco-sock-reader-[]-#6%NODE_I1%-#95%NODE_I1%] TcpDiscoverySpi - Started serving remote node connection [rmtAddr=/0:0:0:0:0:0:0:1:54827, rmtPort=54827] INFO 2021-04-13T12:21:20,633 : [tcp-disco-sock-reader-[]-#6%NODE_I1%-#95%NODE_I1%] TcpDiscoverySpi - Received ping request from the remote node [rmtNodeId=6d8863ec-7499-40d4-ad47-48a075adfed9, rmtAddr=/0:0:0:0:0:0:0:1:54827, rmtPort=54827] INFO 2021-04-13T12:21:20,633 : [tcp-disco-sock-reader-[]-#6%NODE_I1%-#95%NODE_I1%] TcpDiscoverySpi - Finished writing ping response [rmtNodeId=6d8863ec-7499-40d4-ad47-48a075adfed9, rmtAddr=/0:0:0:0:0:0:0:1:54827, rmtPort=54827] INFO 2021-04-13T12:21:20,633 : [tcp-disco-sock-reader-[]-#6%NODE_I1%-#95%NODE_I1%] TcpDiscoverySpi - Finished serving remote node connection [rmtAddr=/0:0:0:0:0:0:0:1:54827, rmtPort=54827 INFO 2021-04-13T12:21:21,642 : [tcp-disco-srvr-[:36830]-#3%NODE_I1%-#61%NODE_I1%] TcpDiscoverySpi - TCP discovery accepted incoming connection [rmtAddr=/0:0:0:0:0:0:0:1, rmtPort=54828] INFO 2021-04-13T12:21:21,642 : [tcp-disco-srvr-[:36830]-#3%NODE_I1%-#61%NODE_I1%] TcpDiscoverySpi - TCP discovery spawning a new thread for connection [rmtAddr=/0:0:0:0:0:0:0:1, rmtPort=54828] INFO 2021-04-13T12:21:21,642 : [tcp-disco-sock-reader-[]-#7%NODE_I1%-#96%NODE_I1%] TcpDiscoverySpi - Started serving remote node connection [rmtAddr=/0:0:0:0:0:0:0:1:54828, rmtPort=54828] INFO 2021-04-13T12:21:21,646 : [tcp-disco-sock-reader-[]-#7%NODE_I1%-#96%NODE_I1%] TcpDiscoverySpi - Received ping request from the remote node [rmtNodeId=6d8863ec-7499-40d4-ad47-48a075adfed9, rmtAddr=/0:0:0:0:0:0:0:1:54828, rmtPort=54828] INFO 2021-04-13T12:21:21,646 : [tcp-disco-sock-reader-[]-#7%NODE_I1%-#96%NODE_I1%] TcpDiscoverySpi - Finished writing ping response [rmtNodeId=6d8863ec-7499-40d4-ad47-48a075adfed9, rmtAddr=/0:0:0:0:0:0:0:1:54828, rmtPort=54828] INFO 2021-04-13T12:21:21,646 : [tcp-disco-sock-reader-[]-#7%NODE_I1%-#96%NODE_I1%] TcpDiscoverySpi - Finished serving remote node connection [rmtAddr=/0:0:0:0:0:0:0:1:54828, rmtPort=54828 INFO 2021-04-13T12:21:22,655 : [tcp-disco-srvr-[:36830]-#3%NODE_I1%-#61%NODE_I1%] TcpDiscoverySpi - TCP discovery accepted incoming connection [rmtAddr=/0:0:0:0:0:0:0:1, rmtPort=54829] INFO 2021-04-13T12:21:22,655 : [tcp-disco-srvr-[:36830]-#3%NODE_I1%-#61%NODE_I1%] TcpDiscoverySpi - TCP discovery spawning a new thread for connection [rmtAddr=/0:0:0:0:0:0:0:1, rmtPort=54829] INFO 2021-04-13T12:21:22,656 : [tcp-disco-sock-reader-[]-#8%NODE_I1%-#97%NODE_I1%] TcpDiscoverySpi - Started serving remote node connection [rmtAddr=/0:0:0:0:0:0:0:1:54829, rmtPort=54829] INFO 2021-04-13T12:21:22,659 : [tcp-disco-sock-reader-[]-#8%NODE_I1%-#97%NODE_I1%] TcpDiscoverySpi - Received ping request from the remote node [rmtNodeId=6d8863ec-7499-40d4-ad47-48a075adfed9, rmtAddr=/0:0:0:0:0:0:0:1:54829, rmtPort=54829] INFO 2021-04-13T12:21:22,659 : [tcp-disco-sock-reader-[]-#8%NODE_I1%-#97%NODE_I1%] TcpDiscoverySpi - Finished writing ping response [rmtNodeId=6d8863ec-7499-40d4-ad47-48a075adfed9, rmtAddr=/0:0:0:0:0:0:0:1:54829, rmtPort=54829] INFO 2021-04-13T12:21:22,659 : [tcp-disco-sock-reader-[]-#8%NODE_I1%-#97%NODE_I1%] TcpDiscoverySpi - Finished serving remote node connection [rmtAddr=/0:0:0:0:0:0:0:1:54829, rmtPort=54829 WARN 2021-04-13T12:21:30,044 : [services-deployment-worker-#76%NODE_I1%] ServiceDeploymentManager - Failed to wait service deployment process or timeout had been reached, timeout=1, taskDepId=ServiceDeploymentProcessId [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=0], reqId=null] WARN 2021-04-13T12:21:30,049 : [exchange-worker-#64%NODE_I1%] diagnostic - Failed to wait for partition map exchange [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=0], node=ada9dea1-579a-4846-b831-64144fb1f370]. Dumping pending objects that might be the cause: WARN 2021-04-13T12:21:30,049 : [exchange-worker-#64%NODE_I1%] diagnostic - Ready affinit