These are the full set of logs, if it helps- [10:10:56,860][WARNING][main][G] Ignite work directory is not provided, automatically resolved to: /home/dsudev/ignite-master/work [10:10:56,873][WARNING][main][G] Consistent ID is not set, it is recommended to set consistent ID for production clusters (use IgniteConfiguration.setConsistentId property) [10:10:57,103][INFO][main][IgniteKernal]
>>> __________ ________________ >>> / _/ ___/ |/ / _/_ __/ __/ >>> _/ // (7 7 // / / / / _/ >>> /___/\___/_/|_/___/ /_/ /___/ >>> >>> ver. 2.8.0#20200226-sha1:341b01df >>> 2020 Copyright(C) Apache Software Foundation >>> >>> Ignite documentation: http://ignite.apache.org [10:10:57,134][INFO][main][IgniteKernal] Config URL: n/a [10:10:57,190][INFO][main][IgniteKernal] IgniteConfiguration [igniteInstanceName=null, pubPoolSize=8, svcPoolSize=8, callbackPoolSize=8, stripedPoolSize=8, sysPoolSize=8, mgmtPoolSize=4, igfsPoolSize=1, dataStreamerPoolSize=8, utilityCachePoolSize=8, utilityCacheKeepAliveTime=60000, p2pPoolSize=2, qryPoolSize=8, sqlQryHistSize=1000, dfltQryTimeout=0, igniteHome=/home/dsudev/ignite-master, igniteWorkDir=/home/dsudev/ignite-master/work, mbeanSrv=com.sun.jmx.mbeanserver.JmxMBeanServer@150fbeb, nodeId=0aad560a-faad-4f86-b65c-a7f161bf2639, marsh=BinaryMarshaller [], marshLocJobs=false, daemon=false, p2pEnabled=true, netTimeout=5000, netCompressionLevel=1, sndRetryDelay=1000, sndRetryCnt=3, metricsHistSize=10000, metricsUpdateFreq=2000, metricsExpTime=9223372036854775807, discoSpi=TcpDiscoverySpi [addrRslvr=null, sockTimeout=0, ackTimeout=0, marsh=null, reconCnt=10, reconDelay=2000, maxAckTimeout=600000, soLinger=5, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null, skipAddrsRandomization=false], segPlc=STOP, segResolveAttempts=2, waitForSegOnStart=true, allResolversPassReq=true, segChkFreq=10000, commSpi=TcpCommunicationSpi [connectGate=null, connPlc=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$FirstConnectionPolicy@1390459, chConnPlc=null, enableForcibleNodeKill=false, enableTroubleshootingLog=false, locAddr=null, locHost=null, locPort=47100, locPortRange=100, shmemPort=-1, directBuf=true, directSndBuf=false, idleConnTimeout=600000, connTimeout=5000, maxConnTimeout=600000, reconCnt=10, sockSndBuf=32768, sockRcvBuf=32768, msgQueueLimit=0, slowClientQueueLimit=0, nioSrvr=null, shmemSrv=null, usePairedConnections=false, connectionsPerNode=1, tcpNoDelay=true, filterReachableAddresses=false, ackSndThreshold=32, unackedMsgsBufSize=0, sockWriteTimeout=2000, boundTcpPort=-1, boundTcpShmemPort=-1, selectorsCnt=4, selectorSpins=0, addrRslvr=null, ctxInitLatch=java.util.concurrent.CountDownLatch@156c3cd[Count = 1], stopping=false, metricsLsnr=null], evtSpi=org.apache.ignite.spi.eventstorage.NoopEventStorageSpi@113052e, colSpi=NoopCollisionSpi [], deploySpi=LocalDeploymentSpi [], indexingSpi=org.apache.ignite.spi.indexing.noop.NoopIndexingSpi@45dbe, addrRslvr=null, encryptionSpi=org.apache.ignite.spi.encryption.noop.NoopEncryptionSpi@14658f7, clientMode=false, rebalanceThreadPoolSize=4, rebalanceTimeout=10000, rebalanceBatchesPrefetchCnt=3, rebalanceThrottle=0, rebalanceBatchSize=524288, txCfg=TransactionConfiguration [txSerEnabled=false, dfltIsolation=REPEATABLE_READ, dfltConcurrency=PESSIMISTIC, dfltTxTimeout=0, txTimeoutOnPartitionMapExchange=0, deadlockTimeout=10000, pessimisticTxLogSize=0, pessimisticTxLogLinger=10000, tmLookupClsName=null, txManagerFactory=null, useJtaSync=false], cacheSanityCheckEnabled=true, discoStartupDelay=60000, deployMode=SHARED, p2pMissedCacheSize=100, locHost=null, timeSrvPortBase=31100, timeSrvPortRange=100, failureDetectionTimeout=10000, sysWorkerBlockedTimeout=null, clientFailureDetectionTimeout=30000, metricsLogFreq=60000, hadoopCfg=null, connectorCfg=ConnectorConfiguration [jettyPath=null, host=null, port=11211, noDelay=true, directBuf=false, sndBufSize=32768, rcvBufSize=32768, idleQryCurTimeout=600000, idleQryCurCheckFreq=60000, sndQueueLimit=0, selectorCnt=1, idleTimeout=7000, sslEnabled=false, sslClientAuth=false, sslCtxFactory=null, sslFactory=null, portRange=100, threadPoolSize=8, msgInterceptor=null], odbcCfg=null, warmupClos=null, atomicCfg=AtomicConfiguration [seqReserveSize=1000, cacheMode=PARTITIONED, backups=1, aff=null, grpName=null], classLdr=null, sslCtxFactory=SslContextFactory[keyStoreType=JKS, proto=TLS, keyStoreFile=/home/dsudev/config/keystore.jks, trustStoreFile=/home/dsudev/config/truststore.jks], platformCfg=PlatformConfiguration [], binaryCfg=BinaryConfiguration [idMapper=BinaryBaseIdMapper [isLowerCase=true], nameMapper=BinaryBaseNameMapper [isSimpleName=true], serializer=null, compactFooter=false], memCfg=null, pstCfg=null, dsCfg=DataStorageConfiguration [sysRegionInitSize=41943040, sysRegionMaxSize=104857600, pageSize=4096, concLvl=0, dfltDataRegConf=DataRegionConfiguration [name=default, maxSize=419430400, initSize=104857600, swapPath=null, pageEvictionMode=DISABLED, evictionThreshold=0.9, emptyPagesPoolSize=100, metricsEnabled=false, metricsSubIntervalCount=5, metricsRateTimeInterval=60000, persistenceEnabled=true, checkpointPageBufSize=0, lazyMemoryAllocation=true], dataRegions=null, storagePath=/media/linux_hdd/keymanagementdata, checkpointFreq=180000, lockWaitTime=10000, checkpointThreads=4, checkpointWriteOrder=SEQUENTIAL, walHistSize=20, maxWalArchiveSize=1073741824, walSegments=10, walSegmentSize=67108864, walPath=/media/linux_hdd/keymanagementdata/db/wal, walArchivePath=/media/linux_hdd/keymanagementdata/db/wal/archive, metricsEnabled=false, walMode=LOG_ONLY, walTlbSize=131072, walBuffSize=0, walFlushFreq=2000, walFsyncDelay=1000, walRecordIterBuffSize=67108864, alwaysWriteFullPages=false, fileIOFactory=org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory@1459c75, metricsSubIntervalCnt=5, metricsRateTimeInterval=60000, walAutoArchiveAfterInactivity=-1, writeThrottlingEnabled=false, walCompactionEnabled=false, walCompactionLevel=1, checkpointReadLockTimeout=null, walPageCompression=DISABLED, walPageCompressionLevel=null], activeOnStart=true, autoActivation=true, longQryWarnTimeout=3000, sqlConnCfg=null, cliConnCfg=ClientConnectorConfiguration [host=null, port=10800, portRange=100, sockSndBufSize=0, sockRcvBufSize=0, tcpNoDelay=true, maxOpenCursorsPerConn=128, threadPoolSize=8, idleTimeout=0, handshakeTimeout=10000, jdbcEnabled=true, odbcEnabled=true, thinCliEnabled=true, sslEnabled=false, useIgniteSslCtxFactory=true, sslClientAuth=false, sslCtxFactory=null, thinCliCfg=ThinClientConfiguration [maxActiveTxPerConn=100]], mvccVacuumThreadCnt=2, mvccVacuumFreq=5000, authEnabled=false, failureHnd=null, commFailureRslvr=null] [10:10:57,191][INFO][main][IgniteKernal] Daemon mode: off [10:10:57,193][INFO][main][IgniteKernal] OS: Linux 4.19.0-8-686 i386 [10:10:57,193][INFO][main][IgniteKernal] OS user: root [10:10:57,211][INFO][main][IgniteKernal] PID: 1461 [10:10:57,222][INFO][main][IgniteKernal] Language runtime: Java Platform API Specification ver. 11 [10:10:57,222][INFO][main][IgniteKernal] VM information: OpenJDK Runtime Environment 11.0.6+10-post-Debian-1deb10u1 Debian OpenJDK Server VM 11.0.6+10-post-Debian-1deb10u1 [10:10:57,223][INFO][main][IgniteKernal] VM total memory: 0.48GB [10:10:57,223][INFO][main][IgniteKernal] Remote Management [restart: off, REST: on, JMX (remote: off)] [10:10:57,223][INFO][main][IgniteKernal] Logger: JavaLogger [quiet=true, config=null] [10:10:57,223][INFO][main][IgniteKernal] IGNITE_HOME=/home/dsudev/ignite-master [10:10:57,223][INFO][main][IgniteKernal] VM arguments: [-DIGNITE_HOME=/home/dsudev/ignite-master, -Xms256m, -Xmx512m, -Dfile.encoding=UTF-8, --add-exports=java.base/jdk.internal.misc=ALL-UNNAMED, --add-exports=java.base/sun.nio.ch=ALL-UNNAMED, --add-exports=java.management/com.sun.jmx.mbeanserver=ALL-UNNAMED, --add-exports=jdk.internal.jvmstat/sun.jvmstat.monitor=ALL-UNNAMED, --add-exports=java.base/sun.reflect.generics.reflectiveObjects=ALL-UNNAMED, --illegal-access=permit] [10:10:57,224][INFO][main][IgniteKernal] System cache's DataRegion size is configured to 40 MB. Use DataStorageConfiguration.systemRegionInitialSize property to change the setting. [10:10:57,224][INFO][main][IgniteKernal] Configured caches [in 'sysMemPlc' dataRegion: ['ignite-sys-cache'], in 'default' dataRegion: ['DSU_DataBase']] [10:10:57,224][WARNING][main][IgniteKernal] Peer class loading is enabled (disable it in production for performance and deployment consistency reasons) [10:10:57,224][WARNING][main][IgniteKernal] Please set system property '-Djava.net.preferIPv4Stack=true' to avoid possible problems in mixed environments. [10:10:57,232][INFO][main][IgniteKernal] 3-rd party licenses can be found at: /home/dsudev/ignite-master/libs/licenses [10:10:57,240][WARNING][pub-#22][GridDiagnostic] Initial heap size is 256MB (should be no less than 512MB, use -Xms512m -Xmx512m). [10:10:57,441][INFO][main][IgnitePluginProcessor] Configured plugins: [10:10:57,441][INFO][main][IgnitePluginProcessor] ^-- None [10:10:57,442][INFO][main][IgnitePluginProcessor] [10:10:57,451][INFO][main][FailureProcessor] Configured failure handler: [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]]] [10:10:58,881][INFO][main][TcpCommunicationSpi] Successfully bound communication NIO server to TCP port [port=47100, locHost=0.0.0.0/0.0.0.0, selectorsCnt=4, selectorSpins=0, pairedConn=false] [10:10:58,903][WARNING][main][TcpCommunicationSpi] Message queue limit is set to 0 which may lead to potential OOMEs when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender and receiver sides. [10:10:58,975][WARNING][main][NoopCheckpointSpi] Checkpoints are disabled (to enable configure any GridCheckpointSpi implementation) [10:10:58,999][WARNING][main][GridCollisionManager] Collision resolution is disabled (all jobs will be activated upon arrival). [10:10:59,005][INFO][main][IgniteKernal] Security status [authentication=off, tls/ssl=on] [10:10:59,263][INFO][main][TcpDiscoverySpi] Successfully bound to TCP port [port=47500, localHost=0.0.0.0/0.0.0.0, locNodeId=0aad560a-faad-4f86-b65c-a7f161bf2639] [10:10:59,276][INFO][main][PdsFoldersResolver] Successfully created new persistent storage folder [/media/linux_hdd/keymanagementdata/node00-9aae0abd-bec2-4d53-93c2-cdf34201d256] [10:10:59,276][INFO][main][PdsFoldersResolver] Consistent ID used for local node is [9aae0abd-bec2-4d53-93c2-cdf34201d256] according to persistence data storage folders [10:10:59,277][INFO][main][CacheObjectBinaryProcessorImpl] Resolved directory for serialized binary metadata: /home/dsudev/ignite-master/work/binary_meta/node00-9aae0abd-bec2-4d53-93c2-cdf34201d256 [10:10:59,655][INFO][main][FilePageStoreManager] Resolved page store work directory: /media/linux_hdd/keymanagementdata/node00-9aae0abd-bec2-4d53-93c2-cdf34201d256 [10:10:59,656][INFO][main][FileWriteAheadLogManager] Resolved write ahead log work directory: /media/linux_hdd/keymanagementdata/db/wal/node00-9aae0abd-bec2-4d53-93c2-cdf34201d256 [10:10:59,656][INFO][main][FileWriteAheadLogManager] Resolved write ahead log archive directory: /media/linux_hdd/keymanagementdata/db/wal/archive/node00-9aae0abd-bec2-4d53-93c2-cdf34201d256 [10:10:59,786][INFO][main][FileWriteAheadLogManager] Reached logical end of the segment for file /media/linux_hdd/keymanagementdata/db/wal/node00-9aae0abd-bec2-4d53-93c2-cdf34201d256/0000000000000000.wal [10:10:59,791][INFO][main][FileHandleManagerImpl] Initialized write-ahead log manager [mode=LOG_ONLY] [10:10:59,796][WARNING][main][GridCacheDatabaseSharedManager] DataRegionConfiguration.maxWalArchiveSize instead DataRegionConfiguration.walHistorySize would be used for removing old archive wal files [10:10:59,830][INFO][main][GridCacheDatabaseSharedManager] Configured data regions initialized successfully [total=4] [10:10:59,863][INFO][main][PartitionsEvictManager] Evict partition permits=2 [10:10:59,950][WARNING][main][IgniteH2Indexing] Serialization of Java objects in H2 was enabled. [10:11:00,422][INFO][main][ClientListenerProcessor] Client connector processor has started on TCP port 10800 [10:11:00,530][INFO][main][GridTcpRestProtocol] Command protocol successfully started [name=TCP binary, host=0.0.0.0/0.0.0.0, port=11211] [10:11:00,592][WARNING][main][PlatformProcessorImpl] Binary configuration is automatically initiated, note that binary name mapper is set to BinaryBaseNameMapper [isSimpleName=true] and binary ID mapper is set to BinaryBaseIdMapper [isLowerCase=true] (other nodes must have the same binary name and ID mapper types). [10:11:00,632][INFO][main][IgniteKernal] Non-loopback local IPs: 192.168.10.5, 192.168.100.10, 192.168.111.111, 192.168.12.112, 192.168.15.111, fe80:0:0:0:a00:27ff:fe72:e439%enp0s9 [10:11:00,632][INFO][main][IgniteKernal] Enabled local MACs: 08002706A1F2, 0800270A9679, 08002772E439, 080027BD35B7 [10:11:00,654][INFO][main][GridCacheDatabaseSharedManager] Read checkpoint status [startMarker=null, endMarker=null] [10:11:00,679][INFO][main][PageMemoryImpl] Started page memory [memoryAllocated=100.0 MiB, pages=24814, tableSize=1.9 MiB, checkpointBuffer=100.0 MiB] [10:11:00,683][INFO][main][GridCacheDatabaseSharedManager] Checking memory state [lastValidPos=FileWALPointer [idx=0, fileOff=0, len=0], lastMarked=FileWALPointer [idx=0, fileOff=0, len=0], lastCheckpointId=00000000-0000-0000-0000-000000000000] [10:11:00,759][INFO][main][GridCacheDatabaseSharedManager] Applying lost cache updates since last checkpoint record [lastMarked=FileWALPointer [idx=0, fileOff=0, len=0], lastCheckpointId=00000000-0000-0000-0000-000000000000] [10:11:00,767][INFO][main][GridCacheDatabaseSharedManager] Finished applying WAL changes [updatesApplied=0, time=10 ms] [10:11:00,767][INFO][main][GridCacheProcessor] Restoring partition state for local groups. [10:11:00,768][INFO][main][GridCacheProcessor] Finished restoring partition state for local groups [groupsProcessed=0, partitionsProcessed=0, time=0ms] [10:11:00,866][INFO][main][FilePageStoreManager] Cleanup cache stores [total=1, left=0, cleanFiles=false] [10:11:00,898][INFO][main][PageMemoryImpl] Started page memory [memoryAllocated=100.0 MiB, pages=24814, tableSize=1.9 MiB, checkpointBuffer=100.0 MiB] [10:11:00,898][INFO][main][PageMemoryImpl] Started page memory [memoryAllocated=100.0 MiB, pages=24814, tableSize=1.9 MiB, checkpointBuffer=100.0 MiB] [10:11:00,910][INFO][main][PageMemoryImpl] Started page memory [memoryAllocated=100.0 MiB, pages=24814, tableSize=1.9 MiB, checkpointBuffer=100.0 MiB] [10:11:00,929][INFO][main][GridCacheDatabaseSharedManager] Configured data regions started successfully [total=4] [10:11:00,930][INFO][main][GridCacheDatabaseSharedManager] Starting binary memory restore for: [-2100569601, -510489548] [10:11:01,167][INFO][main][GridCacheDatabaseSharedManager] Read checkpoint status [startMarker=null, endMarker=null] [10:11:01,169][INFO][main][GridCacheDatabaseSharedManager] Checking memory state [lastValidPos=FileWALPointer [idx=0, fileOff=0, len=0], lastMarked=FileWALPointer [idx=0, fileOff=0, len=0], lastCheckpointId=00000000-0000-0000-0000-000000000000] [10:11:01,182][INFO][main][FileWriteAheadLogManager] Resuming logging to WAL segment [file=/media/linux_hdd/keymanagementdata/db/wal/node00-9aae0abd-bec2-4d53-93c2-cdf34201d256/0000000000000000.wal, offset=0, ver=2] [10:11:01,613][INFO][main][GridCacheProcessor] Started cache in recovery mode [name=ignite-sys-cache, id=-2100569601, dataRegionName=sysMemPlc, mode=REPLICATED, atomicity=TRANSACTIONAL, backups=2147483647, mvcc=false] [10:11:01,632][INFO][main][PageMemoryImpl] Started page memory [memoryAllocated=400.0 MiB, pages=99259, tableSize=7.7 MiB, checkpointBuffer=256.0 MiB] [10:11:01,740][INFO][main][GridCacheProcessor] Started cache in recovery mode [name=DSU_DataBase, id=-510489548, dataRegionName=default, mode=PARTITIONED, atomicity=ATOMIC, backups=1, mvcc=false] [10:11:01,753][INFO][main][GridCacheDatabaseSharedManager] Binary recovery performed in 823 ms. [10:11:01,754][INFO][main][GridCacheDatabaseSharedManager] Read checkpoint status [startMarker=null, endMarker=null] [10:11:01,754][INFO][main][GridCacheDatabaseSharedManager] Applying lost cache updates since last checkpoint record [lastMarked=FileWALPointer [idx=0, fileOff=0, len=0], lastCheckpointId=00000000-0000-0000-0000-000000000000] [10:11:01,888][INFO][main][GridCacheDatabaseSharedManager] Finished applying WAL changes [updatesApplied=0, time=127 ms] [10:11:01,888][INFO][main][GridCacheProcessor] Restoring partition state for local groups. [10:11:01,976][INFO][main][GridCacheProcessor] Finished restoring partition state for local groups [groupsProcessed=2, partitionsProcessed=0, time=93ms] [10:11:02,176][INFO][main][TcpDiscoverySpi] Connection check threshold is calculated: 10000 [10:11:02,244][INFO][disco-notifier-worker-#38][MvccProcessorImpl] Assigned mvcc coordinator [crd=MvccCoordinator [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=0], nodeId=0aad560a-faad-4f86-b65c-a7f161bf2639, ver=1599885662229, local=true, initialized=false]] [10:11:02,320][INFO][disco-notifier-worker-#38][DistributedBaselineConfiguration] Baseline parameter 'baselineAutoAdjustEnabled' was changed from 'null' to 'false' [10:11:02,328][INFO][disco-notifier-worker-#38][DistributedBaselineConfiguration] Baseline parameter 'baselineAutoAdjustTimeout' was changed from 'null' to '300000' [10:11:02,604][INFO][main][IgniteKernal] Performance suggestions for grid (fix if possible) [10:11:02,605][INFO][main][IgniteKernal] To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true [10:11:02,605][INFO][main][IgniteKernal] ^-- Decrease number of backups (set 'backups' to 0) [10:11:02,606][INFO][main][IgniteKernal] ^-- Disable fully synchronous writes (set 'writeSynchronizationMode' to PRIMARY_SYNC or FULL_ASYNC) [10:11:02,607][INFO][main][IgniteKernal] ^-- Switch to the most recent 1.8 JVM version [10:11:02,608][INFO][main][IgniteKernal] ^-- Set max direct memory size if getting 'OOME: Direct buffer memory' (add '-XX:MaxDirectMemorySize=<size>[g|G|m|M|k|K]' to JVM options) [10:11:02,608][INFO][main][IgniteKernal] ^-- Disable processing of calls to System.gc() (add '-XX:+DisableExplicitGC' to JVM options) [10:11:02,610][INFO][main][IgniteKernal] Refer to this page for more performance suggestions: https://apacheignite.readme.io/docs/jvm-and-system-tuning [10:11:02,610][INFO][main][IgniteKernal] [10:11:02,611][INFO][main][IgniteKernal] To start Console Management & Monitoring run ignitevisorcmd.{sh|bat} [10:11:02,615][INFO][main][IgniteKernal] Data Regions Configured: [10:11:02,616][INFO][main][IgniteKernal] ^-- default [initSize=100.0 MiB, maxSize=400.0 MiB, persistence=true, lazyMemoryAllocation=true] [10:11:02,616][INFO][main][IgniteKernal] >>> +----------------------------------------------------------------------+ >>> Ignite ver. 2.8.0#20200226-sha1:341b01dfd8abf2d9b01d468ad1bb26dfe84ac4f6 >>> +----------------------------------------------------------------------+ >>> OS name: Linux 4.19.0-8-686 i386 >>> CPU(s): 1 >>> Heap: 0.48GB >>> VM name: 1461@dsudev >>> Local node [ID=0AAD560A-FAAD-4F86-B65C-A7F161BF2639, order=1, >>> clientMode=false] >>> Local node addresses: [192.168.15.111/0:0:0:0:0:0:0:1%lo, >>> 192.168.111.111/127.0.0.1, 192.168.100.10/192.168.10.5, >>> 192.168.10.5/192.168.100.10, 192.168.12.112/192.168.111.111, >>> /192.168.12.112, /192.168.15.111] >>> Local ports: TCP:10800 TCP:11211 TCP:47100 TCP:47500 [10:11:02,623][INFO][main][IgniteKernal] >>> Ignite cluster is not active (limited functionality available). Use control.(sh|bat) script or IgniteCluster interface to activate. [10:11:02,630][INFO][main][GridDiscoveryManager] Topology snapshot [ver=1, locNode=0aad560a, servers=1, clients=0, state=INACTIVE, CPUs=1, offheap=0.39GB, heap=0.48GB] [10:11:02,631][INFO][main][G] Node started : [stage="Configure system pool" (158 ms),stage="Start managers" (1940 ms),stage="Configure binary metadata" (324 ms),stage="Start processors" (1268 ms),stage="Init metastore" (241 ms),stage="Init and start regions" (63 ms),stage="Restore binary memory" (824 ms),stage="Restore logical state" (222 ms),stage="Finish recovery" (183 ms),stage="Join topology" (81 ms),stage="Await transition" (17 ms),stage="Await exchange" (372 ms),stage="Total time" (5693 ms)] [10:11:02,666][INFO][main][GridClusterStateProcessor] Sending activate request with BaselineTopology BaselineTopology [id=0, branchingHash=-173057973, branchingType='New BaselineTopology', baselineNodes=[9aae0abd-bec2-4d53-93c2-cdf34201d256]] [10:11:02,711][INFO][disco-notifier-worker-#38][GridClusterStateProcessor] Received activate request with BaselineTopology[id=0] [10:11:02,713][INFO][disco-notifier-worker-#38][GridClusterStateProcessor] Started state transition: true [10:11:02,724][INFO][exchange-worker-#40][time] Started exchange init [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], crd=true, evt=DISCOVERY_CUSTOM_EVT, evtNode=0aad560a-faad-4f86-b65c-a7f161bf2639, customEvt=ChangeGlobalStateMessage [id=86bbd908471-259db0fa-9d36-40c3-ad9f-b8359727dc28, reqId=e5a2e4ff-c7aa-4924-bc83-7f86c3f16ea1, initiatingNodeId=0aad560a-faad-4f86-b65c-a7f161bf2639, activate=true, baselineTopology=BaselineTopology [id=0, branchingHash=-173057973, branchingType='New BaselineTopology', baselineNodes=[9aae0abd-bec2-4d53-93c2-cdf34201d256]], forceChangeBaselineTopology=false, timestamp=1599885662666], allowMerge=false, exchangeFreeSwitch=false] [10:11:02,725][INFO][exchange-worker-#40][GridDhtPartitionsExchangeFuture] Start activation process [nodeId=0aad560a-faad-4f86-b65c-a7f161bf2639, client=false, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1]] [10:11:02,726][INFO][exchange-worker-#40][FilePageStoreManager] Resolved page store work directory: /media/linux_hdd/keymanagementdata/node00-9aae0abd-bec2-4d53-93c2-cdf34201d256 [10:11:02,730][INFO][exchange-worker-#40][FileWriteAheadLogManager] Resuming logging to WAL segment [file=/media/linux_hdd/keymanagementdata/db/wal/node00-9aae0abd-bec2-4d53-93c2-cdf34201d256/0000000000000000.wal, offset=13069, ver=2] [10:11:02,732][INFO][exchange-worker-#40][GridClusterStateProcessor] Writing BaselineTopology[id=0] [10:11:02,768][INFO][exchange-worker-#40][GridCacheDatabaseSharedManager] Finish recovery performed in 42 ms. [10:11:02,770][INFO][exchange-worker-#40][msg] Components activation performed in 44 ms. [10:11:02,840][INFO][exchange-worker-#40][GridCacheProcessor] Finished recovery for cache [cache=DSU_DataBase, grp=DSU_DataBase, startVer=AffinityTopologyVersion [topVer=1, minorTopVer=1]] [10:11:02,840][INFO][sys-#49][GridCacheProcessor] Finished recovery for cache [cache=ignite-sys-cache, grp=ignite-sys-cache, startVer=AffinityTopologyVersion [topVer=1, minorTopVer=1]] [10:11:02,886][INFO][exchange-worker-#40][GridAffinityAssignmentCache] Local node affinity assignment distribution is not ideal [cache=DSU_DataBase, expectedPrimary=1024.00, actualPrimary=1024, expectedBackups=1024.00, actualBackups=0, warningThreshold=50.00%] [10:11:02,909][INFO][exchange-worker-#40][GridDhtPartitionsExchangeFuture] Successfully activated caches [nodeId=0aad560a-faad-4f86-b65c-a7f161bf2639, client=false, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1]] [10:11:02,922][INFO][exchange-worker-#40][GridDhtPartitionsExchangeFuture] Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], waitTime=0ms, futInfo=NA, mode=LOCAL] [10:11:03,485][INFO][exchange-worker-#40][GridDhtPartitionsExchangeFuture] finishExchangeOnCoordinator [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], resVer=AffinityTopologyVersion [topVer=1, minorTopVer=1]] [10:11:03,513][INFO][db-checkpoint-thread-#54][GridCacheDatabaseSharedManager] Checkpoint started [checkpointId=d6caae21-b222-4a61-9956-671e24030c13, startPtr=FileWALPointer [idx=0, fileOff=142444, len=21425], checkpointBeforeLockTime=87ms, checkpointLockWait=0ms, checkpointListenersExecuteTime=56ms, checkpointLockHoldTime=80ms, walCpRecordFsyncDuration=9ms, writeCheckpointEntryDuration=10ms, splitAndSortCpPagesDuration=0ms, pages=20, reason='node started'] [10:11:03,545][INFO][db-checkpoint-thread-#54][GridCacheDatabaseSharedManager] Checkpoint finished [cpId=d6caae21-b222-4a61-9956-671e24030c13, pages=20, markPos=FileWALPointer [idx=0, fileOff=142444, len=21425], walSegmentsCleared=0, walSegmentsCovered=[], markDuration=109ms, pagesWrite=5ms, fsync=26ms, total=227ms] [10:11:03,667][INFO][disco-notifier-worker-#38][GridClusterStateProcessor] Received state change finish message: true [10:11:03,669][INFO][exchange-worker-#40][GridDhtPartitionsExchangeFuture] Finish exchange future [startVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], resVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], err=null, rebalanced=true, wasRebalanced=false] [10:11:03,752][INFO][exchange-worker-#40][GridCacheProcessor] Finish proxy initialization, cacheName=ignite-sys-cache, localNodeId=0aad560a-faad-4f86-b65c-a7f161bf2639 [10:11:03,752][INFO][exchange-worker-#40][GridCacheProcessor] Finish proxy initialization, cacheName=DSU_DataBase, localNodeId=0aad560a-faad-4f86-b65c-a7f161bf2639 [10:11:03,766][INFO][sys-#53][DistributedBaselineConfiguration] Baseline auto-adjust is 'disabled' with timeout='300000' ms [10:11:03,767][INFO][sys-#53][GridClusterStateProcessor] Successfully performed final activation steps [nodeId=0aad560a-faad-4f86-b65c-a7f161bf2639, client=false, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1]] [10:11:03,848][INFO][exchange-worker-#40][GridDhtPartitionsExchangeFuture] Completed partition exchange [localNode=0aad560a-faad-4f86-b65c-a7f161bf2639, exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], evt=DISCOVERY_CUSTOM_EVT, evtNode=TcpDiscoveryNode [id=0aad560a-faad-4f86-b65c-a7f161bf2639, consistentId=9aae0abd-bec2-4d53-93c2-cdf34201d256, addrs=ArrayList [0:0:0:0:0:0:0:1%lo, 127.0.0.1, 192.168.10.5, 192.168.100.10, 192.168.111.111, 192.168.12.112, 192.168.15.111], sockAddrs=HashSet [/192.168.111.111:47500, /192.168.15.111:47500, /192.168.12.112:47500, /192.168.10.5:47500, /0:0:0:0:0:0:0:1%lo:47500, /127.0.0.1:47500, /192.168.100.10:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1599885662175, loc=true, ver=2.8.0#20200226-sha1:341b01df, isClient=false], done=true, newCrdFut=null], topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1]] [10:11:03,849][INFO][exchange-worker-#40][GridDhtPartitionsExchangeFuture] Exchange timings [startVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], resVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], stage="Waiting in exchange queue" (1 ms), stage="Exchange parameters initialization" (2 ms), stage="Update caches registry" (57 ms), stage="Start caches" (67 ms), stage="Affinity initialization on cache group start" (59 ms), stage="Determine exchange type" (9 ms), stage="Preloading notification" (0 ms), stage="WAL history reservation" (1 ms), stage="Wait partitions release" (5 ms), stage="Restore partition states" (22 ms), stage="After states restored callback" (536 ms), stage="Waiting for all single messages" (0 ms), stage="Affinity recalculation (crd)" (0 ms), stage="Collect update counters and create affinity messages" (0 ms), stage="Validate partitions states" (8 ms), stage="Assign partitions states" (54 ms), stage="Ideal affinity diff calculation (enforced)" (41 ms), stage="Apply update counters" (4 ms), stage="Full message preparing" (70 ms), stage="Full message sending" (0 ms), stage="State finish message sending" (2 ms), stage="Exchange done" (180 ms), stage="Total time" (1118 ms)] [10:11:03,849][INFO][exchange-worker-#40][GridDhtPartitionsExchangeFuture] Exchange longest local stages [startVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], resVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], stage="Affinity initialization on cache group start [grp=DSU_DataBase]" (59 ms) (parent=Affinity initialization on cache group start), stage="Affinity initialization on cache group start [grp=ignite-sys-cache]" (37 ms) (parent=Affinity initialization on cache group start), stage="Affinity centralized initialization (crd) [grp=ignite-sys-cache, crd=true]" (5 ms) (parent=Determine exchange type), stage="Affinity centralized initialization (crd) [grp=DSU_DataBase, crd=true]" (5 ms) (parent=Determine exchange type), stage="Restore partition states [grp=DSU_DataBase]" (22 ms) (parent=Restore partition states), stage="Restore partition states [grp=ignite-sys-cache]" (14 ms) (parent=Restore partition states), stage="Affinity recalculation (partitions availability) [grp=DSU_DataBase]" (40 ms) (parent=Ideal affinity diff calculation (enforced)), stage="Affinity recalculation (partitions availability) [grp=ignite-sys-cache]" (14 ms) (parent=Ideal affinity diff calculation (enforced))] [10:11:03,849][INFO][exchange-worker-#40][time] Finished exchange init [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], crd=true] [10:11:03,918][INFO][exchange-worker-#40][GridCachePartitionExchangeManager] Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion [topVer=1, minorTopVer=1], force=false, evt=DISCOVERY_CUSTOM_EVT, node=0aad560a-faad-4f86-b65c-a7f161bf2639] [10:11:05,808][INFO][disco-notifier-worker-#38][DistributedBaselineConfiguration] Baseline parameter 'baselineAutoAdjustTimeout' was changed from '300000' to '5' [10:11:05,817][INFO][disco-notifier-worker-#38][DistributedBaselineConfiguration] Baseline parameter 'baselineAutoAdjustEnabled' was changed from 'false' to 'true' [10:12:02,608][INFO][grid-timeout-worker-#23][IgniteKernal] Metrics for local node (to disable set 'metricsLogFrequency' to 0) ^-- Node [id=0aad560a, uptime=00:01:00.002] ^-- H/N/C [hosts=1, nodes=1, CPUs=1] ^-- CPU [cur=-100%, avg=-95.83%, GC=0%] ^-- PageMemory [pages=20] ^-- Heap [used=32MB, free=93.48%, comm=247MB] ^-- Off-heap [used=0MB, free=99.99%, comm=600MB] ^-- sysMemPlc region [used=0MB, free=99.98%, comm=100MB] ^-- default region [used=0MB, free=100%, comm=400MB] ^-- metastoreMemPlc region [used=0MB, free=99.96%, comm=0MB] ^-- TxLog region [used=0MB, free=100%, comm=100MB] ^-- Ignite persistence [used=0MB] ^-- sysMemPlc region [used=0MB] ^-- default region [used=0MB] ^-- metastoreMemPlc region [used=0MB] ^-- TxLog region [used=0MB] ^-- Outbound messages queue [size=0] ^-- Public thread pool [active=0, idle=0, qSize=0] ^-- System thread pool [active=0, idle=8, qSize=0] [10:12:12,305][INFO][ignite-update-notifier-timer][GridUpdateNotifier] Update status is not available. [10:13:02,616][INFO][grid-timeout-worker-#23][IgniteKernal] Metrics for local node (to disable set 'metricsLogFrequency' to 0) ^-- Node [id=0aad560a, uptime=00:02:00.020] ^-- H/N/C [hosts=1, nodes=1, CPUs=1] ^-- CPU [cur=-100%, avg=-97.78%, GC=0%] ^-- PageMemory [pages=20] ^-- Heap [used=35MB, free=92.91%, comm=247MB] ^-- Off-heap [used=0MB, free=99.99%, comm=600MB] ^-- sysMemPlc region [used=0MB, free=99.98%, comm=100MB] ^-- default region [used=0MB, free=100%, comm=400MB] ^-- metastoreMemPlc region [used=0MB, free=99.96%, comm=0MB] ^-- TxLog region [used=0MB, free=100%, comm=100MB] ^-- Ignite persistence [used=0MB] ^-- sysMemPlc region [used=0MB] ^-- default region [used=0MB] ^-- metastoreMemPlc region [used=0MB] ^-- TxLog region [used=0MB] ^-- Outbound messages queue [size=0] ^-- Public thread pool [active=0, idle=0, qSize=0] ^-- System thread pool [active=0, idle=6, qSize=0] [10:14:02,624][INFO][grid-timeout-worker-#23][IgniteKernal] Metrics for local node (to disable set 'metricsLogFrequency' to 0) ^-- Node [id=0aad560a, uptime=00:03:00.025] ^-- H/N/C [hosts=1, nodes=1, CPUs=1] ^-- CPU [cur=-100%, avg=-98.63%, GC=0%] ^-- PageMemory [pages=20] ^-- Heap [used=36MB, free=92.58%, comm=247MB] ^-- Off-heap [used=0MB, free=99.99%, comm=600MB] ^-- sysMemPlc region [used=0MB, free=99.98%, comm=100MB] ^-- default region [used=0MB, free=100%, comm=400MB] ^-- metastoreMemPlc region [used=0MB, free=99.96%, comm=0MB] ^-- TxLog region [used=0MB, free=100%, comm=100MB] ^-- Ignite persistence [used=0MB] ^-- sysMemPlc region [used=0MB] ^-- default region [used=0MB] ^-- metastoreMemPlc region [used=0MB] ^-- TxLog region [used=0MB] ^-- Outbound messages queue [size=0] ^-- Public thread pool [active=0, idle=0, qSize=0] ^-- System thread pool [active=0, idle=6, qSize=0] [10:14:03,465][SEVERE][db-checkpoint-thread-#54][] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=CRITICAL_ERROR, err=class o.a.i.i.processors.cache.persistence.StorageException: Failed to write checkpoint entry [ptr=FileWALPointer [idx=0, fileOff=183931, len=21409], cpTs=1599885843404, cpId=78332869-fedc-4687-9202-180b8de1c2b8, type=START]]] class org.apache.ignite.internal.processors.cache.persistence.StorageException: Failed to write checkpoint entry [ptr=FileWALPointer [idx=0, fileOff=183931, len=21409], cpTs=1599885843404, cpId=78332869-fedc-4687-9202-180b8de1c2b8, type=START] at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.writeCheckpointEntry(GridCacheDatabaseSharedManager.java:3248) at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.markCheckpointBegin(GridCacheDatabaseSharedManager.java:4220) at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.doCheckpoint(GridCacheDatabaseSharedManager.java:3738) at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.body(GridCacheDatabaseSharedManager.java:3623) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: java.nio.file.NoSuchFileException: /media/linux_hdd/keymanagementdata/node00-9aae0abd-bec2-4d53-93c2-cdf34201d256/cp/1599885843404-78332869-fedc-4687-9202-180b8de1c2b8-START.bin.tmp at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) at java.base/sun.nio.fs.UnixFileSystemProvider.newAsynchronousFileChannel(UnixFileSystemProvider.java:201) at java.base/java.nio.channels.AsynchronousFileChannel.open(AsynchronousFileChannel.java:253) at java.base/java.nio.channels.AsynchronousFileChannel.open(AsynchronousFileChannel.java:311) at org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIO.<init>(AsyncFileIO.java:66) at org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory.create(AsyncFileIOFactory.java:44) at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.writeCheckpointEntry(GridCacheDatabaseSharedManager.java:3231) ... 5 more [10:14:03,499][WARNING][db-checkpoint-thread-#54][CacheDiagnosticManager] Page locks dump: Thread=[name=db-checkpoint-thread-#54, id=84], state=RUNNABLE Locked pages = [] Locked pages log: name=db-checkpoint-thread-#54 time=(1599885843482, 2020-09-12 10:14:03.482) Thread=[name=dms-writer-thread-#45, id=75], state=WAITING Locked pages = [] Locked pages log: name=dms-writer-thread-#45 time=(1599885843482, 2020-09-12 10:14:03.482) Thread=[name=exchange-worker-#40, id=69], state=TIMED_WAITING Locked pages = [] Locked pages log: name=exchange-worker-#40 time=(1599885843482, 2020-09-12 10:14:03.482) Thread=[name=main, id=1], state=RUNNABLE Locked pages = [] Locked pages log: name=main time=(1599885843482, 2020-09-12 10:14:03.482) [10:14:03,499][SEVERE][db-checkpoint-thread-#54][] JVM will be halted immediately due to the failure: [failureCtx=FailureContext [type=CRITICAL_ERROR, err=class o.a.i.i.processors.cache.persistence.StorageException: Failed to write checkpoint entry [ptr=FileWALPointer [idx=0, fileOff=183931, len=21409], cpTs=1599885843404, cpId=78332869-fedc-4687-9202-180b8de1c2b8, type=START]]] -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
