Hello! This will happen when this file is deleted while the instance is running. Not sure who deleted it. Maybe you tried to start another node with the same consistent id in the background?
You should avoid calling setActive() every time since it will lead to data loss. Regards, -- Ilya Kasnacheev чт, 7 янв. 2021 г. в 14:04, rakshita04 <rakshita.chaudh...@siemens.com>: > These are the full set of logs, if it helps- > [10:10:56,860][WARNING][main][G] Ignite work directory is not provided, > automatically resolved to: /home/dsudev/ignite-master/work > [10:10:56,873][WARNING][main][G] Consistent ID is not set, it is > recommended > to set consistent ID for production clusters (use > IgniteConfiguration.setConsistentId property) > [10:10:57,103][INFO][main][IgniteKernal] > > >>> __________ ________________ > >>> / _/ ___/ |/ / _/_ __/ __/ > >>> _/ // (7 7 // / / / / _/ > >>> /___/\___/_/|_/___/ /_/ /___/ > >>> > >>> ver. 2.8.0#20200226-sha1:341b01df > >>> 2020 Copyright(C) Apache Software Foundation > >>> > >>> Ignite documentation: http://ignite.apache.org > > [10:10:57,134][INFO][main][IgniteKernal] Config URL: n/a > [10:10:57,190][INFO][main][IgniteKernal] IgniteConfiguration > [igniteInstanceName=null, pubPoolSize=8, svcPoolSize=8, callbackPoolSize=8, > stripedPoolSize=8, sysPoolSize=8, mgmtPoolSize=4, igfsPoolSize=1, > dataStreamerPoolSize=8, utilityCachePoolSize=8, > utilityCacheKeepAliveTime=60000, p2pPoolSize=2, qryPoolSize=8, > sqlQryHistSize=1000, dfltQryTimeout=0, > igniteHome=/home/dsudev/ignite-master, > igniteWorkDir=/home/dsudev/ignite-master/work, > mbeanSrv=com.sun.jmx.mbeanserver.JmxMBeanServer@150fbeb, > nodeId=0aad560a-faad-4f86-b65c-a7f161bf2639, marsh=BinaryMarshaller [], > marshLocJobs=false, daemon=false, p2pEnabled=true, netTimeout=5000, > netCompressionLevel=1, sndRetryDelay=1000, sndRetryCnt=3, > metricsHistSize=10000, metricsUpdateFreq=2000, > metricsExpTime=9223372036854775807, discoSpi=TcpDiscoverySpi > [addrRslvr=null, sockTimeout=0, ackTimeout=0, marsh=null, reconCnt=10, > reconDelay=2000, maxAckTimeout=600000, soLinger=5, forceSrvMode=false, > clientReconnectDisabled=false, internalLsnr=null, > skipAddrsRandomization=false], segPlc=STOP, segResolveAttempts=2, > waitForSegOnStart=true, allResolversPassReq=true, segChkFreq=10000, > commSpi=TcpCommunicationSpi [connectGate=null, > > connPlc=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$FirstConnectionPolicy@1390459 > , > chConnPlc=null, enableForcibleNodeKill=false, > enableTroubleshootingLog=false, locAddr=null, locHost=null, locPort=47100, > locPortRange=100, shmemPort=-1, directBuf=true, directSndBuf=false, > idleConnTimeout=600000, connTimeout=5000, maxConnTimeout=600000, > reconCnt=10, sockSndBuf=32768, sockRcvBuf=32768, msgQueueLimit=0, > slowClientQueueLimit=0, nioSrvr=null, shmemSrv=null, > usePairedConnections=false, connectionsPerNode=1, tcpNoDelay=true, > filterReachableAddresses=false, ackSndThreshold=32, unackedMsgsBufSize=0, > sockWriteTimeout=2000, boundTcpPort=-1, boundTcpShmemPort=-1, > selectorsCnt=4, selectorSpins=0, addrRslvr=null, > ctxInitLatch=java.util.concurrent.CountDownLatch@156c3cd[Count = 1], > stopping=false, metricsLsnr=null], > evtSpi=org.apache.ignite.spi.eventstorage.NoopEventStorageSpi@113052e, > colSpi=NoopCollisionSpi [], deploySpi=LocalDeploymentSpi [], > indexingSpi=org.apache.ignite.spi.indexing.noop.NoopIndexingSpi@45dbe, > addrRslvr=null, > > encryptionSpi=org.apache.ignite.spi.encryption.noop.NoopEncryptionSpi@14658f7 > , > clientMode=false, rebalanceThreadPoolSize=4, rebalanceTimeout=10000, > rebalanceBatchesPrefetchCnt=3, rebalanceThrottle=0, > rebalanceBatchSize=524288, txCfg=TransactionConfiguration > [txSerEnabled=false, dfltIsolation=REPEATABLE_READ, > dfltConcurrency=PESSIMISTIC, dfltTxTimeout=0, > txTimeoutOnPartitionMapExchange=0, deadlockTimeout=10000, > pessimisticTxLogSize=0, pessimisticTxLogLinger=10000, tmLookupClsName=null, > txManagerFactory=null, useJtaSync=false], cacheSanityCheckEnabled=true, > discoStartupDelay=60000, deployMode=SHARED, p2pMissedCacheSize=100, > locHost=null, timeSrvPortBase=31100, timeSrvPortRange=100, > failureDetectionTimeout=10000, sysWorkerBlockedTimeout=null, > clientFailureDetectionTimeout=30000, metricsLogFreq=60000, hadoopCfg=null, > connectorCfg=ConnectorConfiguration [jettyPath=null, host=null, port=11211, > noDelay=true, directBuf=false, sndBufSize=32768, rcvBufSize=32768, > idleQryCurTimeout=600000, idleQryCurCheckFreq=60000, sndQueueLimit=0, > selectorCnt=1, idleTimeout=7000, sslEnabled=false, sslClientAuth=false, > sslCtxFactory=null, sslFactory=null, portRange=100, threadPoolSize=8, > msgInterceptor=null], odbcCfg=null, warmupClos=null, > atomicCfg=AtomicConfiguration [seqReserveSize=1000, cacheMode=PARTITIONED, > backups=1, aff=null, grpName=null], classLdr=null, > sslCtxFactory=SslContextFactory[keyStoreType=JKS, proto=TLS, > keyStoreFile=/home/dsudev/config/keystore.jks, > trustStoreFile=/home/dsudev/config/truststore.jks], > platformCfg=PlatformConfiguration [], binaryCfg=BinaryConfiguration > [idMapper=BinaryBaseIdMapper [isLowerCase=true], > nameMapper=BinaryBaseNameMapper [isSimpleName=true], serializer=null, > compactFooter=false], memCfg=null, pstCfg=null, > dsCfg=DataStorageConfiguration [sysRegionInitSize=41943040, > sysRegionMaxSize=104857600, pageSize=4096, concLvl=0, > dfltDataRegConf=DataRegionConfiguration [name=default, maxSize=419430400, > initSize=104857600, swapPath=null, pageEvictionMode=DISABLED, > evictionThreshold=0.9, emptyPagesPoolSize=100, metricsEnabled=false, > metricsSubIntervalCount=5, metricsRateTimeInterval=60000, > persistenceEnabled=true, checkpointPageBufSize=0, > lazyMemoryAllocation=true], dataRegions=null, > storagePath=/media/linux_hdd/keymanagementdata, checkpointFreq=180000, > lockWaitTime=10000, checkpointThreads=4, checkpointWriteOrder=SEQUENTIAL, > walHistSize=20, maxWalArchiveSize=1073741824, walSegments=10, > walSegmentSize=67108864, walPath=/media/linux_hdd/keymanagementdata/db/wal, > walArchivePath=/media/linux_hdd/keymanagementdata/db/wal/archive, > metricsEnabled=false, walMode=LOG_ONLY, walTlbSize=131072, walBuffSize=0, > walFlushFreq=2000, walFsyncDelay=1000, walRecordIterBuffSize=67108864, > alwaysWriteFullPages=false, > > fileIOFactory=org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory@1459c75 > , > metricsSubIntervalCnt=5, metricsRateTimeInterval=60000, > walAutoArchiveAfterInactivity=-1, writeThrottlingEnabled=false, > walCompactionEnabled=false, walCompactionLevel=1, > checkpointReadLockTimeout=null, walPageCompression=DISABLED, > walPageCompressionLevel=null], activeOnStart=true, autoActivation=true, > longQryWarnTimeout=3000, sqlConnCfg=null, > cliConnCfg=ClientConnectorConfiguration [host=null, port=10800, > portRange=100, sockSndBufSize=0, sockRcvBufSize=0, tcpNoDelay=true, > maxOpenCursorsPerConn=128, threadPoolSize=8, idleTimeout=0, > handshakeTimeout=10000, jdbcEnabled=true, odbcEnabled=true, > thinCliEnabled=true, sslEnabled=false, useIgniteSslCtxFactory=true, > sslClientAuth=false, sslCtxFactory=null, thinCliCfg=ThinClientConfiguration > [maxActiveTxPerConn=100]], mvccVacuumThreadCnt=2, mvccVacuumFreq=5000, > authEnabled=false, failureHnd=null, commFailureRslvr=null] > [10:10:57,191][INFO][main][IgniteKernal] Daemon mode: off > [10:10:57,193][INFO][main][IgniteKernal] OS: Linux 4.19.0-8-686 i386 > [10:10:57,193][INFO][main][IgniteKernal] OS user: root > [10:10:57,211][INFO][main][IgniteKernal] PID: 1461 > [10:10:57,222][INFO][main][IgniteKernal] Language runtime: Java Platform > API > Specification ver. 11 > [10:10:57,222][INFO][main][IgniteKernal] VM information: OpenJDK Runtime > Environment 11.0.6+10-post-Debian-1deb10u1 Debian OpenJDK Server VM > 11.0.6+10-post-Debian-1deb10u1 > [10:10:57,223][INFO][main][IgniteKernal] VM total memory: 0.48GB > [10:10:57,223][INFO][main][IgniteKernal] Remote Management [restart: off, > REST: on, JMX (remote: off)] > [10:10:57,223][INFO][main][IgniteKernal] Logger: JavaLogger [quiet=true, > config=null] > [10:10:57,223][INFO][main][IgniteKernal] > IGNITE_HOME=/home/dsudev/ignite-master > [10:10:57,223][INFO][main][IgniteKernal] VM arguments: > [-DIGNITE_HOME=/home/dsudev/ignite-master, -Xms256m, -Xmx512m, > -Dfile.encoding=UTF-8, > --add-exports=java.base/jdk.internal.misc=ALL-UNNAMED, > --add-exports=java.base/sun.nio.ch=ALL-UNNAMED, > --add-exports=java.management/com.sun.jmx.mbeanserver=ALL-UNNAMED, > --add-exports=jdk.internal.jvmstat/sun.jvmstat.monitor=ALL-UNNAMED, > --add-exports=java.base/sun.reflect.generics.reflectiveObjects=ALL-UNNAMED, > --illegal-access=permit] > [10:10:57,224][INFO][main][IgniteKernal] System cache's DataRegion size is > configured to 40 MB. Use DataStorageConfiguration.systemRegionInitialSize > property to change the setting. > [10:10:57,224][INFO][main][IgniteKernal] Configured caches [in 'sysMemPlc' > dataRegion: ['ignite-sys-cache'], in 'default' dataRegion: > ['DSU_DataBase']] > [10:10:57,224][WARNING][main][IgniteKernal] Peer class loading is enabled > (disable it in production for performance and deployment consistency > reasons) > [10:10:57,224][WARNING][main][IgniteKernal] Please set system property > '-Djava.net.preferIPv4Stack=true' to avoid possible problems in mixed > environments. > [10:10:57,232][INFO][main][IgniteKernal] 3-rd party licenses can be found > at: /home/dsudev/ignite-master/libs/licenses > [10:10:57,240][WARNING][pub-#22][GridDiagnostic] Initial heap size is 256MB > (should be no less than 512MB, use -Xms512m -Xmx512m). > [10:10:57,441][INFO][main][IgnitePluginProcessor] Configured plugins: > [10:10:57,441][INFO][main][IgnitePluginProcessor] ^-- None > [10:10:57,442][INFO][main][IgnitePluginProcessor] > [10:10:57,451][INFO][main][FailureProcessor] Configured failure handler: > [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, > super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet > [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]]] > [10:10:58,881][INFO][main][TcpCommunicationSpi] Successfully bound > communication NIO server to TCP port [port=47100, locHost=0.0.0.0/0.0.0.0, > selectorsCnt=4, selectorSpins=0, pairedConn=false] > [10:10:58,903][WARNING][main][TcpCommunicationSpi] Message queue limit is > set to 0 which may lead to potential OOMEs when running cache operations in > FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender and > receiver sides. > [10:10:58,975][WARNING][main][NoopCheckpointSpi] Checkpoints are disabled > (to enable configure any GridCheckpointSpi implementation) > [10:10:58,999][WARNING][main][GridCollisionManager] Collision resolution is > disabled (all jobs will be activated upon arrival). > [10:10:59,005][INFO][main][IgniteKernal] Security status > [authentication=off, tls/ssl=on] > [10:10:59,263][INFO][main][TcpDiscoverySpi] Successfully bound to TCP port > [port=47500, localHost=0.0.0.0/0.0.0.0, > locNodeId=0aad560a-faad-4f86-b65c-a7f161bf2639] > [10:10:59,276][INFO][main][PdsFoldersResolver] Successfully created new > persistent storage folder > > [/media/linux_hdd/keymanagementdata/node00-9aae0abd-bec2-4d53-93c2-cdf34201d256] > [10:10:59,276][INFO][main][PdsFoldersResolver] Consistent ID used for local > node is [9aae0abd-bec2-4d53-93c2-cdf34201d256] according to persistence > data > storage folders > [10:10:59,277][INFO][main][CacheObjectBinaryProcessorImpl] Resolved > directory for serialized binary metadata: > > /home/dsudev/ignite-master/work/binary_meta/node00-9aae0abd-bec2-4d53-93c2-cdf34201d256 > [10:10:59,655][INFO][main][FilePageStoreManager] Resolved page store work > directory: > > /media/linux_hdd/keymanagementdata/node00-9aae0abd-bec2-4d53-93c2-cdf34201d256 > [10:10:59,656][INFO][main][FileWriteAheadLogManager] Resolved write ahead > log work directory: > > /media/linux_hdd/keymanagementdata/db/wal/node00-9aae0abd-bec2-4d53-93c2-cdf34201d256 > [10:10:59,656][INFO][main][FileWriteAheadLogManager] Resolved write ahead > log archive directory: > > /media/linux_hdd/keymanagementdata/db/wal/archive/node00-9aae0abd-bec2-4d53-93c2-cdf34201d256 > [10:10:59,786][INFO][main][FileWriteAheadLogManager] Reached logical end of > the segment for file > > /media/linux_hdd/keymanagementdata/db/wal/node00-9aae0abd-bec2-4d53-93c2-cdf34201d256/0000000000000000.wal > [10:10:59,791][INFO][main][FileHandleManagerImpl] Initialized write-ahead > log manager [mode=LOG_ONLY] > [10:10:59,796][WARNING][main][GridCacheDatabaseSharedManager] > DataRegionConfiguration.maxWalArchiveSize instead > DataRegionConfiguration.walHistorySize would be used for removing old > archive wal files > [10:10:59,830][INFO][main][GridCacheDatabaseSharedManager] Configured data > regions initialized successfully [total=4] > [10:10:59,863][INFO][main][PartitionsEvictManager] Evict partition > permits=2 > [10:10:59,950][WARNING][main][IgniteH2Indexing] Serialization of Java > objects in H2 was enabled. > [10:11:00,422][INFO][main][ClientListenerProcessor] Client connector > processor has started on TCP port 10800 > [10:11:00,530][INFO][main][GridTcpRestProtocol] Command protocol > successfully started [name=TCP binary, host=0.0.0.0/0.0.0.0, port=11211] > [10:11:00,592][WARNING][main][PlatformProcessorImpl] Binary configuration > is > automatically initiated, note that binary name mapper is set to > BinaryBaseNameMapper [isSimpleName=true] and binary ID mapper is set to > BinaryBaseIdMapper [isLowerCase=true] (other nodes must have the same > binary > name and ID mapper types). > [10:11:00,632][INFO][main][IgniteKernal] Non-loopback local IPs: > 192.168.10.5, 192.168.100.10, 192.168.111.111, 192.168.12.112, > 192.168.15.111, fe80:0:0:0:a00:27ff:fe72:e439%enp0s9 > [10:11:00,632][INFO][main][IgniteKernal] Enabled local MACs: 08002706A1F2, > 0800270A9679, 08002772E439, 080027BD35B7 > [10:11:00,654][INFO][main][GridCacheDatabaseSharedManager] Read checkpoint > status [startMarker=null, endMarker=null] > [10:11:00,679][INFO][main][PageMemoryImpl] Started page memory > [memoryAllocated=100.0 MiB, pages=24814, tableSize=1.9 MiB, > checkpointBuffer=100.0 MiB] > [10:11:00,683][INFO][main][GridCacheDatabaseSharedManager] Checking memory > state [lastValidPos=FileWALPointer [idx=0, fileOff=0, len=0], > lastMarked=FileWALPointer [idx=0, fileOff=0, len=0], > lastCheckpointId=00000000-0000-0000-0000-000000000000] > [10:11:00,759][INFO][main][GridCacheDatabaseSharedManager] Applying lost > cache updates since last checkpoint record [lastMarked=FileWALPointer > [idx=0, fileOff=0, len=0], > lastCheckpointId=00000000-0000-0000-0000-000000000000] > [10:11:00,767][INFO][main][GridCacheDatabaseSharedManager] Finished > applying > WAL changes [updatesApplied=0, time=10 ms] > [10:11:00,767][INFO][main][GridCacheProcessor] Restoring partition state > for > local groups. > [10:11:00,768][INFO][main][GridCacheProcessor] Finished restoring partition > state for local groups [groupsProcessed=0, partitionsProcessed=0, time=0ms] > [10:11:00,866][INFO][main][FilePageStoreManager] Cleanup cache stores > [total=1, left=0, cleanFiles=false] > [10:11:00,898][INFO][main][PageMemoryImpl] Started page memory > [memoryAllocated=100.0 MiB, pages=24814, tableSize=1.9 MiB, > checkpointBuffer=100.0 MiB] > [10:11:00,898][INFO][main][PageMemoryImpl] Started page memory > [memoryAllocated=100.0 MiB, pages=24814, tableSize=1.9 MiB, > checkpointBuffer=100.0 MiB] > [10:11:00,910][INFO][main][PageMemoryImpl] Started page memory > [memoryAllocated=100.0 MiB, pages=24814, tableSize=1.9 MiB, > checkpointBuffer=100.0 MiB] > [10:11:00,929][INFO][main][GridCacheDatabaseSharedManager] Configured data > regions started successfully [total=4] > [10:11:00,930][INFO][main][GridCacheDatabaseSharedManager] Starting binary > memory restore for: [-2100569601, -510489548] > [10:11:01,167][INFO][main][GridCacheDatabaseSharedManager] Read checkpoint > status [startMarker=null, endMarker=null] > [10:11:01,169][INFO][main][GridCacheDatabaseSharedManager] Checking memory > state [lastValidPos=FileWALPointer [idx=0, fileOff=0, len=0], > lastMarked=FileWALPointer [idx=0, fileOff=0, len=0], > lastCheckpointId=00000000-0000-0000-0000-000000000000] > [10:11:01,182][INFO][main][FileWriteAheadLogManager] Resuming logging to > WAL > segment > > [file=/media/linux_hdd/keymanagementdata/db/wal/node00-9aae0abd-bec2-4d53-93c2-cdf34201d256/0000000000000000.wal, > offset=0, ver=2] > [10:11:01,613][INFO][main][GridCacheProcessor] Started cache in recovery > mode [name=ignite-sys-cache, id=-2100569601, dataRegionName=sysMemPlc, > mode=REPLICATED, atomicity=TRANSACTIONAL, backups=2147483647, mvcc=false] > [10:11:01,632][INFO][main][PageMemoryImpl] Started page memory > [memoryAllocated=400.0 MiB, pages=99259, tableSize=7.7 MiB, > checkpointBuffer=256.0 MiB] > [10:11:01,740][INFO][main][GridCacheProcessor] Started cache in recovery > mode [name=DSU_DataBase, id=-510489548, dataRegionName=default, > mode=PARTITIONED, atomicity=ATOMIC, backups=1, mvcc=false] > [10:11:01,753][INFO][main][GridCacheDatabaseSharedManager] Binary recovery > performed in 823 ms. > [10:11:01,754][INFO][main][GridCacheDatabaseSharedManager] Read checkpoint > status [startMarker=null, endMarker=null] > [10:11:01,754][INFO][main][GridCacheDatabaseSharedManager] Applying lost > cache updates since last checkpoint record [lastMarked=FileWALPointer > [idx=0, fileOff=0, len=0], > lastCheckpointId=00000000-0000-0000-0000-000000000000] > [10:11:01,888][INFO][main][GridCacheDatabaseSharedManager] Finished > applying > WAL changes [updatesApplied=0, time=127 ms] > [10:11:01,888][INFO][main][GridCacheProcessor] Restoring partition state > for > local groups. > [10:11:01,976][INFO][main][GridCacheProcessor] Finished restoring partition > state for local groups [groupsProcessed=2, partitionsProcessed=0, > time=93ms] > [10:11:02,176][INFO][main][TcpDiscoverySpi] Connection check threshold is > calculated: 10000 > [10:11:02,244][INFO][disco-notifier-worker-#38][MvccProcessorImpl] Assigned > mvcc coordinator [crd=MvccCoordinator [topVer=AffinityTopologyVersion > [topVer=1, minorTopVer=0], nodeId=0aad560a-faad-4f86-b65c-a7f161bf2639, > ver=1599885662229, local=true, initialized=false]] > > [10:11:02,320][INFO][disco-notifier-worker-#38][DistributedBaselineConfiguration] > Baseline parameter 'baselineAutoAdjustEnabled' was changed from 'null' to > 'false' > > [10:11:02,328][INFO][disco-notifier-worker-#38][DistributedBaselineConfiguration] > Baseline parameter 'baselineAutoAdjustTimeout' was changed from 'null' to > '300000' > [10:11:02,604][INFO][main][IgniteKernal] Performance suggestions for grid > (fix if possible) > [10:11:02,605][INFO][main][IgniteKernal] To disable, set > -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true > [10:11:02,605][INFO][main][IgniteKernal] ^-- Decrease number of backups > (set 'backups' to 0) > [10:11:02,606][INFO][main][IgniteKernal] ^-- Disable fully synchronous > writes (set 'writeSynchronizationMode' to PRIMARY_SYNC or FULL_ASYNC) > [10:11:02,607][INFO][main][IgniteKernal] ^-- Switch to the most recent > 1.8 > JVM version > [10:11:02,608][INFO][main][IgniteKernal] ^-- Set max direct memory size > if > getting 'OOME: Direct buffer memory' (add > '-XX:MaxDirectMemorySize=<size>[g|G|m|M|k|K]' to JVM options) > [10:11:02,608][INFO][main][IgniteKernal] ^-- Disable processing of calls > to System.gc() (add '-XX:+DisableExplicitGC' to JVM options) > [10:11:02,610][INFO][main][IgniteKernal] Refer to this page for more > performance suggestions: > https://apacheignite.readme.io/docs/jvm-and-system-tuning > [10:11:02,610][INFO][main][IgniteKernal] > [10:11:02,611][INFO][main][IgniteKernal] To start Console Management & > Monitoring run ignitevisorcmd.{sh|bat} > [10:11:02,615][INFO][main][IgniteKernal] Data Regions Configured: > [10:11:02,616][INFO][main][IgniteKernal] ^-- default [initSize=100.0 MiB, > maxSize=400.0 MiB, persistence=true, lazyMemoryAllocation=true] > [10:11:02,616][INFO][main][IgniteKernal] > > >>> > +----------------------------------------------------------------------+ > >>> Ignite ver. > 2.8.0#20200226-sha1:341b01dfd8abf2d9b01d468ad1bb26dfe84ac4f6 > >>> > +----------------------------------------------------------------------+ > >>> OS name: Linux 4.19.0-8-686 i386 > >>> CPU(s): 1 > >>> Heap: 0.48GB > >>> VM name: 1461@dsudev > >>> Local node [ID=0AAD560A-FAAD-4F86-B65C-A7F161BF2639, order=1, > >>> clientMode=false] > >>> Local node addresses: [192.168.15.111/0:0:0:0:0:0:0:1%lo, > >>> 192.168.111.111/127.0.0.1, 192.168.100.10/192.168.10.5, > >>> 192.168.10.5/192.168.100.10, 192.168.12.112/192.168.111.111, > >>> /192.168.12.112, /192.168.15.111] > >>> Local ports: TCP:10800 TCP:11211 TCP:47100 TCP:47500 > > [10:11:02,623][INFO][main][IgniteKernal] >>> Ignite cluster is not active > (limited functionality available). Use control.(sh|bat) script or > IgniteCluster interface to activate. > [10:11:02,630][INFO][main][GridDiscoveryManager] Topology snapshot [ver=1, > locNode=0aad560a, servers=1, clients=0, state=INACTIVE, CPUs=1, > offheap=0.39GB, heap=0.48GB] > [10:11:02,631][INFO][main][G] Node started : [stage="Configure system pool" > (158 ms),stage="Start managers" (1940 ms),stage="Configure binary metadata" > (324 ms),stage="Start processors" (1268 ms),stage="Init metastore" (241 > ms),stage="Init and start regions" (63 ms),stage="Restore binary memory" > (824 ms),stage="Restore logical state" (222 ms),stage="Finish recovery" > (183 > ms),stage="Join topology" (81 ms),stage="Await transition" (17 > ms),stage="Await exchange" (372 ms),stage="Total time" (5693 ms)] > [10:11:02,666][INFO][main][GridClusterStateProcessor] Sending activate > request with BaselineTopology BaselineTopology [id=0, > branchingHash=-173057973, branchingType='New BaselineTopology', > baselineNodes=[9aae0abd-bec2-4d53-93c2-cdf34201d256]] > [10:11:02,711][INFO][disco-notifier-worker-#38][GridClusterStateProcessor] > Received activate request with BaselineTopology[id=0] > [10:11:02,713][INFO][disco-notifier-worker-#38][GridClusterStateProcessor] > Started state transition: true > [10:11:02,724][INFO][exchange-worker-#40][time] Started exchange init > [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], crd=true, > evt=DISCOVERY_CUSTOM_EVT, evtNode=0aad560a-faad-4f86-b65c-a7f161bf2639, > customEvt=ChangeGlobalStateMessage > [id=86bbd908471-259db0fa-9d36-40c3-ad9f-b8359727dc28, > reqId=e5a2e4ff-c7aa-4924-bc83-7f86c3f16ea1, > initiatingNodeId=0aad560a-faad-4f86-b65c-a7f161bf2639, activate=true, > baselineTopology=BaselineTopology [id=0, branchingHash=-173057973, > branchingType='New BaselineTopology', > baselineNodes=[9aae0abd-bec2-4d53-93c2-cdf34201d256]], > forceChangeBaselineTopology=false, timestamp=1599885662666], > allowMerge=false, exchangeFreeSwitch=false] > [10:11:02,725][INFO][exchange-worker-#40][GridDhtPartitionsExchangeFuture] > Start activation process [nodeId=0aad560a-faad-4f86-b65c-a7f161bf2639, > client=false, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1]] > [10:11:02,726][INFO][exchange-worker-#40][FilePageStoreManager] Resolved > page store work directory: > > /media/linux_hdd/keymanagementdata/node00-9aae0abd-bec2-4d53-93c2-cdf34201d256 > [10:11:02,730][INFO][exchange-worker-#40][FileWriteAheadLogManager] > Resuming > logging to WAL segment > > [file=/media/linux_hdd/keymanagementdata/db/wal/node00-9aae0abd-bec2-4d53-93c2-cdf34201d256/0000000000000000.wal, > offset=13069, ver=2] > [10:11:02,732][INFO][exchange-worker-#40][GridClusterStateProcessor] > Writing > BaselineTopology[id=0] > [10:11:02,768][INFO][exchange-worker-#40][GridCacheDatabaseSharedManager] > Finish recovery performed in 42 ms. > [10:11:02,770][INFO][exchange-worker-#40][msg] Components activation > performed in 44 ms. > [10:11:02,840][INFO][exchange-worker-#40][GridCacheProcessor] Finished > recovery for cache [cache=DSU_DataBase, grp=DSU_DataBase, > startVer=AffinityTopologyVersion [topVer=1, minorTopVer=1]] > [10:11:02,840][INFO][sys-#49][GridCacheProcessor] Finished recovery for > cache [cache=ignite-sys-cache, grp=ignite-sys-cache, > startVer=AffinityTopologyVersion [topVer=1, minorTopVer=1]] > [10:11:02,886][INFO][exchange-worker-#40][GridAffinityAssignmentCache] > Local > node affinity assignment distribution is not ideal [cache=DSU_DataBase, > expectedPrimary=1024.00, actualPrimary=1024, expectedBackups=1024.00, > actualBackups=0, warningThreshold=50.00%] > [10:11:02,909][INFO][exchange-worker-#40][GridDhtPartitionsExchangeFuture] > Successfully activated caches [nodeId=0aad560a-faad-4f86-b65c-a7f161bf2639, > client=false, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1]] > [10:11:02,922][INFO][exchange-worker-#40][GridDhtPartitionsExchangeFuture] > Finished waiting for partition release future > [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], waitTime=0ms, > futInfo=NA, mode=LOCAL] > [10:11:03,485][INFO][exchange-worker-#40][GridDhtPartitionsExchangeFuture] > finishExchangeOnCoordinator [topVer=AffinityTopologyVersion [topVer=1, > minorTopVer=1], resVer=AffinityTopologyVersion [topVer=1, minorTopVer=1]] > > [10:11:03,513][INFO][db-checkpoint-thread-#54][GridCacheDatabaseSharedManager] > Checkpoint started [checkpointId=d6caae21-b222-4a61-9956-671e24030c13, > startPtr=FileWALPointer [idx=0, fileOff=142444, len=21425], > checkpointBeforeLockTime=87ms, checkpointLockWait=0ms, > checkpointListenersExecuteTime=56ms, checkpointLockHoldTime=80ms, > walCpRecordFsyncDuration=9ms, writeCheckpointEntryDuration=10ms, > splitAndSortCpPagesDuration=0ms, pages=20, reason='node started'] > > [10:11:03,545][INFO][db-checkpoint-thread-#54][GridCacheDatabaseSharedManager] > Checkpoint finished [cpId=d6caae21-b222-4a61-9956-671e24030c13, pages=20, > markPos=FileWALPointer [idx=0, fileOff=142444, len=21425], > walSegmentsCleared=0, walSegmentsCovered=[], markDuration=109ms, > pagesWrite=5ms, fsync=26ms, total=227ms] > [10:11:03,667][INFO][disco-notifier-worker-#38][GridClusterStateProcessor] > Received state change finish message: true > [10:11:03,669][INFO][exchange-worker-#40][GridDhtPartitionsExchangeFuture] > Finish exchange future [startVer=AffinityTopologyVersion [topVer=1, > minorTopVer=1], resVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], > err=null, rebalanced=true, wasRebalanced=false] > [10:11:03,752][INFO][exchange-worker-#40][GridCacheProcessor] Finish proxy > initialization, cacheName=ignite-sys-cache, > localNodeId=0aad560a-faad-4f86-b65c-a7f161bf2639 > [10:11:03,752][INFO][exchange-worker-#40][GridCacheProcessor] Finish proxy > initialization, cacheName=DSU_DataBase, > localNodeId=0aad560a-faad-4f86-b65c-a7f161bf2639 > [10:11:03,766][INFO][sys-#53][DistributedBaselineConfiguration] Baseline > auto-adjust is 'disabled' with timeout='300000' ms > [10:11:03,767][INFO][sys-#53][GridClusterStateProcessor] Successfully > performed final activation steps > [nodeId=0aad560a-faad-4f86-b65c-a7f161bf2639, client=false, > topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1]] > [10:11:03,848][INFO][exchange-worker-#40][GridDhtPartitionsExchangeFuture] > Completed partition exchange > [localNode=0aad560a-faad-4f86-b65c-a7f161bf2639, > exchange=GridDhtPartitionsExchangeFuture [topVer=AffinityTopologyVersion > [topVer=1, minorTopVer=1], evt=DISCOVERY_CUSTOM_EVT, > evtNode=TcpDiscoveryNode [id=0aad560a-faad-4f86-b65c-a7f161bf2639, > consistentId=9aae0abd-bec2-4d53-93c2-cdf34201d256, addrs=ArrayList > [0:0:0:0:0:0:0:1%lo, 127.0.0.1, 192.168.10.5, 192.168.100.10, > 192.168.111.111, 192.168.12.112, 192.168.15.111], sockAddrs=HashSet > [/192.168.111.111:47500, /192.168.15.111:47500, /192.168.12.112:47500, > /192.168.10.5:47500, /0:0:0:0:0:0:0:1%lo:47500, /127.0.0.1:47500, > /192.168.100.10:47500], discPort=47500, order=1, intOrder=1, > lastExchangeTime=1599885662175, loc=true, ver=2.8.0#20200226-sha1:341b01df, > isClient=false], done=true, newCrdFut=null], topVer=AffinityTopologyVersion > [topVer=1, minorTopVer=1]] > [10:11:03,849][INFO][exchange-worker-#40][GridDhtPartitionsExchangeFuture] > Exchange timings [startVer=AffinityTopologyVersion [topVer=1, > minorTopVer=1], resVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], > stage="Waiting in exchange queue" (1 ms), stage="Exchange parameters > initialization" (2 ms), stage="Update caches registry" (57 ms), > stage="Start > caches" (67 ms), stage="Affinity initialization on cache group start" (59 > ms), stage="Determine exchange type" (9 ms), stage="Preloading > notification" > (0 ms), stage="WAL history reservation" (1 ms), stage="Wait partitions > release" (5 ms), stage="Restore partition states" (22 ms), stage="After > states restored callback" (536 ms), stage="Waiting for all single messages" > (0 ms), stage="Affinity recalculation (crd)" (0 ms), stage="Collect update > counters and create affinity messages" (0 ms), stage="Validate partitions > states" (8 ms), stage="Assign partitions states" (54 ms), stage="Ideal > affinity diff calculation (enforced)" (41 ms), stage="Apply update > counters" > (4 ms), stage="Full message preparing" (70 ms), stage="Full message > sending" > (0 ms), stage="State finish message sending" (2 ms), stage="Exchange done" > (180 ms), stage="Total time" (1118 ms)] > [10:11:03,849][INFO][exchange-worker-#40][GridDhtPartitionsExchangeFuture] > Exchange longest local stages [startVer=AffinityTopologyVersion [topVer=1, > minorTopVer=1], resVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], > stage="Affinity initialization on cache group start [grp=DSU_DataBase]" (59 > ms) (parent=Affinity initialization on cache group start), stage="Affinity > initialization on cache group start [grp=ignite-sys-cache]" (37 ms) > (parent=Affinity initialization on cache group start), stage="Affinity > centralized initialization (crd) [grp=ignite-sys-cache, crd=true]" (5 ms) > (parent=Determine exchange type), stage="Affinity centralized > initialization > (crd) [grp=DSU_DataBase, crd=true]" (5 ms) (parent=Determine exchange > type), > stage="Restore partition states [grp=DSU_DataBase]" (22 ms) (parent=Restore > partition states), stage="Restore partition states [grp=ignite-sys-cache]" > (14 ms) (parent=Restore partition states), stage="Affinity recalculation > (partitions availability) [grp=DSU_DataBase]" (40 ms) (parent=Ideal > affinity > diff calculation (enforced)), stage="Affinity recalculation (partitions > availability) [grp=ignite-sys-cache]" (14 ms) (parent=Ideal affinity diff > calculation (enforced))] > [10:11:03,849][INFO][exchange-worker-#40][time] Finished exchange init > [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], crd=true] > > [10:11:03,918][INFO][exchange-worker-#40][GridCachePartitionExchangeManager] > Skipping rebalancing (nothing scheduled) [top=AffinityTopologyVersion > [topVer=1, minorTopVer=1], force=false, evt=DISCOVERY_CUSTOM_EVT, > node=0aad560a-faad-4f86-b65c-a7f161bf2639] > > [10:11:05,808][INFO][disco-notifier-worker-#38][DistributedBaselineConfiguration] > Baseline parameter 'baselineAutoAdjustTimeout' was changed from '300000' to > '5' > > [10:11:05,817][INFO][disco-notifier-worker-#38][DistributedBaselineConfiguration] > Baseline parameter 'baselineAutoAdjustEnabled' was changed from 'false' to > 'true' > [10:12:02,608][INFO][grid-timeout-worker-#23][IgniteKernal] > Metrics for local node (to disable set 'metricsLogFrequency' to 0) > ^-- Node [id=0aad560a, uptime=00:01:00.002] > ^-- H/N/C [hosts=1, nodes=1, CPUs=1] > ^-- CPU [cur=-100%, avg=-95.83%, GC=0%] > ^-- PageMemory [pages=20] > ^-- Heap [used=32MB, free=93.48%, comm=247MB] > ^-- Off-heap [used=0MB, free=99.99%, comm=600MB] > ^-- sysMemPlc region [used=0MB, free=99.98%, comm=100MB] > ^-- default region [used=0MB, free=100%, comm=400MB] > ^-- metastoreMemPlc region [used=0MB, free=99.96%, comm=0MB] > ^-- TxLog region [used=0MB, free=100%, comm=100MB] > ^-- Ignite persistence [used=0MB] > ^-- sysMemPlc region [used=0MB] > ^-- default region [used=0MB] > ^-- metastoreMemPlc region [used=0MB] > ^-- TxLog region [used=0MB] > ^-- Outbound messages queue [size=0] > ^-- Public thread pool [active=0, idle=0, qSize=0] > ^-- System thread pool [active=0, idle=8, qSize=0] > [10:12:12,305][INFO][ignite-update-notifier-timer][GridUpdateNotifier] > Update status is not available. > [10:13:02,616][INFO][grid-timeout-worker-#23][IgniteKernal] > Metrics for local node (to disable set 'metricsLogFrequency' to 0) > ^-- Node [id=0aad560a, uptime=00:02:00.020] > ^-- H/N/C [hosts=1, nodes=1, CPUs=1] > ^-- CPU [cur=-100%, avg=-97.78%, GC=0%] > ^-- PageMemory [pages=20] > ^-- Heap [used=35MB, free=92.91%, comm=247MB] > ^-- Off-heap [used=0MB, free=99.99%, comm=600MB] > ^-- sysMemPlc region [used=0MB, free=99.98%, comm=100MB] > ^-- default region [used=0MB, free=100%, comm=400MB] > ^-- metastoreMemPlc region [used=0MB, free=99.96%, comm=0MB] > ^-- TxLog region [used=0MB, free=100%, comm=100MB] > ^-- Ignite persistence [used=0MB] > ^-- sysMemPlc region [used=0MB] > ^-- default region [used=0MB] > ^-- metastoreMemPlc region [used=0MB] > ^-- TxLog region [used=0MB] > ^-- Outbound messages queue [size=0] > ^-- Public thread pool [active=0, idle=0, qSize=0] > ^-- System thread pool [active=0, idle=6, qSize=0] > [10:14:02,624][INFO][grid-timeout-worker-#23][IgniteKernal] > Metrics for local node (to disable set 'metricsLogFrequency' to 0) > ^-- Node [id=0aad560a, uptime=00:03:00.025] > ^-- H/N/C [hosts=1, nodes=1, CPUs=1] > ^-- CPU [cur=-100%, avg=-98.63%, GC=0%] > ^-- PageMemory [pages=20] > ^-- Heap [used=36MB, free=92.58%, comm=247MB] > ^-- Off-heap [used=0MB, free=99.99%, comm=600MB] > ^-- sysMemPlc region [used=0MB, free=99.98%, comm=100MB] > ^-- default region [used=0MB, free=100%, comm=400MB] > ^-- metastoreMemPlc region [used=0MB, free=99.96%, comm=0MB] > ^-- TxLog region [used=0MB, free=100%, comm=100MB] > ^-- Ignite persistence [used=0MB] > ^-- sysMemPlc region [used=0MB] > ^-- default region [used=0MB] > ^-- metastoreMemPlc region [used=0MB] > ^-- TxLog region [used=0MB] > ^-- Outbound messages queue [size=0] > ^-- Public thread pool [active=0, idle=0, qSize=0] > ^-- System thread pool [active=0, idle=6, qSize=0] > [10:14:03,465][SEVERE][db-checkpoint-thread-#54][] Critical system error > detected. Will be handled accordingly to configured handler > [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, > super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet > [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], > failureCtx=FailureContext [type=CRITICAL_ERROR, err=class > o.a.i.i.processors.cache.persistence.StorageException: Failed to write > checkpoint entry [ptr=FileWALPointer [idx=0, fileOff=183931, len=21409], > cpTs=1599885843404, cpId=78332869-fedc-4687-9202-180b8de1c2b8, > type=START]]] > class > org.apache.ignite.internal.processors.cache.persistence.StorageException: > Failed to write checkpoint entry [ptr=FileWALPointer [idx=0, > fileOff=183931, > len=21409], cpTs=1599885843404, cpId=78332869-fedc-4687-9202-180b8de1c2b8, > type=START] > at > > org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.writeCheckpointEntry(GridCacheDatabaseSharedManager.java:3248) > at > > org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.markCheckpointBegin(GridCacheDatabaseSharedManager.java:4220) > at > > org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.doCheckpoint(GridCacheDatabaseSharedManager.java:3738) > at > > org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager$Checkpointer.body(GridCacheDatabaseSharedManager.java:3623) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) > at java.base/java.lang.Thread.run(Thread.java:834) > Caused by: java.nio.file.NoSuchFileException: > > /media/linux_hdd/keymanagementdata/node00-9aae0abd-bec2-4d53-93c2-cdf34201d256/cp/1599885843404-78332869-fedc-4687-9202-180b8de1c2b8-START.bin.tmp > at > > java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92) > at > > java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) > at > > java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) > at > > java.base/sun.nio.fs.UnixFileSystemProvider.newAsynchronousFileChannel(UnixFileSystemProvider.java:201) > at > > java.base/java.nio.channels.AsynchronousFileChannel.open(AsynchronousFileChannel.java:253) > at > > java.base/java.nio.channels.AsynchronousFileChannel.open(AsynchronousFileChannel.java:311) > at > > org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIO.<init>(AsyncFileIO.java:66) > at > > org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory.create(AsyncFileIOFactory.java:44) > at > > org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.writeCheckpointEntry(GridCacheDatabaseSharedManager.java:3231) > ... 5 more > [10:14:03,499][WARNING][db-checkpoint-thread-#54][CacheDiagnosticManager] > Page locks dump: > > Thread=[name=db-checkpoint-thread-#54, id=84], state=RUNNABLE > Locked pages = [] > Locked pages log: name=db-checkpoint-thread-#54 time=(1599885843482, > 2020-09-12 10:14:03.482) > > > Thread=[name=dms-writer-thread-#45, id=75], state=WAITING > Locked pages = [] > Locked pages log: name=dms-writer-thread-#45 time=(1599885843482, > 2020-09-12 > 10:14:03.482) > > > Thread=[name=exchange-worker-#40, id=69], state=TIMED_WAITING > Locked pages = [] > Locked pages log: name=exchange-worker-#40 time=(1599885843482, 2020-09-12 > 10:14:03.482) > > > Thread=[name=main, id=1], state=RUNNABLE > Locked pages = [] > Locked pages log: name=main time=(1599885843482, 2020-09-12 10:14:03.482) > > > > [10:14:03,499][SEVERE][db-checkpoint-thread-#54][] JVM will be halted > immediately due to the failure: [failureCtx=FailureContext > [type=CRITICAL_ERROR, err=class > o.a.i.i.processors.cache.persistence.StorageException: Failed to write > checkpoint entry [ptr=FileWALPointer [idx=0, fileOff=183931, len=21409], > cpTs=1599885843404, cpId=78332869-fedc-4687-9202-180b8de1c2b8, > type=START]]] > > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
