Re: graceful shutdown for C++ applications
Hello! This will happen when this file is deleted while the instance is running. Not sure who deleted it. Maybe you tried to start another node with the same consistent id in the background? You should avoid calling setActive() every time since it will lead to data loss. Regards, -- Ilya Kasnacheev чт, 7 янв. 2021 г. в 14:04, rakshita04 : > These are the full set of logs, if it helps- > [10:10:56,860][WARNING][main][G] Ignite work directory is not provided, > automatically resolved to: /home/dsudev/ignite-master/work > [10:10:56,873][WARNING][main][G] Consistent ID is not set, it is > recommended > to set consistent ID for production clusters (use > IgniteConfiguration.setConsistentId property) > [10:10:57,103][INFO][main][IgniteKernal] > > >>>__ > >>> / _/ ___/ |/ / _/_ __/ __/ > >>> _/ // (7 7// / / / / _/ > >>> /___/\___/_/|_/___/ /_/ /___/ > >>> > >>> ver. 2.8.0#20200226-sha1:341b01df > >>> 2020 Copyright(C) Apache Software Foundation > >>> > >>> Ignite documentation: http://ignite.apache.org > > [10:10:57,134][INFO][main][IgniteKernal] Config URL: n/a > [10:10:57,190][INFO][main][IgniteKernal] IgniteConfiguration > [igniteInstanceName=null, pubPoolSize=8, svcPoolSize=8, callbackPoolSize=8, > stripedPoolSize=8, sysPoolSize=8, mgmtPoolSize=4, igfsPoolSize=1, > dataStreamerPoolSize=8, utilityCachePoolSize=8, > utilityCacheKeepAliveTime=6, p2pPoolSize=2, qryPoolSize=8, > sqlQryHistSize=1000, dfltQryTimeout=0, > igniteHome=/home/dsudev/ignite-master, > igniteWorkDir=/home/dsudev/ignite-master/work, > mbeanSrv=com.sun.jmx.mbeanserver.JmxMBeanServer@150fbeb, > nodeId=0aad560a-faad-4f86-b65c-a7f161bf2639, marsh=BinaryMarshaller [], > marshLocJobs=false, daemon=false, p2pEnabled=true, netTimeout=5000, > netCompressionLevel=1, sndRetryDelay=1000, sndRetryCnt=3, > metricsHistSize=1, metricsUpdateFreq=2000, > metricsExpTime=9223372036854775807, discoSpi=TcpDiscoverySpi > [addrRslvr=null, sockTimeout=0, ackTimeout=0, marsh=null, reconCnt=10, > reconDelay=2000, maxAckTimeout=60, soLinger=5, forceSrvMode=false, > clientReconnectDisabled=false, internalLsnr=null, > skipAddrsRandomization=false], segPlc=STOP, segResolveAttempts=2, > waitForSegOnStart=true, allResolversPassReq=true, segChkFreq=1, > commSpi=TcpCommunicationSpi [connectGate=null, > > connPlc=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$FirstConnectionPolicy@1390459 > , > chConnPlc=null, enableForcibleNodeKill=false, > enableTroubleshootingLog=false, locAddr=null, locHost=null, locPort=47100, > locPortRange=100, shmemPort=-1, directBuf=true, directSndBuf=false, > idleConnTimeout=60, connTimeout=5000, maxConnTimeout=60, > reconCnt=10, sockSndBuf=32768, sockRcvBuf=32768, msgQueueLimit=0, > slowClientQueueLimit=0, nioSrvr=null, shmemSrv=null, > usePairedConnections=false, connectionsPerNode=1, tcpNoDelay=true, > filterReachableAddresses=false, ackSndThreshold=32, unackedMsgsBufSize=0, > sockWriteTimeout=2000, boundTcpPort=-1, boundTcpShmemPort=-1, > selectorsCnt=4, selectorSpins=0, addrRslvr=null, > ctxInitLatch=java.util.concurrent.CountDownLatch@156c3cd[Count = 1], > stopping=false, metricsLsnr=null], > evtSpi=org.apache.ignite.spi.eventstorage.NoopEventStorageSpi@113052e, > colSpi=NoopCollisionSpi [], deploySpi=LocalDeploymentSpi [], > indexingSpi=org.apache.ignite.spi.indexing.noop.NoopIndexingSpi@45dbe, > addrRslvr=null, > > encryptionSpi=org.apache.ignite.spi.encryption.noop.NoopEncryptionSpi@14658f7 > , > clientMode=false, rebalanceThreadPoolSize=4, rebalanceTimeout=1, > rebalanceBatchesPrefetchCnt=3, rebalanceThrottle=0, > rebalanceBatchSize=524288, txCfg=TransactionConfiguration > [txSerEnabled=false, dfltIsolation=REPEATABLE_READ, > dfltConcurrency=PESSIMISTIC, dfltTxTimeout=0, > txTimeoutOnPartitionMapExchange=0, deadlockTimeout=1, > pessimisticTxLogSize=0, pessimisticTxLogLinger=1, tmLookupClsName=null, > txManagerFactory=null, useJtaSync=false], cacheSanityCheckEnabled=true, > discoStartupDelay=6, deployMode=SHARED, p2pMissedCacheSize=100, > locHost=null, timeSrvPortBase=31100, timeSrvPortRange=100, > failureDetectionTimeout=1, sysWorkerBlockedTimeout=null, > clientFailureDetectionTimeout=3, metricsLogFreq=6, hadoopCfg=null, > connectorCfg=ConnectorConfiguration [jettyPath=null, host=null, port=11211, > noDelay=true, directBuf=false, sndBufSize=32768, rcvBufSize=32768, > idleQryCurTimeout=60, idleQryCurCheckFreq=6, sndQueueLimit=0, > selectorCnt=1, idleTimeout=7000, sslEnabled=false, sslClientAuth=false, > sslCtxFactory=null, sslFactory=null, portRange=100, threadPoolSize=8, > msgInterceptor=null], odbcCfg=null, warmupClos=null, > atomicCfg=AtomicConfiguration [seqReserveSize=1000, cacheMode=PARTITIONED, > backups=1, aff=null, grpName=null], classLdr=null, > sslCtxFactory=SslContextFactory[keyStoreType=JKS, proto=TLS, > keyStoreFile=/home/dsudev/config/keystore.jks, > trustStoreFile=/home/dsudev/config/truststore.jks], >
Re: graceful shutdown for C++ applications
These are the full set of logs, if it helps- [10:10:56,860][WARNING][main][G] Ignite work directory is not provided, automatically resolved to: /home/dsudev/ignite-master/work [10:10:56,873][WARNING][main][G] Consistent ID is not set, it is recommended to set consistent ID for production clusters (use IgniteConfiguration.setConsistentId property) [10:10:57,103][INFO][main][IgniteKernal] >>>__ >>> / _/ ___/ |/ / _/_ __/ __/ >>> _/ // (7 7// / / / / _/ >>> /___/\___/_/|_/___/ /_/ /___/ >>> >>> ver. 2.8.0#20200226-sha1:341b01df >>> 2020 Copyright(C) Apache Software Foundation >>> >>> Ignite documentation: http://ignite.apache.org [10:10:57,134][INFO][main][IgniteKernal] Config URL: n/a [10:10:57,190][INFO][main][IgniteKernal] IgniteConfiguration [igniteInstanceName=null, pubPoolSize=8, svcPoolSize=8, callbackPoolSize=8, stripedPoolSize=8, sysPoolSize=8, mgmtPoolSize=4, igfsPoolSize=1, dataStreamerPoolSize=8, utilityCachePoolSize=8, utilityCacheKeepAliveTime=6, p2pPoolSize=2, qryPoolSize=8, sqlQryHistSize=1000, dfltQryTimeout=0, igniteHome=/home/dsudev/ignite-master, igniteWorkDir=/home/dsudev/ignite-master/work, mbeanSrv=com.sun.jmx.mbeanserver.JmxMBeanServer@150fbeb, nodeId=0aad560a-faad-4f86-b65c-a7f161bf2639, marsh=BinaryMarshaller [], marshLocJobs=false, daemon=false, p2pEnabled=true, netTimeout=5000, netCompressionLevel=1, sndRetryDelay=1000, sndRetryCnt=3, metricsHistSize=1, metricsUpdateFreq=2000, metricsExpTime=9223372036854775807, discoSpi=TcpDiscoverySpi [addrRslvr=null, sockTimeout=0, ackTimeout=0, marsh=null, reconCnt=10, reconDelay=2000, maxAckTimeout=60, soLinger=5, forceSrvMode=false, clientReconnectDisabled=false, internalLsnr=null, skipAddrsRandomization=false], segPlc=STOP, segResolveAttempts=2, waitForSegOnStart=true, allResolversPassReq=true, segChkFreq=1, commSpi=TcpCommunicationSpi [connectGate=null, connPlc=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$FirstConnectionPolicy@1390459, chConnPlc=null, enableForcibleNodeKill=false, enableTroubleshootingLog=false, locAddr=null, locHost=null, locPort=47100, locPortRange=100, shmemPort=-1, directBuf=true, directSndBuf=false, idleConnTimeout=60, connTimeout=5000, maxConnTimeout=60, reconCnt=10, sockSndBuf=32768, sockRcvBuf=32768, msgQueueLimit=0, slowClientQueueLimit=0, nioSrvr=null, shmemSrv=null, usePairedConnections=false, connectionsPerNode=1, tcpNoDelay=true, filterReachableAddresses=false, ackSndThreshold=32, unackedMsgsBufSize=0, sockWriteTimeout=2000, boundTcpPort=-1, boundTcpShmemPort=-1, selectorsCnt=4, selectorSpins=0, addrRslvr=null, ctxInitLatch=java.util.concurrent.CountDownLatch@156c3cd[Count = 1], stopping=false, metricsLsnr=null], evtSpi=org.apache.ignite.spi.eventstorage.NoopEventStorageSpi@113052e, colSpi=NoopCollisionSpi [], deploySpi=LocalDeploymentSpi [], indexingSpi=org.apache.ignite.spi.indexing.noop.NoopIndexingSpi@45dbe, addrRslvr=null, encryptionSpi=org.apache.ignite.spi.encryption.noop.NoopEncryptionSpi@14658f7, clientMode=false, rebalanceThreadPoolSize=4, rebalanceTimeout=1, rebalanceBatchesPrefetchCnt=3, rebalanceThrottle=0, rebalanceBatchSize=524288, txCfg=TransactionConfiguration [txSerEnabled=false, dfltIsolation=REPEATABLE_READ, dfltConcurrency=PESSIMISTIC, dfltTxTimeout=0, txTimeoutOnPartitionMapExchange=0, deadlockTimeout=1, pessimisticTxLogSize=0, pessimisticTxLogLinger=1, tmLookupClsName=null, txManagerFactory=null, useJtaSync=false], cacheSanityCheckEnabled=true, discoStartupDelay=6, deployMode=SHARED, p2pMissedCacheSize=100, locHost=null, timeSrvPortBase=31100, timeSrvPortRange=100, failureDetectionTimeout=1, sysWorkerBlockedTimeout=null, clientFailureDetectionTimeout=3, metricsLogFreq=6, hadoopCfg=null, connectorCfg=ConnectorConfiguration [jettyPath=null, host=null, port=11211, noDelay=true, directBuf=false, sndBufSize=32768, rcvBufSize=32768, idleQryCurTimeout=60, idleQryCurCheckFreq=6, sndQueueLimit=0, selectorCnt=1, idleTimeout=7000, sslEnabled=false, sslClientAuth=false, sslCtxFactory=null, sslFactory=null, portRange=100, threadPoolSize=8, msgInterceptor=null], odbcCfg=null, warmupClos=null, atomicCfg=AtomicConfiguration [seqReserveSize=1000, cacheMode=PARTITIONED, backups=1, aff=null, grpName=null], classLdr=null, sslCtxFactory=SslContextFactory[keyStoreType=JKS, proto=TLS, keyStoreFile=/home/dsudev/config/keystore.jks, trustStoreFile=/home/dsudev/config/truststore.jks], platformCfg=PlatformConfiguration [], binaryCfg=BinaryConfiguration [idMapper=BinaryBaseIdMapper [isLowerCase=true], nameMapper=BinaryBaseNameMapper [isSimpleName=true], serializer=null, compactFooter=false], memCfg=null, pstCfg=null, dsCfg=DataStorageConfiguration [sysRegionInitSize=41943040, sysRegionMaxSize=104857600, pageSize=4096, concLvl=0, dfltDataRegConf=DataRegionConfiguration [name=default, maxSize=419430400, initSize=104857600, swapPath=null, pageEvictionMode=DISABLED, evictionThreshold=0.9,
Re: graceful shutdown for C++ applications
I am also getting below error on my ignite logs- [20:00:50,515][SEVERE][db-checkpoint-thread-#54][] Critical system error detected. Will be handled accordingly to configured handler [hnd=StopNodeOrHaltFailureHandler [tryStop=false, timeout=0, super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=CRITICAL_ERROR, err=class o.a.i.i.processors.cache.persistence.StorageException: Failed to write checkpoint entry [ptr=FileWALPointer [idx=0, fileOff=188385, len=21409], cpTs=1608042650462, cpId=a273b41f-b536-4c7d-afbd-51303114306b, type=START]]] class org.apache.ignite.internal.processors.cache.persistence.StorageException: Failed to write checkpoint entry [ptr=FileWALPointer [idx=0, fileOff=188385, len=21409], cpTs=1608042650462, cpId=a273b41f-b536-4c7d-afbd-51303114306b, type=START] what can cause this? And how to avoid this problem? -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: graceful shutdown for C++ applications
It shouldn’t cause a crash, but since you don’t need to activate an already active cluster maybe it’s not well tested. Sending the node a TERM signal (press ^C) is good way to stop a node. > On 7 Jan 2021, at 09:26, rakshita04 wrote: > > can SetActive() cause the crash? > is this way okay to terminate the process by kill or there is some better > way? > > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: graceful shutdown for C++ applications
can SetActive() cause the crash? is this way okay to terminate the process by kill or there is some better way? -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: graceful shutdown for C++ applications
Not that it excuses the crash, but why are you calling activate every time the node starts? It should be called once, the first time all the nodes are present. The cluster will auto-activate every time after that. Regards, Stephen > On 7 Jan 2021, at 08:56, rakshita04 wrote: > > it works, the process is stopped but when application is started using the > same Database node, it crashes with below logs on terminal- > Ignite node stopped OK [uptime=00:00:55.197] > # > # A fatal error has been detected by the Java Runtime Environment: > # > # SIGSEGV (0xb) at pc=0x76c9f208, pid=26716, tid=26716 > # > # JRE version: OpenJDK Runtime Environment (11.0.6+10) (build > 11.0.6+10-post-Debian-1deb10u1) > # Java VM: OpenJDK Server VM (11.0.6+10-post-Debian-1deb10u1, mixed mode, g1 > gc, linux-) > # Problematic frame: > # C [libignite-2.8.0.44294.so.0+0x11208] > ignite::Ignite::SetActive(bool)+0xb > > Our application is calling SetActive method after node::start(). > > The application is recovered only when Databse is deleted and node is > started again. > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: graceful shutdown for C++ applications
Issuing a command like "kill process_id" doesn't work? regards. On Thu, Jan 7, 2021 at 4:14 PM rakshita04 wrote: > Hi Team, > > We are using apache-ignite for our applications running on 2 machines and > connected over network. > We are facing some issue where if kill is performed on running application, > it somehow corrupts the node and then node never comes up and keep on > rebooting. > Is there a way to handle this shutdown gracefully? so that there is no data > loss and node corruption. > > regards, > Rakshita Chaudhary > > > > -- > Sent from: http://apache-ignite-users.70518.x6.nabble.com/ >
graceful shutdown for C++ applications
Hi Team, We are using apache-ignite for our applications running on 2 machines and connected over network. We are facing some issue where if kill is performed on running application, it somehow corrupts the node and then node never comes up and keep on rebooting. Is there a way to handle this shutdown gracefully? so that there is no data loss and node corruption. regards, Rakshita Chaudhary -- Sent from: http://apache-ignite-users.70518.x6.nabble.com/
