Hi Dmitry. Thanks for you attention to this issue.
I changed repository to jcenter and set Ignite version to 2.4. Unfortunately the reproducer starts with the same error message in the log (see attached). I cannot say whether behavior of the whole cluster will change on 2.4, I mean if the cluster can start on corrupted data on 2.4, because we have wiped the data and restarted the cluster where the problem has arrived. We'll move to 2.4 next week and continue testing of our software. We are moving forward to production in April/May, and it would be good if we get some clue how to deal with such situation with data in the future. Arseny Kovalchuk Senior Software Engineer at Synesis skype: arseny.kovalchuk mobile: +375 (29) 666-16-16 LinkedIn Profile <http://www.linkedin.com/in/arsenykovalchuk/en> On 16 March 2018 at 17:03, Dmitry Pavlov <dpavlov....@gmail.com> wrote: > Hi Arseny, > > I've observed in reproducer > ignite_version=2.3.0 > > Could you check if it is reproducible in our freshest release 2.4.0. > > I'm not sure about ticket number, but it is quite possible issue is > already fixed. > > Sincerely, > Dmitriy Pavlov > > чт, 15 мар. 2018 г. в 19:34, Dmitry Pavlov <dpavlov....@gmail.com>: > >> Hi Alexey, >> >> It may be serious issue. Could you recommend expert here who can pick up >> this? >> >> Sincerely, >> Dmitriy Pavlov >> >> чт, 15 мар. 2018 г. в 19:25, Arseny Kovalchuk < >> arseny.kovalc...@synesis.ru>: >> >>> Hi, guys. >>> >>> I've got a reproducer for a problem which is generally reported as >>> "Caused by: java.lang.IllegalStateException: Failed to get page IO >>> instance (page content is corrupted)". Actually it reproduces the result. I >>> don't have an idea how the data has been corrupted, but the cluster node >>> doesn't want to start with this data. >>> >>> We got the issue again when some of server nodes were restarted several >>> times by kubernetes. I suspect that the data got corrupted during such >>> restarts. But the main functionality that we really desire to have, that >>> the cluster DOESN'T HANG during next restart even if the data is corrupted! >>> Anyway, there is no a tool that can help to correct such data, and as a >>> result we wipe all data manually to start the cluster. So, having warnings >>> about corrupted data in logs and just working cluster is the expected >>> behavior. >>> >>> How to reproduce: >>> 1. Download the data from here https://storage. >>> googleapis.com/pub-data-0/data5.tar.gz (~200Mb) >>> 2. Download and import Gradle project https://storage. >>> googleapis.com/pub-data-0/project.tar.gz (~100Kb) >>> 3. Unpack the data to the home folder, say /home/user1. You should get >>> the path like */home/user1/data5*. Inside data5 you should have >>> binary_meta, db, marshaller. >>> 4. Open *src/main/resources/data-test.xml* and put the absolute path of >>> unpacked data into *workDirectory* property of *igniteCfg5* bean. In >>> this example it should be */home/user1/data5.* Do not >>> edit consistentId! The consistentId is ignite-instance-5, so the real data >>> is in the data5/db/ignite_instance_5 folder >>> 5. Start application from ru.synesis.kipod.DataTestBootApp >>> 6. Enjoy >>> >>> Hope it will help. >>> >>> >>> >>> Arseny Kovalchuk >>> >>> Senior Software Engineer at Synesis >>> skype: arseny.kovalchuk >>> mobile: +375 (29) 666-16-16 <+375%2029%20666-16-16> >>> LinkedIn Profile <http://www.linkedin.com/in/arsenykovalchuk/en> >>> >>> On 26 December 2017 at 21:15, Denis Magda <dma...@apache.org> wrote: >>> >>>> Cross-posting to the dev list. >>>> >>>> Ignite persistence maintainers please chime in. >>>> >>>> — >>>> Denis >>>> >>> On Dec 26, 2017, at 2:17 AM, Arseny Kovalchuk < >>>> arseny.kovalc...@synesis.ru> wrote: >>>> >>>> Hi guys. >>>> >>>> Another issue when using Ignite 2.3 with native persistence enabled. >>>> See details below. >>>> >>>> We deploy Ignite along with our services in Kubernetes (v 1.8) on >>>> premises. Ignite cluster is a StatefulSet of 5 Pods (5 instances) of Ignite >>>> version 2.3. Each Pod mounts PersistentVolume backed by CEPH RBD. >>>> >>>> We put about 230 events/second into Ignite, 70% of events are ~200KB in >>>> size and 30% are 5000KB. Smaller events have indexed fields and we query >>>> them via SQL. >>>> >>>> The cluster is activated from a client node which also streams events >>>> into Ignite from Kafka. We use custom implementation of streamer which uses >>>> cache.putAll() API. >>>> >>>> We started cluster from scratch without any persistent data. After a >>>> while we got corrupted data with the error message. >>>> >>>> [2017-12-26 07:44:14,251] ERROR [sys-#127%ignite-instance-2%] >>>> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPreloader: >>>> - Partition eviction failed, this can cause grid hang. >>>> class org.apache.ignite.IgniteException: Runtime failure on search >>>> row: Row@5b1479d6[ key: 171:1513946618964:3008806055072854, val: >>>> ru.synesis.kipod.event.KipodEvent [idHash=510912646, hash=-387621419, >>>> face_last_name=null, face_list_id=null, channel=171, source=, >>>> face_similarity=null, license_plate_number=null, descriptors=null, >>>> cacheName=kipod_events, cacheKey=171:1513946618964:3008806055072854, >>>> stream=171, alarm=false, processed_at=0, face_id=null, id=3008806055072854, >>>> persistent=false, face_first_name=null, license_plate_first_name=null, >>>> face_full_name=null, level=0, module=Kpx.Synesis.Outdoor, >>>> end_time=1513946624379, params=null, commented_at=0, tags=[vehicle, 0, >>>> human, 0, truck, 0, start_time=1513946618964, processed=false, >>>> kafka_offset=111259, license_plate_last_name=null, armed=false, >>>> license_plate_country=null, topic=MovingObject, comment=, >>>> expiration=1514033024000, original_id=null, license_plate_lists=null], ver: >>>> GridCacheVersion [topVer=125430590, order=1513955001926, nodeOrder=3] ][ >>>> 3008806055072854, MovingObject, Kpx.Synesis.Outdoor, 0, , 1513946618964, >>>> 1513946624379, 171, 171, FALSE, FALSE, , FALSE, FALSE, 0, 0, 111259, >>>> 1514033024000, (vehicle, 0, human, 0, truck, 0), null, null, null, null, >>>> null, null, null, null, null, null, null, null ] >>>> at org.apache.ignite.internal.processors.cache.persistence. >>>> tree.BPlusTree.doRemove(BPlusTree.java:1787) >>>> at org.apache.ignite.internal.processors.cache.persistence. >>>> tree.BPlusTree.remove(BPlusTree.java:1578) >>>> at org.apache.ignite.internal.processors.query.h2.database. >>>> H2TreeIndex.remove(H2TreeIndex.java:216) >>>> at org.apache.ignite.internal.processors.query.h2.opt. >>>> GridH2Table.doUpdate(GridH2Table.java:496) >>>> at org.apache.ignite.internal.processors.query.h2.opt. >>>> GridH2Table.update(GridH2Table.java:423) >>>> at org.apache.ignite.internal.processors.query.h2. >>>> IgniteH2Indexing.remove(IgniteH2Indexing.java:580) >>>> at org.apache.ignite.internal.processors.query. >>>> GridQueryProcessor.remove(GridQueryProcessor.java:2334) >>>> at org.apache.ignite.internal.processors.cache.query. >>>> GridCacheQueryManager.remove(GridCacheQueryManager.java:461) >>>> at org.apache.ignite.internal.processors.cache. >>>> IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishRemove( >>>> IgniteCacheOffheapManagerImpl.java:1453) >>>> at org.apache.ignite.internal.processors.cache. >>>> IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.remove( >>>> IgniteCacheOffheapManagerImpl.java:1416) >>>> at org.apache.ignite.internal.processors.cache.persistence. >>>> GridCacheOffheapManager$GridCacheDataStore.remove( >>>> GridCacheOffheapManager.java:1271) >>>> at org.apache.ignite.internal.processors.cache. >>>> IgniteCacheOffheapManagerImpl.remove(IgniteCacheOffheapManagerImpl. >>>> java:374) >>>> at org.apache.ignite.internal.processors.cache. >>>> GridCacheMapEntry.removeValue(GridCacheMapEntry.java:3233) >>>> at org.apache.ignite.internal.processors.cache.distributed.dht. >>>> GridDhtCacheEntry.clearInternal(GridDhtCacheEntry.java:588) >>>> at org.apache.ignite.internal.processors.cache.distributed.dht. >>>> GridDhtLocalPartition.clearAll(GridDhtLocalPartition.java:951) >>>> at org.apache.ignite.internal.processors.cache.distributed.dht. >>>> GridDhtLocalPartition.tryEvict(GridDhtLocalPartition.java:809) >>>> at org.apache.ignite.internal.processors.cache.distributed.dht. >>>> preloader.GridDhtPreloader$3.call(GridDhtPreloader.java:593) >>>> at org.apache.ignite.internal.processors.cache.distributed.dht. >>>> preloader.GridDhtPreloader$3.call(GridDhtPreloader.java:580) >>>> at org.apache.ignite.internal.util.IgniteUtils. >>>> wrapThreadLoader(IgniteUtils.java:6631) >>>> at org.apache.ignite.internal.processors.closure. >>>> GridClosureProcessor$2.body(GridClosureProcessor.java:967) >>>> at org.apache.ignite.internal.util.worker.GridWorker.run( >>>> GridWorker.java:110) >>>> at java.util.concurrent.ThreadPoolExecutor.runWorker( >>>> ThreadPoolExecutor.java:1149) >>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run( >>>> ThreadPoolExecutor.java:624) >>>> at java.lang.Thread.run(Thread.java:748) >>>> Caused by: java.lang.IllegalStateException: Failed to get page IO >>>> instance (page content is corrupted) >>>> at org.apache.ignite.internal.processors.cache.persistence. >>>> tree.io.IOVersions.forVersion(IOVersions.java:83) >>>> at org.apache.ignite.internal.processors.cache.persistence. >>>> tree.io.IOVersions.forPage(IOVersions.java:95) >>>> at org.apache.ignite.internal.processors.cache.persistence. >>>> CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:148) >>>> at org.apache.ignite.internal.processors.cache.persistence. >>>> CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102) >>>> at org.apache.ignite.internal.processors.query.h2.database. >>>> H2RowFactory.getRow(H2RowFactory.java:62) >>>> at org.apache.ignite.internal.processors.query.h2.database.io. >>>> H2ExtrasLeafIO.getLookupRow(H2ExtrasLeafIO.java:126) >>>> at org.apache.ignite.internal.processors.query.h2.database.io. >>>> H2ExtrasLeafIO.getLookupRow(H2ExtrasLeafIO.java:36) >>>> at org.apache.ignite.internal.processors.query.h2.database. >>>> H2Tree.getRow(H2Tree.java:123) >>>> at org.apache.ignite.internal.processors.query.h2.database. >>>> H2Tree.getRow(H2Tree.java:40) >>>> at org.apache.ignite.internal.processors.cache.persistence. >>>> tree.BPlusTree.getRow(BPlusTree.java:4372) >>>> at org.apache.ignite.internal.processors.query.h2.database. >>>> H2Tree.compare(H2Tree.java:200) >>>> at org.apache.ignite.internal.processors.query.h2.database. >>>> H2Tree.compare(H2Tree.java:40) >>>> at org.apache.ignite.internal.processors.cache.persistence. >>>> tree.BPlusTree.compare(BPlusTree.java:4359) >>>> at org.apache.ignite.internal.processors.cache.persistence. >>>> tree.BPlusTree.findInsertionPoint(BPlusTree.java:4279) >>>> at org.apache.ignite.internal.processors.cache.persistence. >>>> tree.BPlusTree.access$1500(BPlusTree.java:81) >>>> at org.apache.ignite.internal.processors.cache.persistence. >>>> tree.BPlusTree$Search.run0(BPlusTree.java:261) >>>> at org.apache.ignite.internal.processors.cache.persistence. >>>> tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4697) >>>> at org.apache.ignite.internal.processors.cache.persistence. >>>> tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4682) >>>> at org.apache.ignite.internal.processors.cache.persistence. >>>> tree.util.PageHandler.readPage(PageHandler.java:158) >>>> at org.apache.ignite.internal.processors.cache.persistence. >>>> DataStructure.read(DataStructure.java:319) >>>> at org.apache.ignite.internal.processors.cache.persistence. >>>> tree.BPlusTree.removeDown(BPlusTree.java:1823) >>>> at org.apache.ignite.internal.processors.cache.persistence. >>>> tree.BPlusTree.removeDown(BPlusTree.java:1842) >>>> at org.apache.ignite.internal.processors.cache.persistence. >>>> tree.BPlusTree.removeDown(BPlusTree.java:1842) >>>> at org.apache.ignite.internal.processors.cache.persistence. >>>> tree.BPlusTree.removeDown(BPlusTree.java:1842) >>>> at org.apache.ignite.internal.processors.cache.persistence. >>>> tree.BPlusTree.doRemove(BPlusTree.java:1752) >>>> ... 23 more >>>> >>>> >>>> After restart we also get this error. See *ignite-instance-2.log*. >>>> >>>> The *cache-config.xml* is used for *server* instances. >>>> The *ignite-common-cache-conf.xml* is used for *client* instances >>>> which activete cluster and stream data from Kafka into Ignite. >>>> >>>> *Is it possible to tune up (or implement) native persistence in a way >>>> when it just reports about error in data or corrupted data, then skip it >>>> and continue to work without that corrupted part. Thus it will make the >>>> cluster to continue operating regardless of errors on storage?* >>>> >>>> >>>> >>>> Arseny Kovalchuk >>>> >>>> Senior Software Engineer at Synesis >>>> skype: arseny.kovalchuk >>>> mobile: +375 (29) 666-16-16 <+375%2029%20666-16-16> >>>> LinkedIn Profile <http://www.linkedin.com/in/arsenykovalchuk/en> >>>> >>>> <ignite-instance-0.log><ignite-instance-1.log><ignite-instance-2.log> >>>> <ignite-instance-3.log><ignite-instance-4.log><cache-config.xml> >>>> <ignite-discovery-kubernetes.xml><ignite-common.xml> >>>> <ignite-common-storage.xml><ignite-common-entity.xml> >>>> >>>>
. ____ _ __ _ _ /\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \ ( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \ \\/ ___)| |_)| | | | | || (_| | ) ) ) ) ' |____| .__|_| |_|_| |_\__, | / / / / =========|_|==============|___/=/_/_/_/ :: Spring Boot :: (v1.5.9.RELEASE) 2018-03-16 18:15:27.115 INFO 28792 --- [ main] ru.synesis.kipod.DataTestBootApp : Starting DataTestBootApp on arseny-ThinkPad-T460p with PID 28792 (/home/arseny/ignite-logs/project/bin started by arseny in /home/arseny/ignite-logs/project) 2018-03-16 18:15:27.116 DEBUG 28792 --- [ main] ru.synesis.kipod.DataTestBootApp : Running with Spring Boot v1.5.9.RELEASE, Spring v4.3.13.RELEASE 2018-03-16 18:15:27.116 INFO 28792 --- [ main] ru.synesis.kipod.DataTestBootApp : The following profiles are active: standalone 2018-03-16 18:15:27.150 INFO 28792 --- [ main] s.c.a.AnnotationConfigApplicationContext : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@29176cc1: startup date [Fri Mar 16 18:15:27 MSK 2018]; root of context hierarchy 2018-03-16 18:15:27.371 INFO 28792 --- [ main] o.s.b.f.xml.XmlBeanDefinitionReader : Loading XML bean definitions from URL [file:/home/arseny/ignite-logs/project/bin/ignite-common-entity.xml] 2018-03-16 18:15:27.452 INFO 28792 --- [ main] o.s.b.f.xml.XmlBeanDefinitionReader : Loading XML bean definitions from URL [file:/home/arseny/ignite-logs/project/bin/ignite-common-storage.xml] 2018-03-16 18:15:27.469 INFO 28792 --- [ main] o.s.b.f.xml.XmlBeanDefinitionReader : Loading XML bean definitions from URL [file:/home/arseny/ignite-logs/project/bin/ignite-common.xml] 2018-03-16 18:15:27.481 INFO 28792 --- [ main] o.s.b.f.xml.XmlBeanDefinitionReader : Loading XML bean definitions from URL [file:/home/arseny/ignite-logs/project/bin/data-test.xml] 2018-03-16 18:15:27.876 INFO 28792 --- [ main] o.a.i.i.IgniteKernal%ignite-instance-5 : >>> __________ ________________ >>> / _/ ___/ |/ / _/_ __/ __/ >>> _/ // (7 7 // / / / / _/ >>> /___/\___/_/|_/___/ /_/ /___/ >>> >>> ver. 2.4.0#20180305-sha1:aa342270 >>> 2018 Copyright(C) Apache Software Foundation >>> >>> Ignite documentation: http://ignite.apache.org [18:15:27] __________ ________________ [18:15:27] / _/ ___/ |/ / _/_ __/ __/ [18:15:27] _/ // (7 7 // / / / / _/ [18:15:27] /___/\___/_/|_/___/ /_/ /___/ [18:15:27] [18:15:27] ver. 2.4.0#20180305-sha1:aa342270 [18:15:27] 2018 Copyright(C) Apache Software Foundation [18:15:27] [18:15:27] Ignite documentation: http://ignite.apache.org [18:15:27] [18:15:27] Quiet mode. [18:15:27] ^-- Logging by 'Slf4jLogger [impl=Logger[o.a.i.i.IgniteKernal%ignite-instance-5], quiet=true]' [18:15:27] ^-- To see **FULL** console log here add -DIGNITE_QUIET=false or "-v" to ignite.{sh|bat} [18:15:27] 2018-03-16 18:15:27.879 INFO 28792 --- [ main] o.a.i.i.IgniteKernal%ignite-instance-5 : Config URL: n/a 2018-03-16 18:15:27.882 INFO 28792 --- [ main] o.a.i.i.IgniteKernal%ignite-instance-5 : IgniteConfiguration [igniteInstanceName=ignite-instance-5, pubPoolSize=8, svcPoolSize=8, callbackPoolSize=8, stripedPoolSize=8, sysPoolSize=8, mgmtPoolSize=4, igfsPoolSize=4, dataStreamerPoolSize=8, utilityCachePoolSize=4, utilityCacheKeepAliveTime=60000, p2pPoolSize=4, qryPoolSize=8, igniteHome=null, igniteWorkDir=/home/arseny/ignite-logs/data5, mbeanSrv=com.sun.jmx.mbeanserver.JmxMBeanServer@6f94fa3e, nodeId=d62b2da2-771a-45dc-b30c-8a93d21295a1, marsh=org.apache.ignite.internal.binary.BinaryMarshaller@2262b621, marshLocJobs=false, daemon=false, p2pEnabled=false, netTimeout=5000, sndRetryDelay=1000, sndRetryCnt=3, metricsHistSize=10000, metricsUpdateFreq=2000, metricsExpTime=9223372036854775807, discoSpi=TcpDiscoverySpi [addrRslvr=null, sockTimeout=0, ackTimeout=0, marsh=null, reconCnt=10, reconDelay=2000, maxAckTimeout=600000, forceSrvMode=false, clientReconnectDisabled=false], segPlc=STOP, segResolveAttempts=2, waitForSegOnStart=true, allResolversPassReq=true, segChkFreq=10000, commSpi=TcpCommunicationSpi [connectGate=null, connPlc=null, enableForcibleNodeKill=false, enableTroubleshootingLog=false, srvLsnr=org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi$2@7e928e2f, locAddr=null, locHost=null, locPort=47100, locPortRange=100, shmemPort=-1, directBuf=true, directSndBuf=false, idleConnTimeout=600000, connTimeout=5000, maxConnTimeout=600000, reconCnt=10, sockSndBuf=32768, sockRcvBuf=32768, msgQueueLimit=0, slowClientQueueLimit=0, nioSrvr=null, shmemSrv=null, usePairedConnections=false, connectionsPerNode=1, tcpNoDelay=true, filterReachableAddresses=false, ackSndThreshold=32, unackedMsgsBufSize=0, sockWriteTimeout=2000, lsnr=null, boundTcpPort=-1, boundTcpShmemPort=-1, selectorsCnt=4, selectorSpins=0, addrRslvr=null, ctxInitLatch=java.util.concurrent.CountDownLatch@f667fe[Count = 1], stopping=false, metricsLsnr=org.apache.ignite.spi.communication.tcp.TcpCommunicationMetricsListener@788fcafb], evtSpi=org.apache.ignite.spi.eventstorage.NoopEventStorageSpi@4febb875, colSpi=NoopCollisionSpi [], deploySpi=LocalDeploymentSpi [lsnr=null], indexingSpi=org.apache.ignite.spi.indexing.noop.NoopIndexingSpi@328cf0e1, addrRslvr=null, clientMode=false, rebalanceThreadPoolSize=1, txCfg=org.apache.ignite.configuration.TransactionConfiguration@63b1d4fa, cacheSanityCheckEnabled=true, discoStartupDelay=60000, deployMode=SHARED, p2pMissedCacheSize=100, locHost=null, timeSrvPortBase=31100, timeSrvPortRange=100, failureDetectionTimeout=60000, clientFailureDetectionTimeout=60000, metricsLogFreq=0, hadoopCfg=null, connectorCfg=org.apache.ignite.configuration.ConnectorConfiguration@42e3ede4, odbcCfg=null, warmupClos=null, atomicCfg=AtomicConfiguration [seqReserveSize=1000, cacheMode=PARTITIONED, backups=1, aff=null, grpName=null], classLdr=null, sslCtxFactory=null, platformCfg=null, binaryCfg=null, memCfg=null, pstCfg=null, dsCfg=DataStorageConfiguration [sysRegionInitSize=41943040, sysCacheMaxSize=104857600, pageSize=4096, concLvl=8, dfltDataRegConf=DataRegionConfiguration [name=default, maxSize=536870912, initSize=67108864, swapPath=null, pageEvictionMode=RANDOM_2_LRU, evictionThreshold=0.85, emptyPagesPoolSize=1024, metricsEnabled=false, metricsSubIntervalCount=5, metricsRateTimeInterval=60000, persistenceEnabled=true, checkpointPageBufSize=134217728], storagePath=null, checkpointFreq=180000, lockWaitTime=10000, checkpointThreads=4, checkpointWriteOrder=SEQUENTIAL, walHistSize=5, walSegments=10, walSegmentSize=268435456, walPath=db/wal, walArchivePath=db/wal/archive, metricsEnabled=false, walMode=BACKGROUND, walTlbSize=131072, walBuffSize=0, walFlushFreq=10000, walFsyncDelay=1000, walRecordIterBuffSize=67108864, alwaysWriteFullPages=false, fileIOFactory=org.apache.ignite.internal.processors.cache.persistence.file.AsyncFileIOFactory@201b6b6f, metricsSubIntervalCnt=5, metricsRateTimeInterval=60000, walAutoArchiveAfterInactivity=-1, writeThrottlingEnabled=false, walCompactionEnabled=false], activeOnStart=true, autoActivation=true, longQryWarnTimeout=3000, sqlConnCfg=null, cliConnCfg=ClientConnectorConfiguration [host=null, port=10800, portRange=100, sockSndBufSize=0, sockRcvBufSize=0, tcpNoDelay=true, maxOpenCursorsPerConn=128, threadPoolSize=8, idleTimeout=0, jdbcEnabled=true, odbcEnabled=true, thinCliEnabled=true]] 2018-03-16 18:15:27.882 INFO 28792 --- [ main] o.a.i.i.IgniteKernal%ignite-instance-5 : Daemon mode: off [18:15:27] OS: Linux 4.13.0-36-generic amd64 2018-03-16 18:15:27.882 INFO 28792 --- [ main] o.a.i.i.IgniteKernal%ignite-instance-5 : OS: Linux 4.13.0-36-generic amd64 2018-03-16 18:15:27.882 INFO 28792 --- [ main] o.a.i.i.IgniteKernal%ignite-instance-5 : OS user: arseny 2018-03-16 18:15:27.882 INFO 28792 --- [ main] o.a.i.i.IgniteKernal%ignite-instance-5 : PID: 28792 [18:15:27] VM information: OpenJDK Runtime Environment 1.8.0_151-8u151-b12-0ubuntu0.17.10.2-b12 Oracle Corporation OpenJDK 64-Bit Server VM 25.151-b12 2018-03-16 18:15:27.882 INFO 28792 --- [ main] o.a.i.i.IgniteKernal%ignite-instance-5 : Language runtime: Java Platform API Specification ver. 1.8 2018-03-16 18:15:27.882 INFO 28792 --- [ main] o.a.i.i.IgniteKernal%ignite-instance-5 : VM information: OpenJDK Runtime Environment 1.8.0_151-8u151-b12-0ubuntu0.17.10.2-b12 Oracle Corporation OpenJDK 64-Bit Server VM 25.151-b12 2018-03-16 18:15:27.883 INFO 28792 --- [ main] o.a.i.i.IgniteKernal%ignite-instance-5 : VM total memory: 6.9GB 2018-03-16 18:15:27.883 INFO 28792 --- [ main] o.a.i.i.IgniteKernal%ignite-instance-5 : Remote Management [restart: off, REST: on, JMX (remote: on, port: 46363, auth: off, ssl: off)] 2018-03-16 18:15:27.883 INFO 28792 --- [ main] o.a.i.i.IgniteKernal%ignite-instance-5 : Logger: Slf4jLogger [impl=Logger[org.apache.ignite.internal.IgniteKernal%ignite-instance-5], quiet=true] 2018-03-16 18:15:27.883 INFO 28792 --- [ main] o.a.i.i.IgniteKernal%ignite-instance-5 : IGNITE_HOME=null 2018-03-16 18:15:27.883 INFO 28792 --- [ main] o.a.i.i.IgniteKernal%ignite-instance-5 : VM arguments: [-Dcom.sun.management.jmxremote, -Dcom.sun.management.jmxremote.port=46363, -Dcom.sun.management.jmxremote.authenticate=false, -Dcom.sun.management.jmxremote.ssl=false, -Djava.rmi.server.hostname=localhost, -Dspring.liveBeansView.mbeanDomain, -Dspring.application.admin.enabled=true, -Djava.security.egd=file:/dev/./urandom, -Xverify:none, -XX:TieredStopAtLevel=1, -Dfile.encoding=UTF-8] 2018-03-16 18:15:27.883 INFO 28792 --- [ main] o.a.i.i.IgniteKernal%ignite-instance-5 : System cache's DataRegion size is configured to 40 MB. Use DataStorageConfiguration.systemCacheMemorySize property to change the setting. 2018-03-16 18:15:27.887 INFO 28792 --- [ main] o.a.i.i.IgniteKernal%ignite-instance-5 : Configured caches [in 'sysMemPlc' dataRegion: ['ignite-sys-cache']] 2018-03-16 18:15:27.889 WARN 28792 --- [ite-instance-5%] o.apache.ignite.internal.GridDiagnostic : Initial heap size is 500MB (should be no less than 512MB, use -Xms512m -Xmx512m). [18:15:27] Initial heap size is 500MB (should be no less than 512MB, use -Xms512m -Xmx512m). [18:15:27] Configured plugins: 2018-03-16 18:15:27.918 INFO 28792 --- [ main] o.a.i.i.p.plugin.IgnitePluginProcessor : Configured plugins: [18:15:27] ^-- None 2018-03-16 18:15:27.919 INFO 28792 --- [ main] o.a.i.i.p.plugin.IgnitePluginProcessor : ^-- None [18:15:27] 2018-03-16 18:15:27.919 INFO 28792 --- [ main] o.a.i.i.p.plugin.IgnitePluginProcessor : 2018-03-16 18:15:27.935 INFO 28792 --- [ main] o.a.i.s.c.tcp.TcpCommunicationSpi : Successfully bound communication NIO server to TCP port [port=47100, locHost=0.0.0.0/0.0.0.0, selectorsCnt=4, selectorSpins=0, pairedConn=false] 2018-03-16 18:15:28.097 WARN 28792 --- [ main] o.a.i.s.c.tcp.TcpCommunicationSpi : Message queue limit is set to 0 which may lead to potential OOMEs when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender and receiver sides. [18:15:28] Message queue limit is set to 0 which may lead to potential OOMEs when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender and receiver sides. 2018-03-16 18:15:28.109 WARN 28792 --- [ main] o.a.i.s.c.noop.NoopCheckpointSpi : Checkpoints are disabled (to enable configure any GridCheckpointSpi implementation) 2018-03-16 18:15:28.135 WARN 28792 --- [ main] o.a.i.i.m.c.GridCollisionManager : Collision resolution is disabled (all jobs will be activated upon arrival). [18:15:28] Security status [authentication=off, tls/ssl=off] 2018-03-16 18:15:28.136 INFO 28792 --- [ main] o.a.i.i.IgniteKernal%ignite-instance-5 : Security status [authentication=off, tls/ssl=off] 2018-03-16 18:15:28.147 INFO 28792 --- [ main] i.i.p.c.b.CacheObjectBinaryProcessorImpl : Resolved directory for serialized binary metadata: /home/arseny/ignite-logs/data5/binary_meta/ignite_instance_5 2018-03-16 18:15:28.234 INFO 28792 --- [ main] o.a.i.i.p.c.p.file.FilePageStoreManager : Resolved page store work directory: /home/arseny/ignite-logs/data5/db/ignite_instance_5 2018-03-16 18:15:28.235 INFO 28792 --- [ main] o.a.i.i.p.c.p.w.FileWriteAheadLogManager : Resolved write ahead log work directory: /home/arseny/ignite-logs/data5/db/wal/ignite_instance_5 2018-03-16 18:15:28.235 INFO 28792 --- [ main] o.a.i.i.p.c.p.w.FileWriteAheadLogManager : Resolved write ahead log archive directory: /home/arseny/ignite-logs/data5/db/wal/archive/ignite_instance_5 2018-03-16 18:15:28.241 INFO 28792 --- [ main] o.a.i.i.p.c.p.w.FileWriteAheadLogManager : Started write-ahead log manager [mode=BACKGROUND] 2018-03-16 18:15:28.241 WARN 28792 --- [ main] i.i.p.c.p.GridCacheDatabaseSharedManager : Page eviction mode set for [default] data will have no effect because the oldest pages are evicted automatically if Ignite persistence is enabled. 2018-03-16 18:15:28.251 INFO 28792 --- [ main] i.i.p.c.p.GridCacheDatabaseSharedManager : Read checkpoint status [startMarker=/home/arseny/ignite-logs/data5/db/ignite_instance_5/cp/1521101654990-5f49b673-0403-4067-ae91-3af39662c87b-START.bin, endMarker=/home/arseny/ignite-logs/data5/db/ignite_instance_5/cp/1521101654990-5f49b673-0403-4067-ae91-3af39662c87b-END.bin] 2018-03-16 18:15:28.257 INFO 28792 --- [ main] o.a.i.i.p.c.p.pagemem.PageMemoryImpl : Started page memory [memoryAllocated=100.0 MiB, pages=24936, tableSize=1.4 MiB, checkpointBuffer=100.0 MiB] 2018-03-16 18:15:28.258 INFO 28792 --- [ main] i.i.p.c.p.GridCacheDatabaseSharedManager : Checking memory state [lastValidPos=FileWALPointer [idx=111, fileOff=143312353, len=13471], lastMarked=FileWALPointer [idx=111, fileOff=143312353, len=13471], lastCheckpointId=5f49b673-0403-4067-ae91-3af39662c87b] 2018-03-16 18:15:28.294 INFO 28792 --- [ main] i.i.p.c.p.GridCacheDatabaseSharedManager : Found last checkpoint marker [cpId=5f49b673-0403-4067-ae91-3af39662c87b, pos=FileWALPointer [idx=111, fileOff=143312353, len=13471]] 2018-03-16 18:15:31.049 INFO 28792 --- [ main] o.a.i.i.p.c.p.w.FileWriteAheadLogManager : Stopping WAL iteration due to an exception: EOF at position [85689935] expected to read [1] bytes, ptr=FileWALPointer [idx=113, fileOff=85689935, len=0] 2018-03-16 18:15:31.055 INFO 28792 --- [ main] i.i.p.c.p.GridCacheDatabaseSharedManager : Applying lost cache updates since last checkpoint record [lastMarked=FileWALPointer [idx=111, fileOff=143312353, len=13471], lastCheckpointId=5f49b673-0403-4067-ae91-3af39662c87b] 2018-03-16 18:15:33.604 INFO 28792 --- [ main] o.a.i.i.p.c.p.w.FileWriteAheadLogManager : Stopping WAL iteration due to an exception: EOF at position [85689935] expected to read [1] bytes, ptr=FileWALPointer [idx=113, fileOff=85689935, len=0] 2018-03-16 18:15:33.607 INFO 28792 --- [ main] i.i.p.c.p.GridCacheDatabaseSharedManager : Finished applying WAL changes [updatesApplied=0, time=2553ms] 2018-03-16 18:15:33.640 ERROR 28792 --- [ main] : Failed to resolve default logging config file: config/java.util.logging.properties Console logging handler is not configured. 2018-03-16 18:15:33.647 INFO 28792 --- [ main] o.a.i.i.p.odbc.ClientListenerProcessor : Client connector processor has started on TCP port 10800 2018-03-16 18:15:33.661 INFO 28792 --- [ main] o.a.i.i.p.r.p.tcp.GridTcpRestProtocol : Command protocol successfully started [name=TCP binary, host=0.0.0.0/0.0.0.0, port=11211] 2018-03-16 18:15:33.676 INFO 28792 --- [ main] o.a.i.i.IgniteKernal%ignite-instance-5 : Non-loopback local IPs: 172.17.0.1, 192.168.0.62, fe80:0:0:0:40c9:ddf9:8d96:29ab%wlp3s0, fe80:0:0:0:42:15ff:fe1b:937c%docker0 2018-03-16 18:15:33.676 INFO 28792 --- [ main] o.a.i.i.IgniteKernal%ignite-instance-5 : Enabled local MACs: 0242151B937C, E4A471E0B834 2018-03-16 18:15:33.684 INFO 28792 --- [ main] o.a.i.spi.discovery.tcp.TcpDiscoverySpi : Successfully bound to TCP port [port=47500, localHost=0.0.0.0/0.0.0.0, locNodeId=d62b2da2-771a-45dc-b30c-8a93d21295a1] 2018-03-16 18:15:33.685 WARN 28792 --- [ main] .s.d.t.i.m.TcpDiscoveryMulticastIpFinder : TcpDiscoveryMulticastIpFinder has no pre-configured addresses (it is recommended in production to specify at least one address in TcpDiscoveryMulticastIpFinder.getAddresses() configuration property) [18:15:34] Performance suggestions for grid 'ignite-instance-5' (fix if possible) 2018-03-16 18:15:34.939 INFO 28792 --- [ main] o.a.i.i.IgniteKernal%ignite-instance-5 : Performance suggestions for grid 'ignite-instance-5' (fix if possible) [18:15:34] To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true 2018-03-16 18:15:34.940 INFO 28792 --- [ main] o.a.i.i.IgniteKernal%ignite-instance-5 : To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true [18:15:34] ^-- Enable G1 Garbage Collector (add '-XX:+UseG1GC' to JVM options) 2018-03-16 18:15:34.940 INFO 28792 --- [ main] o.a.i.i.IgniteKernal%ignite-instance-5 : ^-- Enable G1 Garbage Collector (add '-XX:+UseG1GC' to JVM options) [18:15:34] ^-- Specify JVM heap max size (add '-Xmx<size>[g|G|m|M|k|K]' to JVM options) 2018-03-16 18:15:34.940 INFO 28792 --- [ main] o.a.i.i.IgniteKernal%ignite-instance-5 : ^-- Specify JVM heap max size (add '-Xmx<size>[g|G|m|M|k|K]' to JVM options) [18:15:34] ^-- Set max direct memory size if getting 'OOME: Direct buffer memory' (add '-XX:MaxDirectMemorySize=<size>[g|G|m|M|k|K]' to JVM options) 2018-03-16 18:15:34.940 INFO 28792 --- [ main] o.a.i.i.IgniteKernal%ignite-instance-5 : ^-- Set max direct memory size if getting 'OOME: Direct buffer memory' (add '-XX:MaxDirectMemorySize=<size>[g|G|m|M|k|K]' to JVM options) [18:15:34] ^-- Disable processing of calls to System.gc() (add '-XX:+DisableExplicitGC' to JVM options) 2018-03-16 18:15:34.940 INFO 28792 --- [ main] o.a.i.i.IgniteKernal%ignite-instance-5 : ^-- Disable processing of calls to System.gc() (add '-XX:+DisableExplicitGC' to JVM options) [18:15:34] Refer to this page for more performance suggestions: https://apacheignite.readme.io/docs/jvm-and-system-tuning 2018-03-16 18:15:34.940 INFO 28792 --- [ main] o.a.i.i.IgniteKernal%ignite-instance-5 : Refer to this page for more performance suggestions: https://apacheignite.readme.io/docs/jvm-and-system-tuning [18:15:34] 2018-03-16 18:15:34.940 INFO 28792 --- [ main] o.a.i.i.IgniteKernal%ignite-instance-5 : [18:15:34] To start Console Management & Monitoring run ignitevisorcmd.{sh|bat} 2018-03-16 18:15:34.940 INFO 28792 --- [ main] o.a.i.i.IgniteKernal%ignite-instance-5 : To start Console Management & Monitoring run ignitevisorcmd.{sh|bat} [18:15:34] [18:15:34] Ignite node started OK (id=d62b2da2, instance name=ignite-instance-5) 2018-03-16 18:15:34.940 INFO 28792 --- [ main] o.a.i.i.IgniteKernal%ignite-instance-5 : 2018-03-16 18:15:34.941 INFO 28792 --- [ main] o.a.i.i.IgniteKernal%ignite-instance-5 : >>> +----------------------------------------------------------------------+ >>> Ignite ver. 2.4.0#20180305-sha1:aa342270b13cc1f4713382a8eb23b2eb7edaa3a5 >>> +----------------------------------------------------------------------+ >>> OS name: Linux 4.13.0-36-generic amd64 >>> CPU(s): 8 >>> Heap: 6.9GB >>> VM name: 28792@arseny-ThinkPad-T460p >>> Ignite instance name: ignite-instance-5 >>> Local node [ID=D62B2DA2-771A-45DC-B30C-8A93D21295A1, order=1, clientMode=false] >>> Local node addresses: [172.17.0.1/0:0:0:0:0:0:0:1%lo, 192.168.0.62/127.0.0.1, /172.17.0.1, /192.168.0.62] >>> Local ports: TCP:10800 TCP:11211 TCP:47100 UDP:47400 TCP:47500 [18:15:34] Topology snapshot [ver=1, servers=1, clients=0, CPUs=8, offheap=2.5GB, heap=6.9GB] [18:15:34] Data Regions Configured: [18:15:34] ^-- default [initSize=64.0 MiB, maxSize=512.0 MiB, persistenceEnabled=true] [18:15:34] ^-- Custom_Region [initSize=1.0 GiB, maxSize=2.0 GiB, persistenceEnabled=true] 2018-03-16 18:15:34.942 INFO 28792 --- [ main] o.a.i.i.m.d.GridDiscoveryManager : Topology snapshot [ver=1, servers=1, clients=0, CPUs=8, offheap=2.5GB, heap=6.9GB] 2018-03-16 18:15:34.942 INFO 28792 --- [ main] o.a.i.i.m.d.GridDiscoveryManager : Data Regions Configured: 2018-03-16 18:15:34.942 INFO 28792 --- [ main] o.a.i.i.m.d.GridDiscoveryManager : ^-- default [initSize=64.0 MiB, maxSize=512.0 MiB, persistenceEnabled=true] 2018-03-16 18:15:34.942 INFO 28792 --- [ main] o.a.i.i.m.d.GridDiscoveryManager : ^-- Custom_Region [initSize=1.0 GiB, maxSize=2.0 GiB, persistenceEnabled=true] 2018-03-16 18:15:34.978 INFO 28792 --- [ main] o.a.i.i.p.c.GridClusterStateProcessor : Sending activate request with BaselineTopology BaselineTopology [id=0, branchingHash=-1777855712, branchingType='New BaselineTopology', baselineNodes=[ignite-instance-5]] 2018-03-16 18:15:34.988 INFO 28792 --- [ite-instance-5%] o.a.i.i.p.c.GridClusterStateProcessor : Received activate request with BaselineTopology[id=0] 2018-03-16 18:15:34.989 INFO 28792 --- [ite-instance-5%] o.a.i.i.p.c.GridClusterStateProcessor : Started state transition: true 2018-03-16 18:15:34.994 INFO 28792 --- [ite-instance-5%] o.apache.ignite.internal.exchange.time : Started exchange init [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], crd=true, evt=DISCOVERY_CUSTOM_EVT, evtNode=d62b2da2-771a-45dc-b30c-8a93d21295a1, customEvt=ChangeGlobalStateMessage [id=889416f2261-904f0bc2-4419-44e0-92f2-346f3a93b5e7, reqId=26f99179-91fe-468d-9c11-6de38f91139c, initiatingNodeId=d62b2da2-771a-45dc-b30c-8a93d21295a1, activate=true, baselineTopology=BaselineTopology [id=0, branchingHash=-1777855712, branchingType='New BaselineTopology', baselineNodes=[ignite-instance-5]], forceChangeBaselineTopology=false, timestamp=1521213334978], allowMerge=false] 2018-03-16 18:15:34.994 INFO 28792 --- [ite-instance-5%] .c.d.d.p.GridDhtPartitionsExchangeFuture : Start activation process [nodeId=d62b2da2-771a-45dc-b30c-8a93d21295a1, client=false, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1]] 2018-03-16 18:15:35.005 INFO 28792 --- [ite-instance-5%] o.a.i.i.p.c.p.file.FilePageStoreManager : Resolved page store work directory: /home/arseny/ignite-logs/data5/db/ignite_instance_5 2018-03-16 18:15:35.006 INFO 28792 --- [ite-instance-5%] o.a.i.i.p.c.p.w.FileWriteAheadLogManager : Resolved write ahead log work directory: /home/arseny/ignite-logs/data5/db/wal/ignite_instance_5 2018-03-16 18:15:35.006 INFO 28792 --- [ite-instance-5%] o.a.i.i.p.c.p.w.FileWriteAheadLogManager : Resolved write ahead log archive directory: /home/arseny/ignite-logs/data5/db/wal/archive/ignite_instance_5 2018-03-16 18:15:35.006 INFO 28792 --- [ite-instance-5%] o.a.i.i.p.c.p.w.FileWriteAheadLogManager : Started write-ahead log manager [mode=BACKGROUND] 2018-03-16 18:15:35.010 INFO 28792 --- [ite-instance-5%] o.a.i.i.p.c.p.pagemem.PageMemoryImpl : Started page memory [memoryAllocated=100.0 MiB, pages=24936, tableSize=1.4 MiB, checkpointBuffer=100.0 MiB] 2018-03-16 18:15:35.021 INFO 28792 --- [ite-instance-5%] o.a.i.i.p.c.p.pagemem.PageMemoryImpl : Started page memory [memoryAllocated=2.0 GiB, pages=510712, tableSize=29.7 MiB, checkpointBuffer=1.0 GiB] 2018-03-16 18:15:35.024 INFO 28792 --- [ite-instance-5%] o.a.i.i.p.c.p.pagemem.PageMemoryImpl : Started page memory [memoryAllocated=512.0 MiB, pages=127672, tableSize=7.4 MiB, checkpointBuffer=128.0 MiB] 2018-03-16 18:15:35.025 INFO 28792 --- [ite-instance-5%] o.a.i.i.p.c.p.pagemem.PageMemoryImpl : Started page memory [memoryAllocated=100.0 MiB, pages=24936, tableSize=1.4 MiB, checkpointBuffer=100.0 MiB] 2018-03-16 18:15:35.109 INFO 28792 --- [ite-instance-5%] i.i.p.c.p.GridCacheDatabaseSharedManager : Read checkpoint status [startMarker=/home/arseny/ignite-logs/data5/db/ignite_instance_5/cp/1521101654990-5f49b673-0403-4067-ae91-3af39662c87b-START.bin, endMarker=/home/arseny/ignite-logs/data5/db/ignite_instance_5/cp/1521101654990-5f49b673-0403-4067-ae91-3af39662c87b-END.bin] 2018-03-16 18:15:35.110 INFO 28792 --- [ite-instance-5%] i.i.p.c.p.GridCacheDatabaseSharedManager : Checking memory state [lastValidPos=FileWALPointer [idx=111, fileOff=143312353, len=13471], lastMarked=FileWALPointer [idx=111, fileOff=143312353, len=13471], lastCheckpointId=5f49b673-0403-4067-ae91-3af39662c87b] 2018-03-16 18:15:35.134 INFO 28792 --- [ite-instance-5%] i.i.p.c.p.GridCacheDatabaseSharedManager : Found last checkpoint marker [cpId=5f49b673-0403-4067-ae91-3af39662c87b, pos=FileWALPointer [idx=111, fileOff=143312353, len=13471]] 2018-03-16 18:15:40.900 INFO 28792 --- [ite-instance-5%] o.a.i.i.p.c.p.w.FileWriteAheadLogManager : Stopping WAL iteration due to an exception: EOF at position [85689935] expected to read [1] bytes, ptr=FileWALPointer [idx=113, fileOff=85689935, len=0] 2018-03-16 18:15:40.910 INFO 28792 --- [ite-instance-5%] o.a.i.i.p.c.p.w.FileWriteAheadLogManager : Resuming logging to WAL segment [file=/home/arseny/ignite-logs/data5/db/wal/ignite_instance_5/0000000000000003.wal, offset=85689935, ver=1] 2018-03-16 18:15:40.911 INFO 28792 --- [ite-instance-5%] o.a.i.i.p.c.p.w.FileWriteAheadLogManager : Record serializer version change detected, will start logging with a new WAL record serializer to a new WAL segment [curFile=org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileWriteHandle@12c9b2c7, newVer=2, oldVer=1] 2018-03-16 18:15:40.958 INFO 28792 --- [ite-instance-5%] o.a.i.i.p.c.GridClusterStateProcessor : Writing BaselineTopology[id=0] 2018-03-16 18:15:41.144 INFO 28792 --- [ite-instance-5%] o.a.i.i.p.cache.GridCacheProcessor : Started cache [name=ignite-sys-cache, id=-2100569601, memoryPolicyName=sysMemPlc, mode=REPLICATED, atomicity=TRANSACTIONAL, backups=2147483647] 2018-03-16 18:15:41.184 INFO 28792 --- [ite-instance-5%] o.a.i.i.p.cache.GridCacheProcessor : Started cache [name=face_events, id=-1984391877, memoryPolicyName=Custom_Region, mode=PARTITIONED, atomicity=ATOMIC, backups=0] 2018-03-16 18:15:41.200 INFO 28792 --- [ite-instance-5%] o.a.i.i.p.cache.GridCacheProcessor : Started cache [name=kipod_events, id=1531474993, memoryPolicyName=Custom_Region, mode=PARTITIONED, atomicity=ATOMIC, backups=0] 2018-03-16 18:15:41.203 INFO 28792 --- [ite-instance-5%] o.a.i.i.p.cache.GridCacheProcessor : Started cache [name=ignite-sys-atomic-cache@default-ds-group, id=1481046058, group=default-ds-group, memoryPolicyName=default, mode=PARTITIONED, atomicity=TRANSACTIONAL, backups=0] 2018-03-16 18:15:41.209 INFO 28792 --- [ite-instance-5%] .c.d.d.p.GridDhtPartitionsExchangeFuture : Successfully activated caches [nodeId=d62b2da2-771a-45dc-b30c-8a93d21295a1, client=false, topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1]] 2018-03-16 18:15:41.212 INFO 28792 --- [ite-instance-5%] .c.d.d.p.GridDhtPartitionsExchangeFuture : Finished waiting for partition release future [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], waitTime=0ms, futInfo=NA] 2018-03-16 18:15:41.214 INFO 28792 --- [ite-instance-5%] i.i.p.c.p.GridCacheDatabaseSharedManager : Read checkpoint status [startMarker=/home/arseny/ignite-logs/data5/db/ignite_instance_5/cp/1521101654990-5f49b673-0403-4067-ae91-3af39662c87b-START.bin, endMarker=/home/arseny/ignite-logs/data5/db/ignite_instance_5/cp/1521101654990-5f49b673-0403-4067-ae91-3af39662c87b-END.bin] 2018-03-16 18:15:41.214 INFO 28792 --- [ite-instance-5%] i.i.p.c.p.GridCacheDatabaseSharedManager : Applying lost cache updates since last checkpoint record [lastMarked=FileWALPointer [idx=111, fileOff=143312353, len=13471], lastCheckpointId=5f49b673-0403-4067-ae91-3af39662c87b] 2018-03-16 18:15:41.292 ERROR 28792 --- [ite-instance-5%] .c.d.d.p.GridDhtPartitionsExchangeFuture : Failed to reinitialize local partitions (preloading will be stopped): GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], discoEvt=DiscoveryCustomEvent [customMsg=ChangeGlobalStateMessage [id=889416f2261-904f0bc2-4419-44e0-92f2-346f3a93b5e7, reqId=26f99179-91fe-468d-9c11-6de38f91139c, initiatingNodeId=d62b2da2-771a-45dc-b30c-8a93d21295a1, activate=true, baselineTopology=BaselineTopology [id=0, branchingHash=-1777855712, branchingType='New BaselineTopology', baselineNodes=[ignite-instance-5]], forceChangeBaselineTopology=false, timestamp=1521213334978], affTopVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], super=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=d62b2da2-771a-45dc-b30c-8a93d21295a1, addrs=[0:0:0:0:0:0:0:1%lo, 127.0.0.1, 172.17.0.1, 192.168.0.62], sockAddrs=[/192.168.0.62:47500, /172.17.0.1:47500, /0:0:0:0:0:0:0:1%lo:47500, /127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1521213333681, loc=true, ver=2.4.0#20180305-sha1:aa342270, isClient=false], topVer=1, nodeId8=d62b2da2, msg=null, type=DISCOVERY_CUSTOM_EVT, tstamp=1521213334989]], nodeId=d62b2da2, evt=DISCOVERY_CUSTOM_EVT] org.apache.ignite.IgniteException: Runtime failure on row: Row@43590108[ key: 9698:1521062945644:170611384279854154, val: ru.synesis.kipod.event.KipodEvent [idHash=1211906587, hash=1264784403, face_last_name=null, face_list_id=null, channel=9698, source=, resource_group_0=901, face_similarity=null, license_plate_number=null, descriptors=null, cacheName=kipod_events, cacheKey=9698:1521062945644:170611384279854154, stream=9698, alarm=false, processed_at=0, face_id=null, descriptor_version=null, id=170611384279854154, persistent=false, face_first_name=null, license_plate_first_name=null, face_full_name=null, level=0, module=Kpx.Synesis.Outdoor, end_time=1521063039531, resource_groups=[901, 927, 1148, 1149], params=null, commented_at=0, resource_group_3=1149, start_time=1521062945644, processed=false, resource_group_1=927, kafka_offset=1062558, resource_group_2=1148, license_plate_last_name=null, armed=false, license_plate_country=null, topic=MotionAlarm, comment=, expiration=1521408639000, original_id=null, license_plate_lists=null], ver: GridCacheVersion [topVer=132499588, order=1521102684509, nodeOrder=6] ][ 170611384279854154, MotionAlarm, Kpx.Synesis.Outdoor, 0, , 1521062945644, 1521063039531, 9698, 9698, FALSE, FALSE, , FALSE, FALSE, 0, 0, 1062558, 1521408639000, null, null, null, null, null, null, null, null, null, null, null, null, 901, 927, 1148, 1149 ] at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2102) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putx(BPlusTree.java:2049) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.putx(H2TreeIndex.java:247) ~[ignite-indexing-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.update(GridH2Table.java:454) ~[ignite-indexing-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.store(IgniteH2Indexing.java:649) ~[ignite-indexing-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.query.GridQueryProcessor.store(GridQueryProcessor.java:1866) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.store(GridCacheQueryManager.java:407) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishUpdate(IgniteCacheOffheapManagerImpl.java:1343) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.update(IgniteCacheOffheapManagerImpl.java:1316) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.update(GridCacheOffheapManager.java:1336) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.update(IgniteCacheOffheapManagerImpl.java:360) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.applyUpdate(GridCacheDatabaseSharedManager.java:2401) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.applyLastUpdates(GridCacheDatabaseSharedManager.java:2137) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreState(GridCacheDatabaseSharedManager.java:1341) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.beforeExchange(GridCacheDatabaseSharedManager.java:1129) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:1066) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:664) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2337) [ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) [ignite-core-2.4.0.jar:2.4.0] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151] Caused by: java.lang.IllegalStateException: Failed to get page IO instance (page content is corrupted) at org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forVersion(IOVersions.java:83) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forPage(IOVersions.java:95) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:148) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.query.h2.database.H2RowFactory.getRow(H2RowFactory.java:61) ~[ignite-indexing-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.query.h2.database.H2Tree.createRowFromLink(H2Tree.java:149) ~[ignite-indexing-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.query.h2.database.io.H2ExtrasInnerIO.getLookupRow(H2ExtrasInnerIO.java:114) ~[ignite-indexing-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.query.h2.database.io.H2ExtrasInnerIO.getLookupRow(H2ExtrasInnerIO.java:36) ~[ignite-indexing-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:167) ~[ignite-indexing-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:46) ~[ignite-indexing-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.getRow(BPlusTree.java:4436) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:244) ~[ignite-indexing-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:46) ~[ignite-indexing-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.compare(BPlusTree.java:4423) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findInsertionPoint(BPlusTree.java:4343) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$1500(BPlusTree.java:82) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:270) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4770) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4755) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:158) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:320) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2317) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2329) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2069) ~[ignite-core-2.4.0.jar:2.4.0] ... 19 common frames omitted 2018-03-16 18:15:41.292 INFO 28792 --- [ite-instance-5%] .c.d.d.p.GridDhtPartitionsExchangeFuture : Finish exchange future [startVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], resVer=null, err=class org.apache.ignite.IgniteException: Runtime failure on row: Row@43590108[ key: 9698:1521062945644:170611384279854154, val: ru.synesis.kipod.event.KipodEvent [idHash=1211906587, hash=1264784403, face_last_name=null, face_list_id=null, channel=9698, source=, resource_group_0=901, face_similarity=null, license_plate_number=null, descriptors=null, cacheName=kipod_events, cacheKey=9698:1521062945644:170611384279854154, stream=9698, alarm=false, processed_at=0, face_id=null, descriptor_version=null, id=170611384279854154, persistent=false, face_first_name=null, license_plate_first_name=null, face_full_name=null, level=0, module=Kpx.Synesis.Outdoor, end_time=1521063039531, resource_groups=[901, 927, 1148, 1149], params=null, commented_at=0, resource_group_3=1149, start_time=1521062945644, processed=false, resource_group_1=927, kafka_offset=1062558, resource_group_2=1148, license_plate_last_name=null, armed=false, license_plate_country=null, topic=MotionAlarm, comment=, expiration=1521408639000, original_id=null, license_plate_lists=null], ver: GridCacheVersion [topVer=132499588, order=1521102684509, nodeOrder=6] ][ 170611384279854154, MotionAlarm, Kpx.Synesis.Outdoor, 0, , 1521062945644, 1521063039531, 9698, 9698, FALSE, FALSE, , FALSE, FALSE, 0, 0, 1062558, 1521408639000, null, null, null, null, null, null, null, null, null, null, null, null, 901, 927, 1148, 1149 ]] 2018-03-16 18:15:41.295 INFO 28792 --- [ite-instance-5%] i.i.p.c.p.GridCacheDatabaseSharedManager : Finished indexes rebuilding for cache [name=ignite-sys-cache, grpName=null] 2018-03-16 18:15:41.298 ERROR 28792 --- [ite-instance-5%] .i.p.c.GridCachePartitionExchangeManager : Failed to wait for completion of partition map exchange (preloading will not start): GridDhtPartitionsExchangeFuture [firstDiscoEvt=DiscoveryCustomEvent [customMsg=null, affTopVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], super=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=d62b2da2-771a-45dc-b30c-8a93d21295a1, addrs=[0:0:0:0:0:0:0:1%lo, 127.0.0.1, 172.17.0.1, 192.168.0.62], sockAddrs=[/192.168.0.62:47500, /172.17.0.1:47500, /0:0:0:0:0:0:0:1%lo:47500, /127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1521213333681, loc=true, ver=2.4.0#20180305-sha1:aa342270, isClient=false], topVer=1, nodeId8=d62b2da2, msg=null, type=DISCOVERY_CUSTOM_EVT, tstamp=1521213334989]], crd=TcpDiscoveryNode [id=d62b2da2-771a-45dc-b30c-8a93d21295a1, addrs=[0:0:0:0:0:0:0:1%lo, 127.0.0.1, 172.17.0.1, 192.168.0.62], sockAddrs=[/192.168.0.62:47500, /172.17.0.1:47500, /0:0:0:0:0:0:0:1%lo:47500, /127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1521213333681, loc=true, ver=2.4.0#20180305-sha1:aa342270, isClient=false], exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], discoEvt=DiscoveryCustomEvent [customMsg=null, affTopVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], super=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=d62b2da2-771a-45dc-b30c-8a93d21295a1, addrs=[0:0:0:0:0:0:0:1%lo, 127.0.0.1, 172.17.0.1, 192.168.0.62], sockAddrs=[/192.168.0.62:47500, /172.17.0.1:47500, /0:0:0:0:0:0:0:1%lo:47500, /127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1521213333681, loc=true, ver=2.4.0#20180305-sha1:aa342270, isClient=false], topVer=1, nodeId8=d62b2da2, msg=null, type=DISCOVERY_CUSTOM_EVT, tstamp=1521213334989]], nodeId=d62b2da2, evt=DISCOVERY_CUSTOM_EVT], added=true, initFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=false, hash=1711674579], init=false, lastVer=null, partReleaseFut=PartitionReleaseFuture [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], futures=[ExplicitLockReleaseFuture [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], futures=[]], TxReleaseFuture [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], futures=[]], AtomicUpdateReleaseFuture [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], futures=[]], DataStreamerReleaseFuture [topVer=AffinityTopologyVersion [topVer=1, minorTopVer=1], futures=[]]]], exchActions=null, affChangeMsg=null, initTs=1521213334989, centralizedAff=false, changeGlobalStateE=null, done=true, state=CRD, evtLatch=0, remaining=[], super=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=class o.a.i.IgniteException: Runtime failure on row: Row@43590108[ key: 9698:1521062945644:170611384279854154, val: ru.synesis.kipod.event.KipodEvent [idHash=1211906587, hash=1264784403, face_last_name=null, face_list_id=null, channel=9698, source=, resource_group_0=901, face_similarity=null, license_plate_number=null, descriptors=null, cacheName=kipod_events, cacheKey=9698:1521062945644:170611384279854154, stream=9698, alarm=false, processed_at=0, face_id=null, descriptor_version=null, id=170611384279854154, persistent=false, face_first_name=null, license_plate_first_name=null, face_full_name=null, level=0, module=Kpx.Synesis.Outdoor, end_time=1521063039531, resource_groups=[901, 927, 1148, 1149], params=null, commented_at=0, resource_group_3=1149, start_time=1521062945644, processed=false, resource_group_1=927, kafka_offset=1062558, resource_group_2=1148, license_plate_last_name=null, armed=false, license_plate_country=null, topic=MotionAlarm, comment=, expiration=1521408639000, original_id=null, license_plate_lists=null], ver: GridCacheVersion [topVer=132499588, order=1521102684509, nodeOrder=6] ][ 170611384279854154, MotionAlarm, Kpx.Synesis.Outdoor, 0, , 1521062945644, 1521063039531, 9698, 9698, FALSE, FALSE, , FALSE, FALSE, 0, 0, 1062558, 1521408639000, null, null, null, null, null, null, null, null, null, null, null, null, 901, 927, 1148, 1149 ], hash=767372141]] org.apache.ignite.IgniteCheckedException: Failed to get page IO instance (page content is corrupted) at org.apache.ignite.internal.util.IgniteUtils.cast(IgniteUtils.java:7244) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.util.future.GridFutureAdapter.resolve(GridFutureAdapter.java:259) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:207) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:159) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2347) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) [ignite-core-2.4.0.jar:2.4.0] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151] Caused by: java.lang.IllegalStateException: Failed to get page IO instance (page content is corrupted) at org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forVersion(IOVersions.java:83) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forPage(IOVersions.java:95) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:148) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.persistence.CacheDataRowAdapter.initFromLink(CacheDataRowAdapter.java:102) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.query.h2.database.H2RowFactory.getRow(H2RowFactory.java:61) ~[ignite-indexing-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.query.h2.database.H2Tree.createRowFromLink(H2Tree.java:149) ~[ignite-indexing-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.query.h2.database.io.H2ExtrasInnerIO.getLookupRow(H2ExtrasInnerIO.java:114) ~[ignite-indexing-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.query.h2.database.io.H2ExtrasInnerIO.getLookupRow(H2ExtrasInnerIO.java:36) ~[ignite-indexing-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:167) ~[ignite-indexing-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.query.h2.database.H2Tree.getRow(H2Tree.java:46) ~[ignite-indexing-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.getRow(BPlusTree.java:4436) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:244) ~[ignite-indexing-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.query.h2.database.H2Tree.compare(H2Tree.java:46) ~[ignite-indexing-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.compare(BPlusTree.java:4423) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.findInsertionPoint(BPlusTree.java:4343) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.access$1500(BPlusTree.java:82) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$Search.run0(BPlusTree.java:270) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4770) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree$GetPageHandler.run(BPlusTree.java:4755) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.persistence.tree.util.PageHandler.readPage(PageHandler.java:158) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.persistence.DataStructure.read(DataStructure.java:320) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2317) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putDown(BPlusTree.java:2329) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.doPut(BPlusTree.java:2069) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.persistence.tree.BPlusTree.putx(BPlusTree.java:2049) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.putx(H2TreeIndex.java:247) ~[ignite-indexing-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.update(GridH2Table.java:454) ~[ignite-indexing-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.store(IgniteH2Indexing.java:649) ~[ignite-indexing-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.query.GridQueryProcessor.store(GridQueryProcessor.java:1866) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.store(GridCacheQueryManager.java:407) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishUpdate(IgniteCacheOffheapManagerImpl.java:1343) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.update(IgniteCacheOffheapManagerImpl.java:1316) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.update(GridCacheOffheapManager.java:1336) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.update(IgniteCacheOffheapManagerImpl.java:360) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.applyUpdate(GridCacheDatabaseSharedManager.java:2401) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.applyLastUpdates(GridCacheDatabaseSharedManager.java:2137) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreState(GridCacheDatabaseSharedManager.java:1341) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.beforeExchange(GridCacheDatabaseSharedManager.java:1129) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:1066) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:664) ~[ignite-core-2.4.0.jar:2.4.0] at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:2337) ~[ignite-core-2.4.0.jar:2.4.0] ... 2 common frames omitted 2018-03-16 18:15:44.923 INFO 28792 --- [-notifier-timer] o.a.i.i.p.cluster.GridUpdateNotifier : Your version is up to date.
