[
https://issues.apache.org/jira/browse/IGNITE-6506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Goncharuk reassigned IGNITE-6506:
----------------------------------------
Assignee: Alexey Goncharuk
> Cluster activation hangs if a node was stopped during persistent storage
> checkpoint
> -----------------------------------------------------------------------------------
>
> Key: IGNITE-6506
> URL: https://issues.apache.org/jira/browse/IGNITE-6506
> Project: Ignite
> Issue Type: Bug
> Components: persistence
> Affects Versions: 2.2
> Reporter: Joel Lang
> Assignee: Alexey Goncharuk
> Priority: Critical
> Labels: important
> Fix For: 2.3
>
>
> I have a cluster with two nodes: A and B.
> On startup, node A and B wait for each other to be connected and then node A
> will attempt to activate the cluster.
> While testing high availability we find that if a node is stopped during the
> persistent store checkpoint, we cannot activate the cluster on startup
> without deleting the persistent storage directory. Specifically in the case
> where node A is stopped during checkpointing, upon the next startup it will
> encounter several exceptions during activation and then hang without
> completing activation.
> Here is the log.
> {noformat}
> 2017-09-26 12:11:24 [tcp-disco-msg-worker-#2%mbe%] INFO
> o.a.i.i.p.c.GridClusterStateProcessor - Start state transition: true
> 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO
> o.a.ignite.internal.exchange.time - Started exchange init
> [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1], crd=true, evt=18,
> node=TcpDiscoveryNode [id=62cf0ccb-e376-4b80-8d2d-98115c3a2990,
> addrs=[10.5.17.19, 127.0.0.1],
> sockAddrs=[shouvdevmbe02.petrolink.net/10.5.17.19:47510, /127.0.0.1:47510],
> discPort=47510, order=1, intOrder=1, lastExchangeTime=1506445884063,
> loc=true, ver=2.2.0#20170915-sha1:5747ce6b, isClient=false],
> evtNode=TcpDiscoveryNode [id=62cf0ccb-e376-4b80-8d2d-98115c3a2990,
> addrs=[10.5.17.19, 127.0.0.1],
> sockAddrs=[shouvdevmbe02.petrolink.net/10.5.17.19:47510, /127.0.0.1:47510],
> discPort=47510, order=1, intOrder=1, lastExchangeTime=1506445884063,
> loc=true, ver=2.2.0#20170915-sha1:5747ce6b, isClient=false],
> customEvt=ChangeGlobalStateMessage
> [id=1d0cb2fbe51-7967bd11-40aa-40fe-b0a6-c43302cd4ee7,
> reqId=f7155dea-fede-4340-b244-7a3b65f167a8,
> initiatingNodeId=62cf0ccb-e376-4b80-8d2d-98115c3a2990, activate=true]]
> 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO
> o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture - Start activation process
> [nodeId=62cf0ccb-e376-4b80-8d2d-98115c3a2990, client=false,
> topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1]]
> 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO
> o.a.i.i.p.c.p.f.FilePageStoreManager - Resolved page store work directory:
> /opt/mbe1/ignite/db/mbe_MBE1
> 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO
> o.a.i.i.p.c.p.w.FileWriteAheadLogManager - Resolved write ahead log work
> directory: /opt/mbe1/ignite/db/wal/mbe_MBE1
> 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO
> o.a.i.i.p.c.p.w.FileWriteAheadLogManager - Resolved write ahead log archive
> directory: /opt/mbe1/ignite/db/wal/archive/mbe_MBE1
> 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] WARN
> o.a.i.i.p.c.p.GridCacheDatabaseSharedManager - No user-defined default
> MemoryPolicy found; system default of 1GB size will be used.
> 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO
> o.a.i.i.p.c.p.pagemem.PageMemoryImpl - Started page memory
> [memoryAllocated=100.0 MiB, pages=48592, tableSize=2.9 MiB,
> checkpointBuffer=819.4 MiB]
> 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO
> o.a.i.i.p.c.p.pagemem.PageMemoryImpl - Started page memory
> [memoryAllocated=3.1 GiB, pages=1544064, tableSize=91.0 MiB,
> checkpointBuffer=819.4 MiB]
> 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO
> o.a.i.i.p.c.p.GridCacheDatabaseSharedManager - Read checkpoint status: start
> marker =
> /opt/mbe1/ignite/db/mbe_MBE1/cp/1506444061104-38b80aaa-8c3d-4572-a42e-5b7a3b472505-START.bin,
> end marker =
> /opt/mbe1/ignite/db/mbe_MBE1/cp/1506442980839-ff65a0dc-3d83-436a-8329-7b3a31fe5ffc-END.bin
> 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO
> o.a.i.i.p.c.p.GridCacheDatabaseSharedManager - Checking memory state
> [lastValidPos=FileWALPointer [idx=139, fileOffset=31406805, len=20731,
> forceFlush=false], lastMarked=FileWALPointer [idx=0, fileOffset=0, len=0,
> forceFlush=false], lastCheckpointId=38b80aaa-8c3d-4572-a42e-5b7a3b472505]
> 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] WARN
> o.a.i.i.p.c.p.GridCacheDatabaseSharedManager - Ignite node stopped in the
> middle of checkpoint. Will restore memory state and finish checkpoint on node
> start.
> 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] ERROR
> o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture - Failed to activate node
> components [nodeId=62cf0ccb-e376-4b80-8d2d-98115c3a2990, client=false,
> topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1]]
> java.lang.ArrayIndexOutOfBoundsException: -1
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forVersion(IOVersions.java:82)
> at
> org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forPage(IOVersions.java:92)
> at
> org.apache.ignite.internal.pagemem.wal.record.delta.DataPageInsertRecord.applyDelta(DataPageInsertRecord.java:57)
> at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreMemory(GridCacheDatabaseSharedManager.java:1415)
> at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.readCheckpointAndRestoreMemory(GridCacheDatabaseSharedManager.java:539)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.onClusterStateChangeRequest(GridDhtPartitionsExchangeFuture.java:770)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:574)
> at
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1901)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
> at java.lang.Thread.run(Thread.java:745)
> 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO
> o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture - Finished waiting for
> partition release future [topVer=AffinityTopologyVersion [topVer=2,
> minorTopVer=1], waitTime=0ms]
> 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO
> o.a.i.i.p.c.p.GridCacheDatabaseSharedManager - Read checkpoint status: start
> marker =
> /opt/mbe1/ignite/db/mbe_MBE1/cp/1506444061104-38b80aaa-8c3d-4572-a42e-5b7a3b472505-START.bin,
> end marker =
> /opt/mbe1/ignite/db/mbe_MBE1/cp/1506442980839-ff65a0dc-3d83-436a-8329-7b3a31fe5ffc-END.bin
> 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO
> o.a.i.i.p.c.p.GridCacheDatabaseSharedManager - Applying lost cache updates
> since last checkpoint record [lastMarked=FileWALPointer [idx=0, fileOffset=0,
> len=0, forceFlush=false],
> lastCheckpointId=38b80aaa-8c3d-4572-a42e-5b7a3b472505]
> 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] ERROR
> o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture - Failed to reinitialize
> local partitions (preloading will be stopped): GridDhtPartitionExchangeId
> [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1], nodeId=62cf0ccb,
> evt=DISCOVERY_CUSTOM_EVT]
> org.apache.ignite.IgniteCheckedException: WAL history is too short
> [descs=[org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@5f,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@60,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@61,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@62,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@63,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@64,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@65,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@66,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@67,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@68,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@69,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6a,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6b,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6c,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6d,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6e,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6f,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@70,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@71,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@72,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@73,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@74,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@75,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@76,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@77,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@78,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@79,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7a,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7b,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7c,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7d,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7e,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7f,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@80,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@81,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@82,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@83,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@84,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@85,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@86,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@87,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@88,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@89,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@8a,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@8b],
> start=FileWALPointer [idx=0, fileOffset=0, len=0, forceFlush=false]]
> at
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.init(FileWriteAheadLogManager.java:2375)
> at
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.<init>(FileWriteAheadLogManager.java:2350)
> at
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.<init>(FileWriteAheadLogManager.java:2290)
> at
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.replay(FileWriteAheadLogManager.java:553)
> at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.applyLastUpdates(GridCacheDatabaseSharedManager.java:1484)
> at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreState(GridCacheDatabaseSharedManager.java:938)
> at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.beforeExchange(GridCacheDatabaseSharedManager.java:742)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:994)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:632)
> at
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1901)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
> at java.lang.Thread.run(Thread.java:745)
> 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] INFO
> o.a.i.i.p.c.d.d.p.GridDhtPartitionsExchangeFuture - Snapshot initialization
> completed [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=1], time=0ms]
> 2017-09-26 12:11:24 [exchange-worker-#34%mbe%] ERROR
> o.a.i.i.p.c.GridCachePartitionExchangeManager - Failed to wait for completion
> of partition map exchange (preloading will not start):
> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false,
> reassign=false, discoEvt=DiscoveryCustomEvent [customMsg=null,
> affTopVer=AffinityTopologyVersion [topVer=2, minorTopVer=1],
> super=DiscoveryEvent [evtNode=TcpDiscoveryNode
> [id=62cf0ccb-e376-4b80-8d2d-98115c3a2990, addrs=[10.5.17.19, 127.0.0.1],
> sockAddrs=[shouvdevmbe02.petrolink.net/10.5.17.19:47510, /127.0.0.1:47510],
> discPort=47510, order=1, intOrder=1, lastExchangeTime=1506445884308,
> loc=true, ver=2.2.0#20170915-sha1:5747ce6b, isClient=false], topVer=2,
> nodeId8=62cf0ccb, msg=null, type=DISCOVERY_CUSTOM_EVT,
> tstamp=1506445884063]], crd=TcpDiscoveryNode
> [id=62cf0ccb-e376-4b80-8d2d-98115c3a2990, addrs=[10.5.17.19, 127.0.0.1],
> sockAddrs=[shouvdevmbe02.petrolink.net/10.5.17.19:47510, /127.0.0.1:47510],
> discPort=47510, order=1, intOrder=1, lastExchangeTime=1506445884308,
> loc=true, ver=2.2.0#20170915-sha1:5747ce6b, isClient=false],
> exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=2,
> minorTopVer=1], nodeId=62cf0ccb, evt=DISCOVERY_CUSTOM_EVT], added=true,
> initFut=GridFutureAdapter [ignoreInterrupts=false, state=DONE, res=false,
> hash=1495762349], init=false, lastVer=null, partReleaseFut=GridCompoundFuture
> [rdc=null, initFlag=1, lsnrCalls=4, done=true, cancelled=false, err=null,
> futs=[true, true, true, true]], exchActions=null, affChangeMsg=null,
> skipPreload=false, clientOnlyExchange=false, initTs=1506445884073,
> centralizedAff=false,
> changeGlobalStateE=java.lang.ArrayIndexOutOfBoundsException: -1,
> forcedRebFut=null, done=true, evtLatch=0,
> remaining=[bd81ae2b-91c0-45f1-b589-e73b7ffb56ee], super=GridFutureAdapter
> [ignoreInterrupts=false, state=DONE, res=class o.a.i.IgniteCheckedException:
> WAL history is too short
> [descs=[o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@5f,
>
> o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@60,
>
> o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@61,
>
> o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@62,
>
> o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@63,
>
> o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@64,
>
> o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@65,
>
> o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@66,
>
> o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@67,
>
> o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@68,
>
> o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@69,
>
> o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6a,
>
> o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6b,
>
> o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6c,
>
> o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6d,
>
> o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6e,
>
> o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6f,
>
> o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@70,
>
> o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@71,
>
> o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@72,
>
> o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@73,
>
> o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@74,
>
> o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@75,
>
> o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@76,
>
> o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@77,
>
> o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@78,
>
> o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@79,
>
> o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7a,
>
> o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7b,
>
> o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7c,
>
> o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7d,
>
> o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7e,
>
> o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7f,
>
> o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@80,
>
> o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@81,
>
> o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@82,
>
> o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@83,
>
> o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@84,
>
> o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@85,
>
> o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@86,
>
> o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@87,
>
> o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@88,
>
> o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@89,
>
> o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@8a,
>
> o.a.i.i.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@8b],
> start=FileWALPointer [idx=0, fileOffset=0, len=0, forceFlush=false]],
> hash=927462353]]
> org.apache.ignite.IgniteCheckedException: WAL history is too short
> [descs=[org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@5f,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@60,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@61,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@62,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@63,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@64,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@65,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@66,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@67,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@68,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@69,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6a,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6b,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6c,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6d,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6e,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@6f,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@70,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@71,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@72,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@73,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@74,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@75,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@76,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@77,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@78,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@79,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7a,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7b,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7c,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7d,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7e,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@7f,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@80,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@81,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@82,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@83,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@84,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@85,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@86,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@87,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@88,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@89,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@8a,
>
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@8b],
> start=FileWALPointer [idx=0, fileOffset=0, len=0, forceFlush=false]]
> at
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.init(FileWriteAheadLogManager.java:2375)
> at
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.<init>(FileWriteAheadLogManager.java:2350)
> at
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.<init>(FileWriteAheadLogManager.java:2290)
> at
> org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.replay(FileWriteAheadLogManager.java:553)
> at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.applyLastUpdates(GridCacheDatabaseSharedManager.java:1484)
> at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreState(GridCacheDatabaseSharedManager.java:938)
> at
> org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.beforeExchange(GridCacheDatabaseSharedManager.java:742)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:994)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:632)
> at
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1901)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
> The thread activating the node is blocked at this point and the node cannot
> continue startup.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
