Ksenia Rybakova created IGNITE-3211:
---------------------------------------
Summary: "Failed to reinitialize local partitions", "Failed to
wait for completion of partition map exchange" errors during failover test
Key: IGNITE-3211
URL: https://issues.apache.org/jira/browse/IGNITE-3211
Project: Ignite
Issue Type: Bug
Affects Versions: 1.6
Reporter: Ksenia Rybakova
"Failed to reinitialize local partitions (preloading will be stopped)" and
"Failed to wait for completion of partition map exchange (preloading will not
start)" errors occured during failover load test. Complete stack trace see
below.
Load config:
- 1 client, 20 servers (5 servers per 1 host)
- warmup 60
- duration 66h
- preload 5M
- key range 10M
- operations: PUT PUT_ALL GET GET_ALL INVOKE INVOKE_ALL REMOVE REMOVE_ALL
PUT_IF_ABSENT REPLACE
- backups count 3
- 3 servers restart every 15 min with 30 sec step, pause between stop and start
5min
{noformat}
[08:32:21,002][ERROR][exchange-worker-#83%null%][GridDhtPartitionsExchangeFuture]
Failed to reinitialize local partitions (preloading will be stopped):
GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=39,
minorTopVer=1], nodeId=20ddc8b7, evt=DISCOVERY_CUSTOM_EVT]
class org.apache.ignite.IgniteException: null
at
org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:506)
at
org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:297)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter.toString(IgniteTxLocalAdapter.java:3743)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocalAdapter.toString(GridDhtTxLocalAdapter.java:868)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.toString(GridDhtTxLocal.java:703)
at java.lang.String.valueOf(String.java:2849)
at java.lang.StringBuilder.append(StringBuilder.java:128)
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.dumpPendingObjects(GridCachePartitionExchangeManager.java:1172)
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.dumpDebugInfo(GridCachePartitionExchangeManager.java:1150)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.dumpPendingObjects(GridDhtPartitionsExchangeFuture.java:894)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.waitPartitionRelease(GridDhtPartitionsExchangeFuture.java:769)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:715)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:472)
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1333)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:745)
Caused by: class org.apache.ignite.IgniteException: null
at
org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:506)
at
org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:364)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxStateImpl.toString(IgniteTxStateImpl.java:443)
at java.lang.String.valueOf(String.java:2849)
at
org.apache.ignite.internal.util.GridStringBuilder.a(GridStringBuilder.java:101)
at
org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:474)
... 15 more
Caused by: java.util.ConcurrentModificationException
at
java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:394)
at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:413)
at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:412)
at java.util.AbstractMap.toString(AbstractMap.java:518)
at java.lang.String.valueOf(String.java:2849)
at
org.apache.ignite.internal.util.GridStringBuilder.a(GridStringBuilder.java:101)
at
org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:474)
... 20 more
[08:32:21,072][ERROR][exchange-worker-#83%null%][GridCachePartitionExchangeManager]
Failed to wait for completion of partition map exchange (preloading will not
start): GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false,
reassign=false, discoEvt=DiscoveryCustomEvent [customMsg=null,
affTopVer=AffinityTopologyVersion [topVer=39, minorTopVer=1],
super=DiscoveryEvent [evtNode=TcpDiscoveryNode
[id=20ddc8b7-fc62-4d8c-be98-2e5edf60a419, addrs=[10.20.0.216, 127.0.0.1],
sockAddrs=[fosters-216/10.20.0.216:47500, /10.20.0.216:47500,
/127.0.0.1:47500], discPort=47500, order=1, intOrder=1,
lastExchangeTime=1464363141003, loc=true, ver=1.6.0#20160525-sha1:48321a40,
isClient=false], topVer=39, nodeId8=20ddc8b7, msg=null,
type=DISCOVERY_CUSTOM_EVT, tstamp=1464363130370]], crd=TcpDiscoveryNode
[id=20ddc8b7-fc62-4d8c-be98-2e5edf60a419, addrs=[10.20.0.216, 127.0.0.1],
sockAddrs=[fosters-216/10.20.0.216:47500, /10.20.0.216:47500,
/127.0.0.1:47500], discPort=47500, order=1, intOrder=1,
lastExchangeTime=1464363141003, loc=true, ver=1.6.0#20160525-sha1:48321a40,
isClient=false], exchId=GridDhtPartitionExchangeId
[topVer=AffinityTopologyVersion [topVer=39, minorTopVer=1], nodeId=20ddc8b7,
evt=DISCOVERY_CUSTOM_EVT], added=true, initFut=GridFutureAdapter [resFlag=2,
res=false, startTime=1464363130370, endTime=1464363141057,
ignoreInterrupts=false, state=DONE], init=false, topSnapshot=null,
lastVer=GridCacheVersion [topVer=75840698, time=1464363140466,
order=1464462577173, nodeOrder=38], partReleaseFut=GridCompoundFuture
[rdc=null, initFlag=1, lsnrCalls=2, done=false, cancelled=false, err=null,
futs=[true, false, true]], affChangeMsg=CacheAffinityChangeMessage
[id=9560ad2f451-621c2001-9d60-40f4-ab21-20834a2b6b33,
topVer=AffinityTopologyVersion [topVer=39, minorTopVer=0], exchId=null,
partsMsg=null, exchangeNeeded=true], skipPreload=false,
clientOnlyExchange=false, initTs=1464363130370, centralizedAff=false,
evtLatch=0, remaining=[e7f032f5-4ec6-48c8-9cd4-ac78b0f8ccc4,
a550929c-1f4b-4de3-8d57-47b4a8232010, c7ff42e7-acd5-4cb4-9cd5-179aa18b88b0,
fc79bff1-480f-44b3-9915-2548f937726c, c013837c-b46e-4667-9e47-e3c41412b5a1,
ba89970d-8908-4276-9d3c-b679a335c2d2, a7f0feb2-b932-4843-ac93-560fc547700e,
3c462bba-1d12-43ab-ad21-761f9ef94aa1, 9c184df6-e5e0-49b3-9836-65932d58be6a,
317b61fc-186b-4bf7-9622-6c4094409814, 425b461c-5117-403d-af2b-e323b7c1aa39,
a4d0911c-d54a-4a36-b911-b74eddc3f0cc, d0e84363-590d-45bf-bad3-b88cde5d03fe,
b51cc458-c884-41ce-9e5b-b83ea111218b, 6970f13f-fe82-4583-a128-af93f86e99e7,
a9b8fe06-cda9-4907-871d-0684bcc2c3ca, c978e55f-4de6-4ffd-98b1-459215db2232,
cab5d0e0-7365-4774-8f99-d9f131c5d896, 0859feda-b70d-4fab-870a-4db18bd5984b],
srvNodes=[TcpDiscoveryNode [id=20ddc8b7-fc62-4d8c-be98-2e5edf60a419,
addrs=[10.20.0.216, 127.0.0.1], sockAddrs=[fosters-216/10.20.0.216:47500,
/10.20.0.216:47500, /127.0.0.1:47500], discPort=47500, order=1, intOrder=1,
lastExchangeTime=1464363141003, loc=true, ver=1.6.0#20160525-sha1:48321a40,
isClient=false], TcpDiscoveryNode [id=cab5d0e0-7365-4774-8f99-d9f131c5d896,
addrs=[10.20.0.221, 127.0.0.1], sockAddrs=[fosters-221/10.20.0.221:47500,
/10.20.0.221:47500, /127.0.0.1:47500], discPort=47500, order=2, intOrder=2,
lastExchangeTime=1464360659615, loc=false, ver=1.6.0#20160525-sha1:48321a40,
isClient=false], TcpDiscoveryNode [id=c7ff42e7-acd5-4cb4-9cd5-179aa18b88b0,
addrs=[10.20.0.222, 127.0.0.1], sockAddrs=[fosters-222/10.20.0.222:47500,
/10.20.0.222:47500, /127.0.0.1:47500], discPort=47500, order=3, intOrder=3,
lastExchangeTime=1464360660162, loc=false, ver=1.6.0#20160525-sha1:48321a40,
isClient=false], TcpDiscoveryNode [id=0859feda-b70d-4fab-870a-4db18bd5984b,
addrs=[10.20.0.223, 127.0.0.1], sockAddrs=[fosters-223/10.20.0.223:47500,
/10.20.0.223:47500, /127.0.0.1:47500], discPort=47500, order=4, intOrder=4,
lastExchangeTime=1464360660689, loc=false, ver=1.6.0#20160525-sha1:48321a40,
isClient=false], TcpDiscoveryNode [id=ba89970d-8908-4276-9d3c-b679a335c2d2,
addrs=[10.20.0.216, 127.0.0.1], sockAddrs=[fosters-216/10.20.0.216:47501,
/10.20.0.216:47501, /127.0.0.1:47501], discPort=47501, order=5, intOrder=5,
lastExchangeTime=1464360661276, loc=false, ver=1.6.0#20160525-sha1:48321a40,
isClient=false], TcpDiscoveryNode [id=317b61fc-186b-4bf7-9622-6c4094409814,
addrs=[10.20.0.221, 127.0.0.1], sockAddrs=[fosters-221/10.20.0.221:47501,
/10.20.0.221:47501, /127.0.0.1:47501], discPort=47501, order=6, intOrder=6,
lastExchangeTime=1464360662086, loc=false, ver=1.6.0#20160525-sha1:48321a40,
isClient=false], TcpDiscoveryNode [id=425b461c-5117-403d-af2b-e323b7c1aa39,
addrs=[10.20.0.222, 127.0.0.1], sockAddrs=[fosters-222/10.20.0.222:47501,
/10.20.0.222:47501, /127.0.0.1:47501], discPort=47501, order=7, intOrder=7,
lastExchangeTime=1464360662471, loc=false, ver=1.6.0#20160525-sha1:48321a40,
isClient=false], TcpDiscoveryNode [id=a550929c-1f4b-4de3-8d57-47b4a8232010,
addrs=[10.20.0.223, 127.0.0.1], sockAddrs=[fosters-223/10.20.0.223:47501,
/10.20.0.223:47501, /127.0.0.1:47501], discPort=47501, order=8, intOrder=8,
lastExchangeTime=1464360662886, loc=false, ver=1.6.0#20160525-sha1:48321a40,
isClient=false], TcpDiscoveryNode [id=e7f032f5-4ec6-48c8-9cd4-ac78b0f8ccc4,
addrs=[10.20.0.216, 127.0.0.1], sockAddrs=[fosters-216/10.20.0.216:47502,
/10.20.0.216:47502, /127.0.0.1:47502], discPort=47502, order=9, intOrder=9,
lastExchangeTime=1464360663373, loc=false, ver=1.6.0#20160525-sha1:48321a40,
isClient=false], TcpDiscoveryNode [id=b51cc458-c884-41ce-9e5b-b83ea111218b,
addrs=[10.20.0.221, 127.0.0.1], sockAddrs=[fosters-221/10.20.0.221:47502,
/10.20.0.221:47502, /127.0.0.1:47502], discPort=47502, order=10, intOrder=10,
lastExchangeTime=1464360664058, loc=false, ver=1.6.0#20160525-sha1:48321a40,
isClient=false], TcpDiscoveryNode [id=fc79bff1-480f-44b3-9915-2548f937726c,
addrs=[10.20.0.222, 127.0.0.1], sockAddrs=[fosters-222/10.20.0.222:47502,
/10.20.0.222:47502, /127.0.0.1:47502], discPort=47502, order=11, intOrder=11,
lastExchangeTime=1464360664685, loc=false, ver=1.6.0#20160525-sha1:48321a40,
isClient=false], TcpDiscoveryNode [id=a7f0feb2-b932-4843-ac93-560fc547700e,
addrs=[10.20.0.223, 127.0.0.1], sockAddrs=[fosters-223/10.20.0.223:47502,
/10.20.0.223:47502, /127.0.0.1:47502], discPort=47502, order=12, intOrder=12,
lastExchangeTime=1464360665216, loc=false, ver=1.6.0#20160525-sha1:48321a40,
isClient=false], TcpDiscoveryNode [id=9c184df6-e5e0-49b3-9836-65932d58be6a,
addrs=[10.20.0.216, 127.0.0.1], sockAddrs=[fosters-216/10.20.0.216:47503,
/10.20.0.216:47503, /127.0.0.1:47503], discPort=47503, order=13, intOrder=13,
lastExchangeTime=1464360665661, loc=false, ver=1.6.0#20160525-sha1:48321a40,
isClient=false], TcpDiscoveryNode [id=6970f13f-fe82-4583-a128-af93f86e99e7,
addrs=[10.20.0.216, 127.0.0.1], sockAddrs=[fosters-216/10.20.0.216:47504,
/10.20.0.216:47504, /127.0.0.1:47504], discPort=47504, order=17, intOrder=17,
lastExchangeTime=1464360668158, loc=false, ver=1.6.0#20160525-sha1:48321a40,
isClient=false], TcpDiscoveryNode [id=d0e84363-590d-45bf-bad3-b88cde5d03fe,
addrs=[10.20.0.221, 127.0.0.1], sockAddrs=[fosters-221/10.20.0.221:47504,
/10.20.0.221:47504, /127.0.0.1:47504], discPort=47504, order=18, intOrder=18,
lastExchangeTime=1464360668768, loc=false, ver=1.6.0#20160525-sha1:48321a40,
isClient=false], TcpDiscoveryNode [id=c013837c-b46e-4667-9e47-e3c41412b5a1,
addrs=[10.20.0.222, 127.0.0.1], sockAddrs=[fosters-222/10.20.0.222:47504,
/10.20.0.222:47504, /127.0.0.1:47504], discPort=47504, order=19, intOrder=19,
lastExchangeTime=1464360669745, loc=false, ver=1.6.0#20160525-sha1:48321a40,
isClient=false], TcpDiscoveryNode [id=3c462bba-1d12-43ab-ad21-761f9ef94aa1,
addrs=[10.20.0.223, 127.0.0.1], sockAddrs=[fosters-223/10.20.0.223:47504,
/10.20.0.223:47504, /127.0.0.1:47504], discPort=47504, order=20, intOrder=20,
lastExchangeTime=1464360669836, loc=false, ver=1.6.0#20160525-sha1:48321a40,
isClient=false], TcpDiscoveryNode [id=c978e55f-4de6-4ffd-98b1-459215db2232,
addrs=[10.20.0.221, 127.0.0.1], sockAddrs=[fosters-221/10.20.0.221:47503,
/10.20.0.221:47503, /127.0.0.1:47503], discPort=47503, order=37, intOrder=28,
lastExchangeTime=1464362949633, loc=false, ver=1.6.0#20160525-sha1:48321a40,
isClient=false], TcpDiscoveryNode [id=a9b8fe06-cda9-4907-871d-0684bcc2c3ca,
addrs=[10.20.0.222, 127.0.0.1], sockAddrs=[fosters-222/10.20.0.222:47503,
/10.20.0.222:47503, /127.0.0.1:47503], discPort=47503, order=38, intOrder=29,
lastExchangeTime=1464362981162, loc=false, ver=1.6.0#20160525-sha1:48321a40,
isClient=false], TcpDiscoveryNode [id=a4d0911c-d54a-4a36-b911-b74eddc3f0cc,
addrs=[10.20.0.223, 127.0.0.1], sockAddrs=[fosters-223/10.20.0.223:47503,
/10.20.0.223:47503, /127.0.0.1:47503], discPort=47503, order=39, intOrder=30,
lastExchangeTime=1464363011016, loc=false, ver=1.6.0#20160525-sha1:48321a40,
isClient=false]], super=GridFutureAdapter [resFlag=1, res=class
o.a.i.IgniteException: null, startTime=1464363130370, endTime=1464363141003,
ignoreInterrupts=false, state=DONE]]
class org.apache.ignite.IgniteCheckedException: null
at
org.apache.ignite.internal.util.IgniteUtils.cast(IgniteUtils.java:7067)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:168)
at
org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:117)
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1335)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.ConcurrentModificationException
at
java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:394)
at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:413)
at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:412)
at java.util.AbstractMap.toString(AbstractMap.java:518)
at java.lang.String.valueOf(String.java:2849)
at
org.apache.ignite.internal.util.GridStringBuilder.a(GridStringBuilder.java:101)
at
org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:474)
at
org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:364)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxStateImpl.toString(IgniteTxStateImpl.java:443)
at java.lang.String.valueOf(String.java:2849)
at
org.apache.ignite.internal.util.GridStringBuilder.a(GridStringBuilder.java:101)
at
org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:474)
at
org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:297)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter.toString(IgniteTxLocalAdapter.java:3743)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocalAdapter.toString(GridDhtTxLocalAdapter.java:868)
at
org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.toString(GridDhtTxLocal.java:703)
at java.lang.String.valueOf(String.java:2849)
at java.lang.StringBuilder.append(StringBuilder.java:128)
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.dumpPendingObjects(GridCachePartitionExchangeManager.java:1172)
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.dumpDebugInfo(GridCachePartitionExchangeManager.java:1150)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.dumpPendingObjects(GridDhtPartitionsExchangeFuture.java:894)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.waitPartitionRelease(GridDhtPartitionsExchangeFuture.java:769)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:715)
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:472)
at
org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1333)
... 2 more
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
