[
https://issues.apache.org/jira/browse/IGNITE-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dmitriy Setrakyan updated IGNITE-3211:
--------------------------------------
Fix Version/s: 1.7
> "Failed to reinitialize local partitions", "Failed to wait for completion of
> partition map exchange" errors during failover test
> --------------------------------------------------------------------------------------------------------------------------------
>
> Key: IGNITE-3211
> URL: https://issues.apache.org/jira/browse/IGNITE-3211
> Project: Ignite
> Issue Type: Bug
> Affects Versions: 1.6
> Reporter: Ksenia Rybakova
> Fix For: 1.7
>
>
> "Failed to reinitialize local partitions (preloading will be stopped)" and
> "Failed to wait for completion of partition map exchange (preloading will not
> start)" errors occured during failover load test. Complete stack trace see
> below.
> Load config:
> - 1 client, 20 servers (5 servers per 1 host)
> - warmup 60
> - duration 66h
> - preload 5M
> - key range 10M
> - operations: PUT PUT_ALL GET GET_ALL INVOKE INVOKE_ALL REMOVE REMOVE_ALL
> PUT_IF_ABSENT REPLACE
> - backups count 3
> - 3 servers restart every 15 min with 30 sec step, pause between stop and
> start 5min
> {noformat}
> [08:32:21,002][ERROR][exchange-worker-#83%null%][GridDhtPartitionsExchangeFuture]
> Failed to reinitialize local partitions (preloading will be stopped):
> GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=39,
> minorTopVer=1], nodeId=20ddc8b7, evt=DISCOVERY_CUSTOM_EVT]
> class org.apache.ignite.IgniteException: null
> at
> org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:506)
> at
> org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:297)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter.toString(IgniteTxLocalAdapter.java:3743)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocalAdapter.toString(GridDhtTxLocalAdapter.java:868)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.toString(GridDhtTxLocal.java:703)
> at java.lang.String.valueOf(String.java:2849)
> at java.lang.StringBuilder.append(StringBuilder.java:128)
> at
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.dumpPendingObjects(GridCachePartitionExchangeManager.java:1172)
> at
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.dumpDebugInfo(GridCachePartitionExchangeManager.java:1150)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.dumpPendingObjects(GridDhtPartitionsExchangeFuture.java:894)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.waitPartitionRelease(GridDhtPartitionsExchangeFuture.java:769)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:715)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:472)
> at
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1333)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: class org.apache.ignite.IgniteException: null
> at
> org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:506)
> at
> org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:364)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxStateImpl.toString(IgniteTxStateImpl.java:443)
> at java.lang.String.valueOf(String.java:2849)
> at
> org.apache.ignite.internal.util.GridStringBuilder.a(GridStringBuilder.java:101)
> at
> org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:474)
> ... 15 more
> Caused by: java.util.ConcurrentModificationException
> at
> java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:394)
> at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:413)
> at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:412)
> at java.util.AbstractMap.toString(AbstractMap.java:518)
> at java.lang.String.valueOf(String.java:2849)
> at
> org.apache.ignite.internal.util.GridStringBuilder.a(GridStringBuilder.java:101)
> at
> org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:474)
> ... 20 more
> [08:32:21,072][ERROR][exchange-worker-#83%null%][GridCachePartitionExchangeManager]
> Failed to wait for completion of partition map exchange (preloading will not
> start): GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false,
> reassign=false, discoEvt=DiscoveryCustomEvent [customMsg=null,
> affTopVer=AffinityTopologyVersion [topVer=39, minorTopVer=1],
> super=DiscoveryEvent [evtNode=TcpDiscoveryNode
> [id=20ddc8b7-fc62-4d8c-be98-2e5edf60a419, addrs=[10.20.0.216, 127.0.0.1],
> sockAddrs=[fosters-216/10.20.0.216:47500, /10.20.0.216:47500,
> /127.0.0.1:47500], discPort=47500, order=1, intOrder=1,
> lastExchangeTime=1464363141003, loc=true, ver=1.6.0#20160525-sha1:48321a40,
> isClient=false], topVer=39, nodeId8=20ddc8b7, msg=null,
> type=DISCOVERY_CUSTOM_EVT, tstamp=1464363130370]], crd=TcpDiscoveryNode
> [id=20ddc8b7-fc62-4d8c-be98-2e5edf60a419, addrs=[10.20.0.216, 127.0.0.1],
> sockAddrs=[fosters-216/10.20.0.216:47500, /10.20.0.216:47500,
> /127.0.0.1:47500], discPort=47500, order=1, intOrder=1,
> lastExchangeTime=1464363141003, loc=true, ver=1.6.0#20160525-sha1:48321a40,
> isClient=false], exchId=GridDhtPartitionExchangeId
> [topVer=AffinityTopologyVersion [topVer=39, minorTopVer=1], nodeId=20ddc8b7,
> evt=DISCOVERY_CUSTOM_EVT], added=true, initFut=GridFutureAdapter [resFlag=2,
> res=false, startTime=1464363130370, endTime=1464363141057,
> ignoreInterrupts=false, state=DONE], init=false, topSnapshot=null,
> lastVer=GridCacheVersion [topVer=75840698, time=1464363140466,
> order=1464462577173, nodeOrder=38], partReleaseFut=GridCompoundFuture
> [rdc=null, initFlag=1, lsnrCalls=2, done=false, cancelled=false, err=null,
> futs=[true, false, true]], affChangeMsg=CacheAffinityChangeMessage
> [id=9560ad2f451-621c2001-9d60-40f4-ab21-20834a2b6b33,
> topVer=AffinityTopologyVersion [topVer=39, minorTopVer=0], exchId=null,
> partsMsg=null, exchangeNeeded=true], skipPreload=false,
> clientOnlyExchange=false, initTs=1464363130370, centralizedAff=false,
> evtLatch=0, remaining=[e7f032f5-4ec6-48c8-9cd4-ac78b0f8ccc4,
> a550929c-1f4b-4de3-8d57-47b4a8232010, c7ff42e7-acd5-4cb4-9cd5-179aa18b88b0,
> fc79bff1-480f-44b3-9915-2548f937726c, c013837c-b46e-4667-9e47-e3c41412b5a1,
> ba89970d-8908-4276-9d3c-b679a335c2d2, a7f0feb2-b932-4843-ac93-560fc547700e,
> 3c462bba-1d12-43ab-ad21-761f9ef94aa1, 9c184df6-e5e0-49b3-9836-65932d58be6a,
> 317b61fc-186b-4bf7-9622-6c4094409814, 425b461c-5117-403d-af2b-e323b7c1aa39,
> a4d0911c-d54a-4a36-b911-b74eddc3f0cc, d0e84363-590d-45bf-bad3-b88cde5d03fe,
> b51cc458-c884-41ce-9e5b-b83ea111218b, 6970f13f-fe82-4583-a128-af93f86e99e7,
> a9b8fe06-cda9-4907-871d-0684bcc2c3ca, c978e55f-4de6-4ffd-98b1-459215db2232,
> cab5d0e0-7365-4774-8f99-d9f131c5d896, 0859feda-b70d-4fab-870a-4db18bd5984b],
> srvNodes=[TcpDiscoveryNode [id=20ddc8b7-fc62-4d8c-be98-2e5edf60a419,
> addrs=[10.20.0.216, 127.0.0.1], sockAddrs=[fosters-216/10.20.0.216:47500,
> /10.20.0.216:47500, /127.0.0.1:47500], discPort=47500, order=1, intOrder=1,
> lastExchangeTime=1464363141003, loc=true, ver=1.6.0#20160525-sha1:48321a40,
> isClient=false], TcpDiscoveryNode [id=cab5d0e0-7365-4774-8f99-d9f131c5d896,
> addrs=[10.20.0.221, 127.0.0.1], sockAddrs=[fosters-221/10.20.0.221:47500,
> /10.20.0.221:47500, /127.0.0.1:47500], discPort=47500, order=2, intOrder=2,
> lastExchangeTime=1464360659615, loc=false, ver=1.6.0#20160525-sha1:48321a40,
> isClient=false], TcpDiscoveryNode [id=c7ff42e7-acd5-4cb4-9cd5-179aa18b88b0,
> addrs=[10.20.0.222, 127.0.0.1], sockAddrs=[fosters-222/10.20.0.222:47500,
> /10.20.0.222:47500, /127.0.0.1:47500], discPort=47500, order=3, intOrder=3,
> lastExchangeTime=1464360660162, loc=false, ver=1.6.0#20160525-sha1:48321a40,
> isClient=false], TcpDiscoveryNode [id=0859feda-b70d-4fab-870a-4db18bd5984b,
> addrs=[10.20.0.223, 127.0.0.1], sockAddrs=[fosters-223/10.20.0.223:47500,
> /10.20.0.223:47500, /127.0.0.1:47500], discPort=47500, order=4, intOrder=4,
> lastExchangeTime=1464360660689, loc=false, ver=1.6.0#20160525-sha1:48321a40,
> isClient=false], TcpDiscoveryNode [id=ba89970d-8908-4276-9d3c-b679a335c2d2,
> addrs=[10.20.0.216, 127.0.0.1], sockAddrs=[fosters-216/10.20.0.216:47501,
> /10.20.0.216:47501, /127.0.0.1:47501], discPort=47501, order=5, intOrder=5,
> lastExchangeTime=1464360661276, loc=false, ver=1.6.0#20160525-sha1:48321a40,
> isClient=false], TcpDiscoveryNode [id=317b61fc-186b-4bf7-9622-6c4094409814,
> addrs=[10.20.0.221, 127.0.0.1], sockAddrs=[fosters-221/10.20.0.221:47501,
> /10.20.0.221:47501, /127.0.0.1:47501], discPort=47501, order=6, intOrder=6,
> lastExchangeTime=1464360662086, loc=false, ver=1.6.0#20160525-sha1:48321a40,
> isClient=false], TcpDiscoveryNode [id=425b461c-5117-403d-af2b-e323b7c1aa39,
> addrs=[10.20.0.222, 127.0.0.1], sockAddrs=[fosters-222/10.20.0.222:47501,
> /10.20.0.222:47501, /127.0.0.1:47501], discPort=47501, order=7, intOrder=7,
> lastExchangeTime=1464360662471, loc=false, ver=1.6.0#20160525-sha1:48321a40,
> isClient=false], TcpDiscoveryNode [id=a550929c-1f4b-4de3-8d57-47b4a8232010,
> addrs=[10.20.0.223, 127.0.0.1], sockAddrs=[fosters-223/10.20.0.223:47501,
> /10.20.0.223:47501, /127.0.0.1:47501], discPort=47501, order=8, intOrder=8,
> lastExchangeTime=1464360662886, loc=false, ver=1.6.0#20160525-sha1:48321a40,
> isClient=false], TcpDiscoveryNode [id=e7f032f5-4ec6-48c8-9cd4-ac78b0f8ccc4,
> addrs=[10.20.0.216, 127.0.0.1], sockAddrs=[fosters-216/10.20.0.216:47502,
> /10.20.0.216:47502, /127.0.0.1:47502], discPort=47502, order=9, intOrder=9,
> lastExchangeTime=1464360663373, loc=false, ver=1.6.0#20160525-sha1:48321a40,
> isClient=false], TcpDiscoveryNode [id=b51cc458-c884-41ce-9e5b-b83ea111218b,
> addrs=[10.20.0.221, 127.0.0.1], sockAddrs=[fosters-221/10.20.0.221:47502,
> /10.20.0.221:47502, /127.0.0.1:47502], discPort=47502, order=10, intOrder=10,
> lastExchangeTime=1464360664058, loc=false, ver=1.6.0#20160525-sha1:48321a40,
> isClient=false], TcpDiscoveryNode [id=fc79bff1-480f-44b3-9915-2548f937726c,
> addrs=[10.20.0.222, 127.0.0.1], sockAddrs=[fosters-222/10.20.0.222:47502,
> /10.20.0.222:47502, /127.0.0.1:47502], discPort=47502, order=11, intOrder=11,
> lastExchangeTime=1464360664685, loc=false, ver=1.6.0#20160525-sha1:48321a40,
> isClient=false], TcpDiscoveryNode [id=a7f0feb2-b932-4843-ac93-560fc547700e,
> addrs=[10.20.0.223, 127.0.0.1], sockAddrs=[fosters-223/10.20.0.223:47502,
> /10.20.0.223:47502, /127.0.0.1:47502], discPort=47502, order=12, intOrder=12,
> lastExchangeTime=1464360665216, loc=false, ver=1.6.0#20160525-sha1:48321a40,
> isClient=false], TcpDiscoveryNode [id=9c184df6-e5e0-49b3-9836-65932d58be6a,
> addrs=[10.20.0.216, 127.0.0.1], sockAddrs=[fosters-216/10.20.0.216:47503,
> /10.20.0.216:47503, /127.0.0.1:47503], discPort=47503, order=13, intOrder=13,
> lastExchangeTime=1464360665661, loc=false, ver=1.6.0#20160525-sha1:48321a40,
> isClient=false], TcpDiscoveryNode [id=6970f13f-fe82-4583-a128-af93f86e99e7,
> addrs=[10.20.0.216, 127.0.0.1], sockAddrs=[fosters-216/10.20.0.216:47504,
> /10.20.0.216:47504, /127.0.0.1:47504], discPort=47504, order=17, intOrder=17,
> lastExchangeTime=1464360668158, loc=false, ver=1.6.0#20160525-sha1:48321a40,
> isClient=false], TcpDiscoveryNode [id=d0e84363-590d-45bf-bad3-b88cde5d03fe,
> addrs=[10.20.0.221, 127.0.0.1], sockAddrs=[fosters-221/10.20.0.221:47504,
> /10.20.0.221:47504, /127.0.0.1:47504], discPort=47504, order=18, intOrder=18,
> lastExchangeTime=1464360668768, loc=false, ver=1.6.0#20160525-sha1:48321a40,
> isClient=false], TcpDiscoveryNode [id=c013837c-b46e-4667-9e47-e3c41412b5a1,
> addrs=[10.20.0.222, 127.0.0.1], sockAddrs=[fosters-222/10.20.0.222:47504,
> /10.20.0.222:47504, /127.0.0.1:47504], discPort=47504, order=19, intOrder=19,
> lastExchangeTime=1464360669745, loc=false, ver=1.6.0#20160525-sha1:48321a40,
> isClient=false], TcpDiscoveryNode [id=3c462bba-1d12-43ab-ad21-761f9ef94aa1,
> addrs=[10.20.0.223, 127.0.0.1], sockAddrs=[fosters-223/10.20.0.223:47504,
> /10.20.0.223:47504, /127.0.0.1:47504], discPort=47504, order=20, intOrder=20,
> lastExchangeTime=1464360669836, loc=false, ver=1.6.0#20160525-sha1:48321a40,
> isClient=false], TcpDiscoveryNode [id=c978e55f-4de6-4ffd-98b1-459215db2232,
> addrs=[10.20.0.221, 127.0.0.1], sockAddrs=[fosters-221/10.20.0.221:47503,
> /10.20.0.221:47503, /127.0.0.1:47503], discPort=47503, order=37, intOrder=28,
> lastExchangeTime=1464362949633, loc=false, ver=1.6.0#20160525-sha1:48321a40,
> isClient=false], TcpDiscoveryNode [id=a9b8fe06-cda9-4907-871d-0684bcc2c3ca,
> addrs=[10.20.0.222, 127.0.0.1], sockAddrs=[fosters-222/10.20.0.222:47503,
> /10.20.0.222:47503, /127.0.0.1:47503], discPort=47503, order=38, intOrder=29,
> lastExchangeTime=1464362981162, loc=false, ver=1.6.0#20160525-sha1:48321a40,
> isClient=false], TcpDiscoveryNode [id=a4d0911c-d54a-4a36-b911-b74eddc3f0cc,
> addrs=[10.20.0.223, 127.0.0.1], sockAddrs=[fosters-223/10.20.0.223:47503,
> /10.20.0.223:47503, /127.0.0.1:47503], discPort=47503, order=39, intOrder=30,
> lastExchangeTime=1464363011016, loc=false, ver=1.6.0#20160525-sha1:48321a40,
> isClient=false]], super=GridFutureAdapter [resFlag=1, res=class
> o.a.i.IgniteException: null, startTime=1464363130370, endTime=1464363141003,
> ignoreInterrupts=false, state=DONE]]
> class org.apache.ignite.IgniteCheckedException: null
> at
> org.apache.ignite.internal.util.IgniteUtils.cast(IgniteUtils.java:7067)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:168)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:117)
> at
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1335)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.util.ConcurrentModificationException
> at
> java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:394)
> at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:413)
> at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:412)
> at java.util.AbstractMap.toString(AbstractMap.java:518)
> at java.lang.String.valueOf(String.java:2849)
> at
> org.apache.ignite.internal.util.GridStringBuilder.a(GridStringBuilder.java:101)
> at
> org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:474)
> at
> org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:364)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxStateImpl.toString(IgniteTxStateImpl.java:443)
> at java.lang.String.valueOf(String.java:2849)
> at
> org.apache.ignite.internal.util.GridStringBuilder.a(GridStringBuilder.java:101)
> at
> org.apache.ignite.internal.util.tostring.GridToStringBuilder.toStringImpl(GridToStringBuilder.java:474)
> at
> org.apache.ignite.internal.util.tostring.GridToStringBuilder.toString(GridToStringBuilder.java:297)
> at
> org.apache.ignite.internal.processors.cache.transactions.IgniteTxLocalAdapter.toString(IgniteTxLocalAdapter.java:3743)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocalAdapter.toString(GridDhtTxLocalAdapter.java:868)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtTxLocal.toString(GridDhtTxLocal.java:703)
> at java.lang.String.valueOf(String.java:2849)
> at java.lang.StringBuilder.append(StringBuilder.java:128)
> at
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.dumpPendingObjects(GridCachePartitionExchangeManager.java:1172)
> at
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager.dumpDebugInfo(GridCachePartitionExchangeManager.java:1150)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.dumpPendingObjects(GridDhtPartitionsExchangeFuture.java:894)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.waitPartitionRelease(GridDhtPartitionsExchangeFuture.java:769)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.distributedExchange(GridDhtPartitionsExchangeFuture.java:715)
> at
> org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(GridDhtPartitionsExchangeFuture.java:472)
> at
> org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePartitionExchangeManager.java:1333)
> ... 2 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
