Vyacheslav Koptilin created IGNITE-17542:
--------------------------------------------
Summary: Test
CacheLateAffinityAssignmentTest.testAffinitySimpleNoCacheOnCoordinator2 became
flaky after IGNITE-17507
Key: IGNITE-17542
URL: https://issues.apache.org/jira/browse/IGNITE-17542
Project: Ignite
Issue Type: Bug
Affects Versions: 2.14
Reporter: Vyacheslav Koptilin
Assignee: Vyacheslav Koptilin
Fix For: 2.14
The test
CacheLateAffinityAssignmentTest.testAffinitySimpleNoCacheOnCoordinator2 became
flay due to IGNITE-17507.
The root cause of the issue that _CacheAffinityChangeMessage_ mutates the
message outside the _disco-notifier_ thread, and this fact may lead to the
following exception:
{noformat}
[2022-08-16T21:10:32,133][ERROR][tcp-disco-msg-worker-[0448095b
127.0.0.1:47502]-#5308%distributed.CacheLateAffinityAssignmentTest3%-#98199%distributed.CacheLateAffinityAssignmentTest3%][TestTcpDiscoverySpi]
TcpDiscoverSpi's message worker thread failed abnormally. Stopping the node in
order to prevent cluster wide instability.
org.apache.ignite.IgniteException: Failed to marshal mutable discovery
message: CacheAffinityChangeMessage
[id=ea31ffaa281-0286b465-6baf-4ad8-9e3b-3f8cb755d1dd, topVer=null,
exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=6,
minorTopVer=0], discoEvt=null, nodeId=f9f9faf0, evt=NODE_LEFT],
partsMsg=GridDhtPartitionsFullMessage [parts=HashMap
{-2100569601=GridDhtPartitionFullMap
{f57cbb85-44ba-40d1-814e-937f96c00003=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=6, minorTopVer=0], updateSeq=111,
size=100], 0448095b-02d8-470c-ab90-6a5bcf800002=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=6, minorTopVer=0], updateSeq=116,
size=100]}, 1251687457=GridDhtPartitionFullMap
{f57cbb85-44ba-40d1-814e-937f96c00003=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=6, minorTopVer=0], updateSeq=1035,
size=1024], 0448095b-02d8-470c-ab90-6a5bcf800002=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=4, minorTopVer=0], updateSeq=3, size=0]}},
partCntrs=IgniteDhtPartitionCountersMap [], partCntrs2=null,
partHistSuppliers=IgniteDhtPartitionHistorySuppliersMap [],
partsToReload=IgniteDhtPartitionsToReloadMap [], topVer=AffinityTopologyVersion
[topVer=6, minorTopVer=0], errs=null, resTopVer=null, flags=0, partCnt=2,
super=GridDhtPartitionsAbstractMessage [exchId=GridDhtPartitionExchangeId
[topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], discoEvt=null,
nodeId=f9f9faf0, evt=NODE_LEFT], lastVer=GridCacheVersion [topVer=0,
order=1660673425660, nodeOrder=0, dataCenterId=0], super=GridCacheMessage
[msgId=-1, depInfo=null, lastAffChangedTopVer=null, err=null,
skipPrepare=false]]], exchangeNeeded=false, stopProc=false]
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.notifyDiscoveryListener(ServerImpl.java:6423)
~[classes/:?]
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processCustomMessage(ServerImpl.java:6243)
~[classes/:?]
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:3260)
~[classes/:?]
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:2918)
~[classes/:?]
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:8058)
~[classes/:?]
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:3089)
[classes/:?]
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:125)
[classes/:?]
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7989)
[classes/:?]
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:58)
[classes/:?]
Caused by: org.apache.ignite.IgniteCheckedException: Failed to serialize
object: CacheAffinityChangeMessage
[id=ea31ffaa281-0286b465-6baf-4ad8-9e3b-3f8cb755d1dd, topVer=null,
exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=6,
minorTopVer=0], discoEvt=null, nodeId=f9f9faf0, evt=NODE_LEFT],
partsMsg=GridDhtPartitionsFullMessage [parts=HashMap
{-2100569601=GridDhtPartitionFullMap
{f57cbb85-44ba-40d1-814e-937f96c00003=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=6, minorTopVer=0], updateSeq=111,
size=100], 0448095b-02d8-470c-ab90-6a5bcf800002=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=6, minorTopVer=0], updateSeq=116,
size=100]}, 1251687457=GridDhtPartitionFullMap
{f57cbb85-44ba-40d1-814e-937f96c00003=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=6, minorTopVer=0], updateSeq=1035,
size=1024], 0448095b-02d8-470c-ab90-6a5bcf800002=GridDhtPartitionMap [moving=0,
top=AffinityTopologyVersion [topVer=4, minorTopVer=0], updateSeq=3, size=0]}},
partCntrs=IgniteDhtPartitionCountersMap [], partCntrs2=null,
partHistSuppliers=IgniteDhtPartitionHistorySuppliersMap [],
partsToReload=IgniteDhtPartitionsToReloadMap [], topVer=AffinityTopologyVersion
[topVer=6, minorTopVer=0], errs=null, resTopVer=null, flags=0, partCnt=2,
super=GridDhtPartitionsAbstractMessage [exchId=GridDhtPartitionExchangeId
[topVer=AffinityTopologyVersion [topVer=6, minorTopVer=0], discoEvt=null,
nodeId=f9f9faf0, evt=NODE_LEFT], lastVer=GridCacheVersion [topVer=0,
order=1660673425660, nodeOrder=0, dataCenterId=0], super=GridCacheMessage
[msgId=-1, depInfo=null, lastAffChangedTopVer=null, err=null,
skipPrepare=false]]], exchangeNeeded=false, stopProc=false]
at
org.apache.ignite.marshaller.jdk.JdkMarshaller.marshal0(JdkMarshaller.java:102)
~[classes/:?]
at
org.apache.ignite.marshaller.jdk.JdkMarshaller.marshal0(JdkMarshaller.java:109)
~[classes/:?]
at
org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.marshal(AbstractNodeNameAwareMarshaller.java:56)
~[classes/:?]
at
org.apache.ignite.internal.util.IgniteUtils.marshal(IgniteUtils.java:10827)
[classes/:?]
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.notifyDiscoveryListener(ServerImpl.java:6420)
~[classes/:?]
... 8 more
Caused by: java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextNode(HashMap.java:1493) ~[?:?]
at java.util.HashMap$EntryIterator.next(HashMap.java:1526) ~[?:?]
at java.util.HashMap$EntryIterator.next(HashMap.java:1524) ~[?:?]
at
org.apache.ignite.internal.util.IgniteUtils.writeMap(IgniteUtils.java:5706)
~[classes/:?]
at
org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionFullMap.writeExternal(GridDhtPartitionFullMap.java:188)
~[classes/:?]
at
java.io.ObjectOutputStream.writeExternalData(ObjectOutputStream.java:1460)
~[?:?]
at
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
~[?:?]
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1179)
~[?:?]
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:349)
~[?:?]
at java.util.HashMap.internalWriteEntries(HashMap.java:1840) ~[?:?]
at java.util.HashMap.writeObject(HashMap.java:1411) ~[?:?]
at jdk.internal.reflect.GeneratedMethodAccessor17.invoke(Unknown Source)
~[?:?]
at
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:1145)
~[?:?]
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1497)
~[?:?]
at
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1433)
~[?:?]
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1179)
~[?:?]
at
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1553)
~[?:?]
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1510)
~[?:?]
at
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1433)
~[?:?]
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1179)
~[?:?]
at
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1553)
~[?:?]
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1510)
~[?:?]
at
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1433)
~[?:?]
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1179)
~[?:?]
at
java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1553)
~[?:?]
at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1510)
~[?:?]
at
java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1433)
~[?:?]
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1179)
~[?:?]
at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:349)
~[?:?]
at
org.apache.ignite.marshaller.jdk.JdkMarshaller.marshal0(JdkMarshaller.java:97)
~[classes/:?]
at
org.apache.ignite.marshaller.jdk.JdkMarshaller.marshal0(JdkMarshaller.java:109)
~[classes/:?]
at
org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.marshal(AbstractNodeNameAwareMarshaller.java:56)
~[classes/:?]
at
org.apache.ignite.internal.util.IgniteUtils.marshal(IgniteUtils.java:10827)
[classes/:?]
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.notifyDiscoveryListener(ServerImpl.java:6420)
~[classes/:?]
... 8 more
{noformat}
In my understanding, the discovery message should not be changed in any way
outside the _disco-notifier_ thread. At least,
_CacheAffinityChangeMessage.partitionsMessage()_ method should always return a
deep copy of _GridDhtPartitionsFullMessage_.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
