[
https://issues.apache.org/jira/browse/IGNITE-3465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15384319#comment-15384319
]
Semen Boikov commented on IGNITE-3465:
--------------------------------------
Looks like crash happened at the moment when ignite tried to call 'toString'
for non initialized BinaryObject (possible when object is in process of reading
from socket). Added check in 'toString' to avoid crash.
> Java crash: problematic frame
> org.apache.ignite.internal.binary.BinaryObjectImpl.hashCode()
> -------------------------------------------------------------------------------------------
>
> Key: IGNITE-3465
> URL: https://issues.apache.org/jira/browse/IGNITE-3465
> Project: Ignite
> Issue Type: Bug
> Reporter: Ksenia Rybakova
> Assignee: Semen Boikov
> Attachments: hs_err_pid12976.log, ignite-base-load-config.xml,
> run-load.properties, run-load.xml
>
>
> During streaming load test some of JVMs crashed while logging info about
> "Failed to wait for partition map exchange" problem
> {noformat}
> [01:05:23,949][WARN
> ][exchange-worker-#96%null%][GridCachePartitionExchangeManager] Failed to
> wait for partition map exchange [topVer=AffinityTopologyVersion [topVer=61,
> minorTopVer=0], node=4f7b195a-62e3-4c32-b496-4a98de50b277]. Dumping
> [01:05:23,949][WARN
> ][exchange-worker-#96%null%][GridCachePartitionExchangeManager] Ready
> affinity version: AffinityTopologyVersion [topVer=60, minorTopVer=0]
> [01:05:24,107][WARN
> ][exchange-worker-#96%null%][GridCachePartitionExchangeManager] Last exchange
> future: GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false,
> reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
> [01:05:24,108][WARN
> ][exchange-worker-#96%null%][GridCachePartitionExchangeManager] Pending
> exchange futures:
> [01:05:24,108][WARN
> ][exchange-worker-#96%null%][GridCachePartitionExchangeManager] Last 10
> exchange futures (total: 48):
> [01:05:24,109][WARN
> ][exchange-worker-#96%null%][GridCachePartitionExchangeManager] >>>
> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false,
> reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
> [id=7291b03d-ce91
> [01:05:24,110][WARN
> ][exchange-worker-#96%null%][GridCachePartitionExchangeManager] >>>
> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false,
> reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
> [id=739ab624-7bce
> [01:05:24,111][WARN
> ][exchange-worker-#96%null%][GridCachePartitionExchangeManager] >>>
> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false,
> reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
> [id=93572383-07e8
> [01:05:24,111][WARN
> ][exchange-worker-#96%null%][GridCachePartitionExchangeManager] >>>
> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false,
> reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
> [id=3c24caa1-d4b2
> [01:05:24,112][WARN
> ][exchange-worker-#96%null%][GridCachePartitionExchangeManager] >>>
> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false,
> reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
> [id=e762044c-8ccc
> [01:05:24,118][WARN
> ][exchange-worker-#96%null%][GridCachePartitionExchangeManager] >>>
> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false,
> reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
> [id=5ffbe144-f843
> [01:05:24,119][WARN
> ][exchange-worker-#96%null%][GridCachePartitionExchangeManager] >>>
> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false,
> reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
> [id=28199123-fa5b
> [01:05:24,119][WARN
> ][exchange-worker-#96%null%][GridCachePartitionExchangeManager] >>>
> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false,
> reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
> [id=14d27ae8-ee26
> [01:05:24,120][WARN
> ][exchange-worker-#96%null%][GridCachePartitionExchangeManager] >>>
> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false,
> reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
> [id=fea5c90e-0467
> [01:05:24,120][WARN
> ][exchange-worker-#96%null%][GridCachePartitionExchangeManager] >>>
> GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false,
> reassign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
> [id=a5370f23-42bc
> [01:05:24,122][WARN
> ][exchange-worker-#96%null%][GridCachePartitionExchangeManager] Pending
> transactions:
> [01:05:24,767][WARN
> ][exchange-worker-#96%null%][GridCachePartitionExchangeManager] >>>
> GridDhtTxRemote [nearNodeId=fea5c90e-0467-462a-8755-928b539fb278,
> rmtFutId=a52022ed551-938a204f-bf03-451b-8425-be7047673bf6,
> nearXidVer=GridCacheVers
> [01:05:24,769][WARN
> ][exchange-worker-#96%null%][GridCachePartitionExchangeManager] >>>
> GridDhtTxRemote [nearNodeId=739ab624-7bce-4e4b-81cc-3696f65247bd,
> rmtFutId=ddfe12ed551-15e1d48c-8146-44ac-8d7b-370ea34fa44b,
> nearXidVer=GridCacheVers
> [01:05:24,771][WARN
> ][exchange-worker-#96%null%][GridCachePartitionExchangeManager] >>>
> GridDhtTxRemote [nearNodeId=5ffbe144-f843-4ad1-9ade-688a18f0f872,
> rmtFutId=401f12ed551-62727d5b-0dd3-48a0-bb04-72f7363ec19b,
> nearXidVer=GridCacheVers
> [01:05:24,771][WARN
> ][exchange-worker-#96%null%][GridCachePartitionExchangeManager] Pending
> explicit locks:
> [01:05:24,771][WARN
> ][exchange-worker-#96%null%][GridCachePartitionExchangeManager] Pending cache
> futures:
> [01:05:24,771][WARN
> ][exchange-worker-#96%null%][GridCachePartitionExchangeManager] Pending
> atomic cache futures:
> [01:05:24,786][WARN ][exchange-worker-#96%null%][TcpCommunicationSpi]
> Communication SPI recovery descriptors:
> [key=ClientKey [nodeId=14d27ae8-ee26-49a3-a7b5-6d1482850f21, order=54],
> msgsSent=181, msgsAckedByRmt=176, msgsRcvd=234, lastAcked=224, reserveCnt=1,
> descIdHash=952738531]
> [key=ClientKey [nodeId=5ffbe144-f843-4ad1-9ade-688a18f0f872, order=56],
> msgsSent=131, msgsAckedByRmt=128, msgsRcvd=168, lastAcked=160, reserveCnt=1,
> descIdHash=1038418485]
> [key=ClientKey [nodeId=876bec01-deea-4db8-a5ad-9159864ecae9, order=13],
> msgsSent=88, msgsAckedByRmt=88, msgsRcvd=88, lastAcked=88, reserveCnt=1,
> descIdHash=99878226]
> [key=ClientKey [nodeId=e762044c-8ccc-45d4-bd15-703baced868e, order=57],
> msgsSent=122, msgsAckedByRmt=112, msgsRcvd=128, lastAcked=128, reserveCnt=1,
> descIdHash=1468616075]
> [key=ClientKey [nodeId=e0454859-339d-4dd4-9766-bb96e71200c5, order=41],
> msgsSent=136, msgsAckedByRmt=128, msgsRcvd=165, lastAcked=160, reserveCnt=1,
> descIdHash=2000817554]
> [key=ClientKey [nodeId=93572383-07e8-4a6b-9482-642af1b87281, order=59],
> msgsSent=177, msgsAckedByRmt=176, msgsRcvd=186, lastAcked=176, reserveCnt=1,
> descIdHash=294201705]
> [key=ClientKey [nodeId=a5370f23-42bc-449d-93f7-83b04ea13668, order=52],
> msgsSent=282, msgsAckedByRmt=272, msgsRcvd=279, lastAcked=272, reserveCnt=1,
> descIdHash=1140700541]
> [key=ClientKey [nodeId=75627e99-8f2c-40e3-a0ba-79e28c1b4c32, order=44],
> msgsSent=139, msgsAckedByRmt=128, msgsRcvd=196, lastAcked=192, reserveCnt=1,
> descIdHash=1167475818]
> [key=ClientKey [nodeId=52f6e634-4ab7-4018-802f-c6f6cf7c9d3f, order=16],
> msgsSent=195, msgsAckedByRmt=192, msgsRcvd=197, lastAcked=192, reserveCnt=1,
> descIdHash=207990487]
> [key=ClientKey [nodeId=cd11646a-d9df-4611-bcc5-9b9625dd588b, order=17],
> msgsSent=187, msgsAckedByRmt=176, msgsRcvd=187, lastAcked=176, reserveCnt=3,
> descIdHash=1740221340]
> [key=ClientKey [nodeId=408d0657-3342-48c4-9ba0-883fba5ecea8, order=42],
> msgsSent=179, msgsAckedByRmt=176, msgsRcvd=205, lastAcked=192, reserveCnt=1,
> descIdHash=1712625195]
> [key=ClientKey [nodeId=aa52c200-1686-4985-acd7-7aadad71d874, order=50],
> msgsSent=89, msgsAckedByRmt=80, msgsRcvd=122, lastAcked=112, reserveCnt=1,
> descIdHash=1368331624]
> [key=ClientKey [nodeId=3c24caa1-d4b2-4827-9a12-be3472c09f2a, order=58],
> msgsSent=70, msgsAckedByRmt=64, msgsRcvd=55, lastAcked=48, reserveCnt=1,
> descIdHash=147228845]
> [key=ClientKey [nodeId=47935631-2877-4d7d-be19-08333ae87942, order=49],
> msgsSent=86, msgsAckedByRmt=80, msgsRcvd=112, lastAcked=112, reserveCnt=1,
> descIdHash=574036571]
> [key=ClientKey [nodeId=28199123-fa5b-4fa3-98d3-ee5c449662b0, order=55],
> msgsSent=151, msgsAckedByRmt=144, msgsRcvd=221, lastAcked=208, reserveCnt=1,
> descIdHash=1336106473]
> [key=ClientKey [nodeId=2c1893ad-ffe9-4f5f-92d2-5ecf38080337, order=43],
> msgsSent=76, msgsAckedByRmt=64, msgsRcvd=205, lastAcked=192, reserveCnt=1,
> descIdHash=1852478679]
> [key=ClientKey [nodeId=fff86516-4034-43af-b417-0d845f49c142, order=47],
> msgsSent=134, msgsAckedByRmt=128, msgsRcvd=135, lastAcked=128, reserveCnt=1,
> descIdHash=391243747]
> [key=ClientKey [nodeId=796d2fe8-4598-4794-9ec3-53ff3e4f1630, order=1],
> msgsSent=62, msgsAckedByRmt=61, msgsRcvd=112, lastAcked=112, reserveCnt=1,
> descIdHash=38728715]
> [key=ClientKey [nodeId=e4d6fe30-29da-4958-8a8b-f50c5c8c8f3f, order=2],
> msgsSent=62, msgsAckedByRmt=62, msgsRcvd=62, lastAcked=62, reserveCnt=1,
> descIdHash=225967964]
> [key=ClientKey [nodeId=c7955df7-d6ef-4a8e-976a-059eb644fa10, order=45],
> msgsSent=185, msgsAckedByRmt=176, msgsRcvd=244, lastAcked=240, reserveCnt=1,
> descIdHash=1092483052]
> [key=ClientKey [nodeId=6d9b4404-561d-475e-a9e2-4c854b2d10e7, order=28],
> msgsSent=159, msgsAckedByRmt=144, msgsRcvd=159, lastAcked=144, reserveCnt=1,
> descIdHash=377473114]
> [key=ClientKey [nodeId=bc0e61c1-fe0e-432e-bafa-6c159fcab4d9, order=6],
> msgsSent=0, msgsAckedByRmt=0, msgsRcvd=20, lastAcked=20, reserveCnt=1,
> descIdHash=523933760]
> [key=ClientKey [nodeId=739ab624-7bce-4e4b-81cc-3696f65247bd, order=60],
> msgsSent=43, msgsAckedByRmt=43, msgsRcvd=12, lastAcked=12, reserveCnt=1,
> descIdHash=1071751620]
> [key=ClientKey [nodeId=38ffbc41-33ac-4c5f-9569-197cefa678b3, order=48],
> msgsSent=161, msgsAckedByRmt=160, msgsRcvd=190, lastAcked=176, reserveCnt=1,
> descIdHash=1984129951]
> [key=ClientKey [nodeId=fea5c90e-0467-462a-8755-928b539fb278, order=53],
> msgsSent=151, msgsAckedByRmt=144, msgsRcvd=179, lastAcked=176, reserveCnt=1,
> descIdHash=1146015841]
> [key=ClientKey [nodeId=78d09665-9ded-4d1c-a46d-11a212af90fa, order=46],
> msgsSent=162, msgsAckedByRmt=160, msgsRcvd=192, lastAcked=192, reserveCnt=1,
> descIdHash=307409744]
> [key=ClientKey [nodeId=072afa77-8869-446a-8454-36e47956ede9, order=51],
> msgsSent=49, msgsAckedByRmt=49, msgsRcvd=18, lastAcked=18, reserveCnt=1,
> descIdHash=1937396373]
> #
> # A fatal error has been detected by the Java Runtime Environment:
> #
> # SIGSEGV (0xb) at pc=0x00007fb6853771c4, pid=12976, tid=140414314960640
> #
> # JRE version: Java(TM) SE Runtime Environment (7.0_80-b15) (build
> 1.7.0_80-b15)
> # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.80-b11 mixed mode
> linux-amd64 compressed oops)
> # Problematic frame:
> # J 1501 C2 org.apache.ignite.internal.binary.BinaryObjectImpl.hashCode()I
> (15 bytes) @ 0x00007fb6853771c4 [0x00007fb685377180+0x44]
> #
> # Failed to write core dump. Core dumps have been disabled. To enable core
> dumping, try "ulimit -c unlimited" before starting Java again
> #
> # An error report file with more information is saved as:
> # hs_err_pid12976.log
> [thread 140414317065984 also had an error]
> #
> # If you would like to submit a bug report, please visit:
> # http://bugreport.java.com/bugreport/crash.jsp
> #
> {noformat}
> Load test config:
> - Benchmark name: IgniteCacheRandomOperationBenchmark
> - Operations: preloading (streaming)
> - Heap: 8Gb for servers, 4Gb for clients
> - 20 clients, 40 servers
> - Preload amount: 8M
> - Number of caches: 12
> - Types of caches (atomicity mode): different (atomic, transactional)
> - Types of caches (tiered storage mode) : different (onheap without
> eviction, onheap with eviction, offheap_tired, offheap_values)
> - Types of caches (indexing): different (with and without indexes)
> - Types of caches (cache mode): partitioned
> - Backups count: 1
> Attachments:
> - Error report file hs_err_pid12976.log
> - xml configs and property file that were used to run yardstick
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
