Hi,
My ignite cluster hung producing the following error trace. Can someone help
me identify the reason for the failure and cluster hang?
[09:57:38,964][INFO][grid-timeout-worker-#19%null%][IgniteKernal] FreeList
[name=null, buckets=256, dataPages=1208250, reusePages=0]
[09:58:38,963][INFO][grid-timeout-worker-#19%null%][IgniteKernal]
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=635920a1, name=null, uptime=27:24:08:084]
^-- H/N/C [hosts=16, nodes=16, CPUs=60]
^-- CPU [cur=0.17%, avg=0.66%, GC=0%]
^-- PageMemory [pages=19546713]
^-- Heap [used=1437MB, free=79.94%, comm=4095MB]
^-- Non heap [used=74MB, free=95.07%, comm=76MB]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=0, idle=6, qSize=0]
^-- Outbound messages queue [size=0]
[09:58:38,964][INFO][grid-timeout-worker-#19%null%][IgniteKernal] FreeList
[name=null, buckets=256, dataPages=1208250, reusePages=0]
[09:58:57,610][INFO][grid-nio-worker-tcp-comm-0-#21%null%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/10.65.97.216:47100,
rmtAddr=/10.247.193.243:46910]
[09:59:08,242][INFO][grid-nio-worker-tcp-comm-1-#22%null%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/10.65.97.216:47100,
rmtAddr=/10.117.234.100:49012]
[09:59:08,273][INFO][grid-nio-worker-tcp-comm-2-#23%null%][TcpCommunicationSpi]
Established outgoing communication connection [locAddr=/10.65.97.216:40504,
rmtAddr=/10.247.197.103:47100]
[09:59:08,474][INFO][grid-nio-worker-tcp-comm-3-#24%null%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/10.65.97.216:47100,
rmtAddr=/10.65.100.14:47386]
[09:59:08,563][INFO][grid-nio-worker-tcp-comm-0-#21%null%][TcpCommunicationSpi]
Accepted incoming communication connection [locAddr=/10.65.97.216:47100,
rmtAddr=/10.117.234.32:58504]
[09:59:26,270][SEVERE][sys-stripe-6-#7%null%][GridDhtTxRemote] Commit
failed.
class
org.apache.ignite.internal.transactions.IgniteTxHeuristicCheckedException:
Commit produced a runtime exception (all transaction entries will be
invalidated):
GridDhtTxRemote[id=45c108f7d51-00000000-06b5-15d4-0000-000000000010,
concurrency=PESSIMISTIC, isolation=REPEATABLE_READ, state=COMMITTING,
invalidate=false, rollbackOnly=false,
nodeId=04540138-5619-4184-863c-2df07914ab02, duration=0]
at
org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitIfLocked(GridDistributedTxRemoteAdapter.java:719)
at
org.apache.ignite.internal.processors.cache.distributed.GridDistributedTxRemoteAdapter.commitRemoteTx(GridDistributedTxRemoteAdapter.java:789)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.finish(IgniteTxHandler.java:1238)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.processDhtTxPrepareRequest(IgniteTxHandler.java:965)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler.access$400(IgniteTxHandler.java:95)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$5.apply(IgniteTxHandler.java:165)
at
org.apache.ignite.internal.processors.cache.transactions.IgniteTxHandler$5.apply(IgniteTxHandler.java:163)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:863)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:386)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:308)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:100)
at
org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:253)
at
org.apache.ignite.internal.managers.communication.GridIoManager.invokeListener(GridIoManager.java:1257)
at
org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:885)
at
org.apache.ignite.internal.managers.communication.GridIoManager.access$2100(GridIoManager.java:114)
at
org.apache.ignite.internal.managers.communication.GridIoManager$7.run(GridIoManager.java:802)
at
org.apache.ignite.internal.util.StripedExecutor$Stripe.run(StripedExecutor.java:483)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.OutOfMemoryError
at sun.misc.Unsafe.allocateMemory(Native Method)
at
org.apache.ignite.internal.util.GridUnsafe.allocateMemory(GridUnsafe.java:1054)
at
org.apache.ignite.internal.mem.unsafe.UnsafeMemoryProvider.nextRegion(UnsafeMemoryProvider.java:76)
at
org.apache.ignite.internal.pagemem.impl.PageMemoryNoStoreImpl.addSegment(PageMemoryNoStoreImpl.java:616)
at
org.apache.ignite.internal.pagemem.impl.PageMemoryNoStoreImpl.allocatePage(PageMemoryNoStoreImpl.java:279)
at
org.apache.ignite.internal.processors.cache.database.freelist.FreeListImpl.allocateDataPage(FreeListImpl.java:450)
at
org.apache.ignite.internal.processors.cache.database.freelist.FreeListImpl.insertDataRow(FreeListImpl.java:488)
at
org.apache.ignite.internal.processors.cache.database.RowStore.addRow(RowStore.java:72)
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.createRow(IgniteCacheOffheapManagerImpl.java:978)
at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry$UpdateClosure.call(GridCacheMapEntry.java:3937)
at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry$UpdateClosure.call(GridCacheMapEntry.java:3895)
at
org.apache.ignite.internal.processors.cache.database.tree.BPlusTree$Invoke.invokeClosure(BPlusTree.java:2966)
at
org.apache.ignite.internal.processors.cache.database.tree.BPlusTree$Invoke.access$6200(BPlusTree.java:2860)
at
org.apache.ignite.internal.processors.cache.database.tree.BPlusTree.invokeDown(BPlusTree.java:1696)
at
org.apache.ignite.internal.processors.cache.database.tree.BPlusTree.invokeDown(BPlusTree.java:1673)
at
org.apache.ignite.internal.processors.cache.database.tree.BPlusTree.invoke(BPlusTree.java:1585)
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.invoke(IgniteCacheOffheapManagerImpl.java:925)
at
org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.invoke(IgniteCacheOffheapManagerImpl.java:326)
at
org.apache.ignite.internal.processors.cache.GridCacheMapEntry.storeValue(GridCacheMapEntry.java:3241)
--
View this message in context:
http://apache-ignite-users.70518.x6.nabble.com/Error-Commit-produced-a-runtime-exception-tp15768.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.
