[jira] [Commented] (IGNITE-10644) CorruptedTreeException might occur after force node kill during transaction
[ https://issues.apache.org/jira/browse/IGNITE-10644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767242#comment-16767242 ] Ilya Kasnacheev commented on IGNITE-10644: -- I cannot reproduce it after IGNITE-9303 any more! [~pvinokurov] please check if you can still observe it. > CorruptedTreeException might occur after force node kill during transaction > --- > > Key: IGNITE-10644 > URL: https://issues.apache.org/jira/browse/IGNITE-10644 > Project: Ignite > Issue Type: Bug >Reporter: Pavel Voronkin >Priority: Major > > Partition eviction process on the other hand: > > 2018-12-10 20:59:24.426 > [ERROR]sys-#204%_GRID%GridNodeName%[o.a.i.i.p.c.d.d.t.PartitionsEvictManager] > Partition eviction failed, this can cause grid hang. > org.h2.message.DbException: General error: "class > org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: > Runtime failure on search row: Row@3580787f[ key: 4071535538120363041, val: > X.common.dpl.model.backstream.DBackStreamMessage_DPL_PROXY > [idHash=1961442513, hash=529139710, colocationKey=14465, entityType=I, > lastChangeDate=1544464745135, errorMessage=No api > [X.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] services > available for route: [*][*][kbt] (zone-node-module).IP: [*]. > List of services violations: > NODE MODULE FILTER VIOLATIONS > No services or violations were found for routing, partition_X_id=5, > messageId=1211871172446406939, entityId=1211871174131851324, ownerId=ucp, > responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10 > 20:59:05 MSK 2018, id=4071535538120363041], ver: GridCacheVersion > [topVer=155940834, order=1544596983071, nodeOrder=114] ][ I, null, > 1211871172446406939, 1211871174131851324, null, 1, 2018-12-10 20:59:05.115, > No api [X.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] > services available for route: [*][*][kbt] (zone-node-module).IP: [*]. > List of services violations: > NODE MODULE FILTER VIOLATIONS > No services or violations were found for routing, 4071535538120363041, FALSE, > 5 ]" [5-195] > at org.h2.message.DbException.get(DbException.java:168) > at org.h2.message.DbException.convert(DbException.java:295) > at > org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.removex(H2TreeIndex.java:293) > at > org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.remove(GridH2Table.java:515) > at > org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.remove(IgniteH2Indexing.java:738) > at > org.apache.ignite.internal.processors.query.GridQueryProcessor.remove(GridQueryProcessor.java:2487) > at > org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.remove(GridCacheQueryManager.java:433) > at > org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishRemove(IgniteCacheOffheapManagerImpl.java:1465) > at > org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.remove(IgniteCacheOffheapManagerImpl.java:1435) > at > org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.remove(GridCacheOffheapManager.java:1633) > at > org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.remove(IgniteCacheOffheapManagerImpl.java:383) > at > org.apache.ignite.internal.processors.cache.GridCacheMapEntry.removeValue(GridCacheMapEntry.java:3706) > at > org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry.clearInternal(GridDhtCacheEntry.java:652) > at > org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition.clearAll(GridDhtLocalPartition.java:1079) > at > org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition.tryClear(GridDhtLocalPartition.java:915) > at > org.apache.ignite.internal.processors.cache.distributed.dht.topology.PartitionsEvictManager$PartitionEvictionTask.run(PartitionsEvictManager.java:423) > at > org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6782) > at > org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:827) > at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.h2.jdbc.JdbcSQLException: General error: "class > org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: > Runtime failure on search row: Row@3580787f[ key: 4071535538120363041, val: >
[jira] [Commented] (IGNITE-10644) CorruptedTreeException might occur after force node kill during transaction
[
https://issues.apache.org/jira/browse/IGNITE-10644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16765123#comment-16765123
]
Ilya Kasnacheev commented on IGNITE-10644:
--
I can also get JVM crash:
{code}
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x7f0d91a9748f, pid=22056, tid=0x7f0d64eda700
#
# JRE version: Java(TM) SE Runtime Environment (8.0_144-b01) (build
1.8.0_144-b01)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.144-b01 mixed mode linux-amd64
compressed oops)
# Problematic frame:
# J 3661 C2
org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions.forPage(J)Lorg/apache/ignite/internal/processors/cache/persistence/tree/io/PageIO;
(76 bytes) @ 0x7f0d91a9748f [0x7f0d91a97460+0x2f]
#
# Failed to write core dump. Core dumps have been disabled. To enable core
dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /home/gridgain/w/index-reproducer/hs_err_pid22056.log
Compiled method (c2) 10820 3661 4
org.apache.ignite.internal.processors.cache.persistence.tree.io.IOVersions::forPage
(76 bytes)
total in heap [0x7f0d91a97310,0x7f0d91a97620] = 784
relocation [0x7f0d91a97438,0x7f0d91a97450] = 24
main code [0x7f0d91a97460,0x7f0d91a97520] = 192
stub code [0x7f0d91a97520,0x7f0d91a97538] = 24
oops [0x7f0d91a97538,0x7f0d91a97540] = 8
metadata [0x7f0d91a97540,0x7f0d91a97550] = 16
scopes data[0x7f0d91a97550,0x7f0d91a975b8] = 104
scopes pcs [0x7f0d91a975b8,0x7f0d91a97608] = 80
dependencies [0x7f0d91a97608,0x7f0d91a97610] = 8
nul chk table [0x7f0d91a97610,0x7f0d91a97620] = 16
#
# If you would like to submit a bug report, please visit:
# http://bugreport.java.com/bugreport/crash.jsp
#
{code}
> CorruptedTreeException might occur after force node kill during transaction
> ---
>
> Key: IGNITE-10644
> URL: https://issues.apache.org/jira/browse/IGNITE-10644
> Project: Ignite
> Issue Type: Bug
>Reporter: Pavel Voronkin
>Priority: Major
>
> Partition eviction process on the other hand:
>
> 2018-12-10 20:59:24.426
> [ERROR]sys-#204%_GRID%GridNodeName%[o.a.i.i.p.c.d.d.t.PartitionsEvictManager]
> Partition eviction failed, this can cause grid hang.
> org.h2.message.DbException: General error: "class
> org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException:
> Runtime failure on search row: Row@3580787f[ key: 4071535538120363041, val:
> X.common.dpl.model.backstream.DBackStreamMessage_DPL_PROXY
> [idHash=1961442513, hash=529139710, colocationKey=14465, entityType=I,
> lastChangeDate=1544464745135, errorMessage=No api
> [X.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] services
> available for route: [*][*][kbt] (zone-node-module).IP: [*].
> List of services violations:
> NODE MODULE FILTER VIOLATIONS
> No services or violations were found for routing, partition_X_id=5,
> messageId=1211871172446406939, entityId=1211871174131851324, ownerId=ucp,
> responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10
> 20:59:05 MSK 2018, id=4071535538120363041], ver: GridCacheVersion
> [topVer=155940834, order=1544596983071, nodeOrder=114] ][ I, null,
> 1211871172446406939, 1211871174131851324, null, 1, 2018-12-10 20:59:05.115,
> No api [X.scripts.ucp.retail.propagate.publicapi.ClientPropagateService]
> services available for route: [*][*][kbt] (zone-node-module).IP: [*].
> List of services violations:
> NODE MODULE FILTER VIOLATIONS
> No services or violations were found for routing, 4071535538120363041, FALSE,
> 5 ]" [5-195]
> at org.h2.message.DbException.get(DbException.java:168)
> at org.h2.message.DbException.convert(DbException.java:295)
> at
> org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.removex(H2TreeIndex.java:293)
> at
> org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.remove(GridH2Table.java:515)
> at
> org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.remove(IgniteH2Indexing.java:738)
> at
> org.apache.ignite.internal.processors.query.GridQueryProcessor.remove(GridQueryProcessor.java:2487)
> at
> org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.remove(GridCacheQueryManager.java:433)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishRemove(IgniteCacheOffheapManagerImpl.java:1465)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.remove(IgniteCacheOffheapManagerImpl.java:1435)
> at
>
[jira] [Commented] (IGNITE-10644) CorruptedTreeException might occur after force node kill during transaction
[ https://issues.apache.org/jira/browse/IGNITE-10644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16721084#comment-16721084 ] Pavel Voronkin commented on IGNITE-10644: - Reproducer attached. > CorruptedTreeException might occur after force node kill during transaction > --- > > Key: IGNITE-10644 > URL: https://issues.apache.org/jira/browse/IGNITE-10644 > Project: Ignite > Issue Type: Bug >Reporter: Pavel Voronkin >Priority: Major > Attachments: IndexingTest.java > > > Partition eviction process on the other hand: > > 2018-12-10 20:59:24.426 > [ERROR]sys-#204%_GRID%GridNodeName%[o.a.i.i.p.c.d.d.t.PartitionsEvictManager] > Partition eviction failed, this can cause grid hang. > org.h2.message.DbException: General error: "class > org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: > Runtime failure on search row: Row@3580787f[ key: 4071535538120363041, val: > X.common.dpl.model.backstream.DBackStreamMessage_DPL_PROXY > [idHash=1961442513, hash=529139710, colocationKey=14465, entityType=I, > lastChangeDate=1544464745135, errorMessage=No api > [X.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] services > available for route: [*][*][kbt] (zone-node-module).IP: [*]. > List of services violations: > NODE MODULE FILTER VIOLATIONS > No services or violations were found for routing, partition_X_id=5, > messageId=1211871172446406939, entityId=1211871174131851324, ownerId=ucp, > responseDate=null, entityVersion=1, isDeleted=false, requestDate=Mon Dec 10 > 20:59:05 MSK 2018, id=4071535538120363041], ver: GridCacheVersion > [topVer=155940834, order=1544596983071, nodeOrder=114] ][ I, null, > 1211871172446406939, 1211871174131851324, null, 1, 2018-12-10 20:59:05.115, > No api [X.scripts.ucp.retail.propagate.publicapi.ClientPropagateService] > services available for route: [*][*][kbt] (zone-node-module).IP: [*]. > List of services violations: > NODE MODULE FILTER VIOLATIONS > No services or violations were found for routing, 4071535538120363041, FALSE, > 5 ]" [5-195] > at org.h2.message.DbException.get(DbException.java:168) > at org.h2.message.DbException.convert(DbException.java:295) > at > org.apache.ignite.internal.processors.query.h2.database.H2TreeIndex.removex(H2TreeIndex.java:293) > at > org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.remove(GridH2Table.java:515) > at > org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.remove(IgniteH2Indexing.java:738) > at > org.apache.ignite.internal.processors.query.GridQueryProcessor.remove(GridQueryProcessor.java:2487) > at > org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.remove(GridCacheQueryManager.java:433) > at > org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.finishRemove(IgniteCacheOffheapManagerImpl.java:1465) > at > org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl$CacheDataStoreImpl.remove(IgniteCacheOffheapManagerImpl.java:1435) > at > org.apache.ignite.internal.processors.cache.persistence.GridCacheOffheapManager$GridCacheDataStore.remove(GridCacheOffheapManager.java:1633) > at > org.apache.ignite.internal.processors.cache.IgniteCacheOffheapManagerImpl.remove(IgniteCacheOffheapManagerImpl.java:383) > at > org.apache.ignite.internal.processors.cache.GridCacheMapEntry.removeValue(GridCacheMapEntry.java:3706) > at > org.apache.ignite.internal.processors.cache.distributed.dht.GridDhtCacheEntry.clearInternal(GridDhtCacheEntry.java:652) > at > org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition.clearAll(GridDhtLocalPartition.java:1079) > at > org.apache.ignite.internal.processors.cache.distributed.dht.topology.GridDhtLocalPartition.tryClear(GridDhtLocalPartition.java:915) > at > org.apache.ignite.internal.processors.cache.distributed.dht.topology.PartitionsEvictManager$PartitionEvictionTask.run(PartitionsEvictManager.java:423) > at > org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6782) > at > org.apache.ignite.internal.processors.closure.GridClosureProcessor$1.body(GridClosureProcessor.java:827) > at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > at java.lang.Thread.run(Thread.java:745) > Caused by: org.h2.jdbc.JdbcSQLException: General error: "class > org.apache.ignite.internal.processors.cache.persistence.tree.CorruptedTreeException: > Runtime failure on search row: Row@3580787f[ key: 4071535538120363041, val: >
[jira] [Commented] (IGNITE-10644) CorruptedTreeException might occur after force node kill during transaction
[ https://issues.apache.org/jira/browse/IGNITE-10644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16717275#comment-16717275 ] Dmitriy Pavlov commented on IGNITE-10644: - [~voropava] could you please fill ticket description? How contributor could possibly understand the context of the problem? > CorruptedTreeException might occur after force node kill during transaction > --- > > Key: IGNITE-10644 > URL: https://issues.apache.org/jira/browse/IGNITE-10644 > Project: Ignite > Issue Type: Bug >Reporter: Pavel Voronkin >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
