Sergey Kozlov created IGNITE-2615: ------------------------------------- Summary: SQL query fails for cache with an eviction policy to offheap Key: IGNITE-2615 URL: https://issues.apache.org/jira/browse/IGNITE-2615 Project: Ignite Issue Type: Bug Affects Versions: 1.5.0.final Reporter: Sergey Kozlov Priority: Critical
1 Start server node 2. Start client that creates an onheap cache and performs random put/get/sqlquery 3. Client fails at query execution after starting eviciton in offheap: Client log {noformat} >>> ----------------------------------------------------------------------- >>> Cache configuation used: 1 >>> CacheConfiguration [name=cache_d37d89a5_239_09203172203377125, >>> storeConcurrentLoadAllThreshold=5, rebalancePoolSize= 2, rebalanceTimeout=10000, evictPlc=FifoEvictionPolicy [max=30000, batchSize=1, maxMemSize=0, memSize=0], evictSync=fals e, evictKeyBufSize=1024, evictSyncConcurrencyLvl=4, evictSyncTimeout=10000, evictFilter=null, evictMaxOverflowRatio=10.0 , eagerTtl=true, dfltLockTimeout=0, startSize=1500000, nearCfg=null, writeSync=PRIMARY_SYNC, storeFactory=null, storeKee pBinary=false, loadPrevVal=false, aff=null, cacheMode=REPLICATED, atomicityMode=ATOMIC, atomicWriteOrderMode=null, backu ps=0, invalidate=false, tmLookupClsName=null, rebalanceMode=SYNC, rebalanceOrder=0, rebalanceBatchSize=524288, rebalance BatchesPrefetchCount=2, offHeapMaxMem=10000000, swapEnabled=false, maxConcurrentAsyncOps=500, writeBehindEnabled=false, writeBehindFlushSize=10240, writeBehindFlushFreq=5000, writeBehindFlushThreadCnt=1, writeBehindBatchSize=512, memMode=ON HEAP_TIERED, affMapper=null, rebalanceDelay=0, rebalanceThrottle=0, interceptor=null, longQryWarnTimeout=3000, readFromB ackup=true, nodeFilter=null, sqlSchema=null, sqlEscapeAll=false, sqlOnheapRowCacheSize=10240, snapshotableIdx=false, cpO nRead=true, topValidator=null] >>> ----------------------------------------------------------------------- [13:07:57,506][INFO][exchange-worker-#38%null%][GridCacheProcessor] Started cache [name=cache_d37d89a5_239_0920317220337 7125, mode=REPLICATED] >>> Node for ordered messages: 86d4d306 >>> Node for unordered messages: 86d4d306 >>>Empty cache, loading data by streamer ... Loaded 30000 key(s) >>> 1309 operation(s), op/sec: 261, cache size: 33840 >>> Latest value is Data [modified=0, longCol=14658, >>> doubleCol=0.3403571235222742, stringCol='14658*****', boolean=true, int=14658, bytesCol='[49, 52, 54, 53, 56, 42, 42, 42, 42, 42]', innerDataCol=inner [longCol=7329]', mapCol='{map_key_14 658=map_value_14658}'] >>> Operation counters: >>> replace=81,scanquery=86,put=444,remove=77,get=379,putall=36,removeall=44,sqlquery=81,msg=0,contq uery=0,getall=81 >>> 2596 operation(s), op/sec: 259, cache size: 36128 >>> Latest value is Data [modified=0, longCol=33912, >>> doubleCol=0.9348676568832568, stringCol='33912*****', boolean=true, int=33912, bytesCol='[51, 51, 57, 49, 50, 42, 42, 42, 42, 42]', innerDataCol=inner [longCol=16956]', mapCol='{map_key_3 3912=map_value_33912}'] >>> Operation counters: >>> replace=157,scanquery=166,put=856,remove=154,get=782,putall=69,removeall=90,sqlquery=170,msg=0,c ontquery=0,getall=152 >>> 4022 operation(s), op/sec: 268, cache size: 40428 >>> Latest value is Data [modified=0, longCol=31128, >>> doubleCol=0.33278377289395045, stringCol='31128*****', boolean=true , int=31128, bytesCol='[51, 49, 49, 50, 56, 42, 42, 42, 42, 42]', innerDataCol=inner [longCol=15564]', mapCol='{map_key_ 31128=map_value_31128}'] >>> Operation counters: >>> replace=238,scanquery=241,put=1314,remove=244,get=1255,putall=109,removeall=133,sqlquery=250,msg =0,contquery=0,getall=238 [13:08:25,463][INFO][grid-timeout-worker-#23%null%][IgniteKernal] Metrics for local node (to disable set 'metricsLogFrequency' to 0) ^-- Node [id=d37d89a5, name=null] ^-- H/N/C [hosts=1, nodes=2, CPUs=8] ^-- CPU [cur=0,07%, avg=3,76%, GC=0%] ^-- Heap [used=68MB, free=73,05%, comm=253MB] ^-- Public thread pool [active=0, idle=16, qSize=0] ^-- System thread pool [active=0, idle=16, qSize=0] ^-- Outbound messages queue [size=0] [13:09:17,128][INFO][sys-#15%null%][GridDeploymentLocalStore] Class locally deployed: class org.apache.ignite.IgniteChec kedException [13:09:17,133][INFO][sys-#15%null%][GridDeploymentLocalStore] Class locally deployed: class org.apache.ignite.IgniteExce ption [13:09:17,145][INFO][main][GridTcpRestProtocol] Command protocol successfully stopped: TCP binary [13:09:17,167][INFO][main][GridCacheProcessor] Stopped cache: ignite-marshaller-sys-cache [13:09:17,170][INFO][main][GridCacheProcessor] Stopped cache: ignite-sys-cache [13:09:17,182][INFO][main][GridCacheProcessor] Stopped cache: ignite-atomics-sys-cache [13:09:17,193][INFO][main][GridCacheProcessor] Stopped cache: cache_d37d89a5_239_09203172203377125 [13:09:17,207][INFO][main][GridDeploymentLocalStore] Removed undeployed class: GridDeployment [ts=1455098845210, depMode =SHARED, clsLdr=sun.misc.Launcher$AppClassLoader@1c8825a5, clsLdrId=4fa46aac251-d37d89a5-d829-4d99-8da4-6c257d577d95, us erVer=0, loc=true, sampleClsName=org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionF ullMap, pendingUndeploy=false, undeployed=true, usage=0] [13:09:17,252][INFO][main][IgniteKernal] >>> +---------------------------------------------------------------------------------+ >>> Ignite ver. 1.5.6#20160130-sha1:b146c0f5ac227e270d79c6bf18017676db6a1164 >>> stopped OK >>> +---------------------------------------------------------------------------------+ >>> Grid uptime: 00:01:51:799 Exception in thread "main" org.apache.ignite.cache.CachePartialUpdateException: Failed to update keys (retry update if p ossible).: [54103, 33040, 68549, 3279, 68546, 97582, 13984, 347, 50618, 17292, 34415, 74063, 33355, 34410, 75988, 84280, 85474, 30217, 34209, 8439, 75570, 96001, 6888, 14892, 78388, 75562, 7316, 82726, 84875, 40768, 91385, 97347, 70927, 496 2, 49508, 85821, 14246, 61801, 46378, 41204, 8907, 20978, 95389, 28009, 13963, 66695, 1407, 80195, 1203, 90020, 50836, 7 362, 8512, 34437, 32427, 88272, 21571, 53578, 58815, 2750, 13348, 13549, 36041, 48852, 48083, 73862, 35538, 73675, 1647, 35735, 18409, 9173, 14425, 47061, 60527, 43444, 27745, 21338, 14439, 86006, 96695, 23801, 63640, 70736, 68300, 56163, 4 9598, 11187, 32932, 94587, 66639, 93437, 24681, 31771, 41497, 52068, 42616, 81643, 39418, 31764, 90498, 88890, 19740, 82 246, 5739, 1575, 44544, 60614, 89745, 89741, 47726, 76518, 6602, 55724, 47513, 74433, 71287, 30603, 35099, 23117, 82409, 32194, 5774, 93766, 12572, 99699, 57076, 24534, 18053, 18269, 43563, 55519, 96495, 26315, 76338, 92923, 49898, 59708, 1 1402, 31459, 43745, 92379, 25040, 59715, 77229, 70251, 96058, 90845, 34372, 30812, 45462, 97505, 18291, 51625, 26766, 70 469, 71746, 12170, 50567, 17148, 48202, 4435, 23184, 17459, 16041, 36916, 71468, 5992, 49183, 18973, 8578, 22737, 61982, 8788, 61591, 97955, 5786, 45168, 10106, 20623, 3903, 45804, 84652, 45805, 3908, 90186, 63235, 35248, 84665, 95180, 6936 7, 26508, 18960, 46417, 64386, 90174, 31250, 7494, 46067, 21430, 23519, 64489, 57203, 17386, 43349, 17387, 28122, 169, 5 7213, 97166, 10623, 6367, 43166, 78079, 217, 22408, 55117, 12975, 43365, 47330, 42911, 48503, 71562, 19230, 43377, 91277 , 79740, 41909, 15254, 15901, 8843, 41716, 82805, 78890, 33832, 27830, 48745, 58043, 14122, 67095, 2234, 82832, 93133, 7 0116, 94267, 55348, 72298, 9459, 44964, 20106, 22463, 39566, 39565, 3532, 23702, 54030, 54720, 70170, 35848, 63741, 8964 2, 13235, 21693, 80664, 89643, 3314, 36194, 60456, 93367, 41340, 41724, 86156, 35612, 46290, 9299, 46960, 46013, 37875, 89435, 37879, 21465, 93191, 12558, 52710, 53845, 54967, 81795, 29630, 46936, 82628, 3175, 43804, 42960, 83659, 58706, 29 635, 79177, 49274, 6710, 43807, 60423, 33630, 80011, 42382, 43630, 99147, 4515, 68963, 49764, 92891, 54791, 49760, 56743 , 37580, 76606, 30056, 73059, 45095, 23903, 51793, 21230, 28333, 45083, 65534, 11355, 72659, 43871, 59822, 4556, 68126, 44637, 56104, 23226, 37900, 84584, 10436, 99786, 98730, 34033, 49755, 73294, 22196, 41301, 17095, 83439, 18139, 67805, 2 3240, 1003, 4304, 81014, 44472, 70357, 289, 94710, 45335, 54315, 64218, 49097, 90095, 50433, 84711, 51713, 45353, 44690, 84516, 88547, 37123, 56561, 26272, 8658, 91588, 88760, 5895, 88766, 51994, 20099, 15463, 78108, 50482, 43076, 96156, 71 30, 70322, 46530, 97409, 68193, 235, 98931, 95295, 5868, 67335, 63130, 86321, 28102, 84762, 1966, 95280, 11783, 76010] at org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1604) at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.cacheException(IgniteCacheProxy.java:1876) at org.apache.ignite.internal.processors.cache.IgniteCacheProxy.removeAll(IgniteCacheProxy.java:1323) at org.apache.ignite.testtools.Iron.main(Iron.java:297) Caused by: class org.apache.ignite.internal.processors.cache.CachePartialUpdateCheckedException: Failed to update keys ( retry update if possible).: [54103, 33040, 68549, 3279, 68546, 97582, 13984, 347, 50618, 17292, 34415, 74063, 33355, 344 10, 75988, 84280, 85474, 30217, 34209, 8439, 75570, 96001, 6888, 14892, 78388, 75562, 7316, 82726, 84875, 40768, 91385, 97347, 70927, 4962, 49508, 85821, 14246, 61801, 46378, 41204, 8907, 20978, 95389, 28009, 13963, 66695, 1407, 80195, 1203 , 90020, 50836, 7362, 8512, 34437, 32427, 88272, 21571, 53578, 58815, 2750, 13348, 13549, 36041, 48852, 48083, 73862, 35 538, 73675, 1647, 35735, 18409, 9173, 14425, 47061, 60527, 43444, 27745, 21338, 14439, 86006, 96695, 23801, 63640, 70736 , 68300, 56163, 49598, 11187, 32932, 94587, 66639, 93437, 24681, 31771, 41497, 52068, 42616, 81643, 39418, 31764, 90498, 88890, 19740, 82246, 5739, 1575, 44544, 60614, 89745, 89741, 47726, 76518, 6602, 55724, 47513, 74433, 71287, 30603, 350 99, 23117, 82409, 32194, 5774, 93766, 12572, 99699, 57076, 24534, 18053, 18269, 43563, 55519, 96495, 26315, 76338, 92923 , 49898, 59708, 11402, 31459, 43745, 92379, 25040, 59715, 77229, 70251, 96058, 90845, 34372, 30812, 45462, 97505, 18291, 51625, 26766, 70469, 71746, 12170, 50567, 17148, 48202, 4435, 23184, 17459, 16041, 36916, 71468, 5992, 49183, 18973, 85 78, 22737, 61982, 8788, 61591, 97955, 5786, 45168, 10106, 20623, 3903, 45804, 84652, 45805, 3908, 90186, 63235, 35248, 8 4665, 95180, 69367, 26508, 18960, 46417, 64386, 90174, 31250, 7494, 46067, 21430, 23519, 64489, 57203, 17386, 43349, 173 87, 28122, 169, 57213, 97166, 10623, 6367, 43166, 78079, 217, 22408, 55117, 12975, 43365, 47330, 42911, 48503, 71562, 19 230, 43377, 91277, 79740, 41909, 15254, 15901, 8843, 41716, 82805, 78890, 33832, 27830, 48745, 58043, 14122, 67095, 2234 , 82832, 93133, 70116, 94267, 55348, 72298, 9459, 44964, 20106, 22463, 39566, 39565, 3532, 23702, 54030, 54720, 70170, 3 5848, 63741, 89642, 13235, 21693, 80664, 89643, 3314, 36194, 60456, 93367, 41340, 41724, 86156, 35612, 46290, 9299, 4696 0, 46013, 37875, 89435, 37879, 21465, 93191, 12558, 52710, 53845, 54967, 81795, 29630, 46936, 82628, 3175, 43804, 42960, 83659, 58706, 29635, 79177, 49274, 6710, 43807, 60423, 33630, 80011, 42382, 43630, 99147, 4515, 68963, 49764, 92891, 54 791, 49760, 56743, 37580, 76606, 30056, 73059, 45095, 23903, 51793, 21230, 28333, 45083, 65534, 11355, 72659, 43871, 598 22, 4556, 68126, 44637, 56104, 23226, 37900, 84584, 10436, 99786, 98730, 34033, 49755, 73294, 22196, 41301, 17095, 83439 , 18139, 67805, 23240, 1003, 4304, 81014, 44472, 70357, 289, 94710, 45335, 54315, 64218, 49097, 90095, 50433, 84711, 517 13, 45353, 44690, 84516, 88547, 37123, 56561, 26272, 8658, 91588, 88760, 5895, 88766, 51994, 20099, 15463, 78108, 50482, 43076, 96156, 7130, 70322, 46530, 97409, 68193, 235, 98931, 95295, 5868, 67335, 63130, 86321, 28102, 84762, 1966, 95280 , 11783, 76010] at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateFuture$UpdateState.add FailedKeys(GridNearAtomicUpdateFuture.java:1241) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateFuture$UpdateState.onR esult(GridNearAtomicUpdateFuture.java:661) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateFuture.onResult(GridNe arAtomicUpdateFuture.java:344) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdate Response(GridDhtAtomicCache.java:2709) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$700(GridDhtAtomi cCache.java:128) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$6.apply(GridDhtAtomicCa che.java:263) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$6.apply(GridDhtAtomicCa che.java:261) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:582) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:280) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:204) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:80) at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:163) at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:821 ) at org.apache.ignite.internal.managers.communication.GridIoManager.access$1600(GridIoManager.java:103) at org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:784) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Suppressed: class org.apache.ignite.IgniteCheckedException: Failed to update keys on primary node. at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridNearAtomicUpdateResponse.addFa iledKeys(GridNearAtomicUpdateResponse.java:368) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncI nternal0(GridDhtAtomicCache.java:1473) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncI nternal(GridDhtAtomicCache.java:1282) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtom icUpdateRequest(GridDhtAtomicCache.java:2692) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(Grid DhtAtomicCache.java:128) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDht AtomicCache.java:257) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDht AtomicCache.java:255) ... 11 more Suppressed: class org.apache.ignite.IgniteException: Failed to get value for key: 352. This can happen d ue to a long GC pause. at org.apache.ignite.internal.processors.query.h2.opt.GridH2AbstractKeyValueRow.getValue(GridH2A bstractKeyValueRow.java:253) at org.apache.ignite.internal.processors.query.h2.opt.GridH2AbstractKeyValueRow.getValue(GridH2A bstractKeyValueRow.java:284) at org.apache.ignite.internal.processors.query.h2.opt.GridH2IndexBase.compareRows(GridH2IndexBas e.java:119) at org.apache.ignite.internal.processors.query.h2.opt.GridH2TreeIndex.compare(GridH2TreeIndex.ja va:248) at org.apache.ignite.internal.processors.query.h2.opt.GridH2TreeIndex.compare(GridH2TreeIndex.ja va:49) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap$2.compareTo(GridOffHeap SnapTreeMap.java:1350) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap$2.compareTo(GridOffHeap SnapTreeMap.java:1346) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHe apSnapTreeMap.java:2102) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHe apSnapTreeMap.java:2217) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHe apSnapTreeMap.java:2217) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHe apSnapTreeMap.java:2217) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHe apSnapTreeMap.java:2217) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHe apSnapTreeMap.java:2217) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHe apSnapTreeMap.java:2217) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHe apSnapTreeMap.java:2217) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHe apSnapTreeMap.java:2217) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHe apSnapTreeMap.java:2217) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHe apSnapTreeMap.java:2217) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.updateUnderRoot(GridOff HeapSnapTreeMap.java:2034) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.update(GridOffHeapSnapT reeMap.java:1915) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.remove(GridOffHeapSnapT reeMap.java:1884) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.remove(GridOffHeapSnapT reeMap.java:108) at org.apache.ignite.internal.processors.query.h2.opt.GridH2TreeIndex.remove(GridH2TreeIndex.jav a:408) at org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.doUpdate(GridH2Table.java:434) at org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.update(GridH2Table.java:340) at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.remove(IgniteH2Indexing.java: 586) at org.apache.ignite.internal.processors.query.GridQueryProcessor.remove(GridQueryProcessor.java :1014) at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.remove(GridCacheQuery Manager.java:431) at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.clearIndex(GridCacheMapEntry.ja va:3760) at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerUpdate(GridCacheMapEntry.j ava:2334) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateS ingle(GridDhtAtomicCache.java:1997) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateA llAsyncInternal0(GridDhtAtomicCache.java:1407) ... 16 more {noformat} Server log: {noformat} [13:09:16,846][SEVERE][sys-#17%null%][GridDhtAtomicCache] <cache_d37d89a5_239_09203172203377125> Unexpected exception du ring cache update class org.apache.ignite.IgniteException: Failed to get value for key: 352. This can happen due to a long GC pause. at org.apache.ignite.internal.processors.query.h2.opt.GridH2AbstractKeyValueRow.getValue(GridH2AbstractKeyValueR ow.java:253) at org.apache.ignite.internal.processors.query.h2.opt.GridH2AbstractKeyValueRow.getValue(GridH2AbstractKeyValueR ow.java:284) at org.apache.ignite.internal.processors.query.h2.opt.GridH2IndexBase.compareRows(GridH2IndexBase.java:119) at org.apache.ignite.internal.processors.query.h2.opt.GridH2TreeIndex.compare(GridH2TreeIndex.java:248) at org.apache.ignite.internal.processors.query.h2.opt.GridH2TreeIndex.compare(GridH2TreeIndex.java:49) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap$2.compareTo(GridOffHeapSnapTreeMap.java :1350) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap$2.compareTo(GridOffHeapSnapTreeMap.java :1346) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja va:2102) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja va:2217) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja va:2217) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja va:2217) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja va:2217) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja va:2217) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja va:2217) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja va:2217) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja va:2217) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja va:2217) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja va:2217) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.updateUnderRoot(GridOffHeapSnapTreeMap. java:2034) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.update(GridOffHeapSnapTreeMap.java:1915 ) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.remove(GridOffHeapSnapTreeMap.java:1884 ) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.remove(GridOffHeapSnapTreeMap.java:108) at org.apache.ignite.internal.processors.query.h2.opt.GridH2TreeIndex.remove(GridH2TreeIndex.java:408) at org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.doUpdate(GridH2Table.java:434) at org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.update(GridH2Table.java:340) at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.remove(IgniteH2Indexing.java:586) at org.apache.ignite.internal.processors.query.GridQueryProcessor.remove(GridQueryProcessor.java:1014) at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.remove(GridCacheQueryManager.java:431 ) at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.clearIndex(GridCacheMapEntry.java:3760) at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.innerUpdate(GridCacheMapEntry.java:2334) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateSingle(GridDhtAto micCache.java:1997) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal0 (GridDhtAtomicCache.java:1407) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.updateAllAsyncInternal( GridDhtAtomicCache.java:1282) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.processNearAtomicUpdate Request(GridDhtAtomicCache.java:2692) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache.access$600(GridDhtAtomi cCache.java:128) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCa che.java:257) at org.apache.ignite.internal.processors.cache.distributed.dht.atomic.GridDhtAtomicCache$5.apply(GridDhtAtomicCa che.java:255) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.processMessage(GridCacheIoManager.java:582) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.onMessage0(GridCacheIoManager.java:280) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.handleMessage(GridCacheIoManager.java:204) at org.apache.ignite.internal.processors.cache.GridCacheIoManager.access$000(GridCacheIoManager.java:80) at org.apache.ignite.internal.processors.cache.GridCacheIoManager$1.onMessage(GridCacheIoManager.java:163) at org.apache.ignite.internal.managers.communication.GridIoManager.processRegularMessage0(GridIoManager.java:821 ) at org.apache.ignite.internal.managers.communication.GridIoManager.access$1600(GridIoManager.java:103) at org.apache.ignite.internal.managers.communication.GridIoManager$5.run(GridIoManager.java:784) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) [13:09:17,158][INFO][disco-event-worker-#46%null%][GridDiscoveryManager] Node left topology: TcpDiscoveryNode [id=d37d89 a5-d829-4d99-8da4-6c257d577d95, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 192.168.1.112, 2001:0:9d38:6ab8:2c8e:661e:a1b7:c399], sockAddrs=[work-pc/192.168.1.112:0, /0:0:0:0:0:0:0:1:0, work-pc/192.168.1.112:0, /127.0.0.1:0, /192.168.1.112:0, /2001: 0:9d38:6ab8:2c8e:661e:a1b7:c399:0], discPort=0, order=2, intOrder=2, lastExchangeTime=1455098845119, loc=false, ver=1.5. 6#20160130-sha1:b146c0f5, isClient=true] [13:09:17,182][INFO][disco-event-worker-#46%null%][GridDiscoveryManager] Topology snapshot [ver=3, servers=1, clients=0, CPUs=8, heap=2.0GB] [13:09:17,193][INFO][disco-event-worker-#46%null%][GridDeploymentPerVersionStore] Class was undeployed in SHARED or CONT INUOUS mode [cls=class org.apache.ignite.testtools.Iron$Data, alias=org.apache.ignite.testtools.Iron$Data] [13:09:17,207][INFO][disco-event-worker-#46%null%][GridDeploymentPerVersionStore] Class was undeployed in SHARED or CONT INUOUS mode [cls=class org.apache.ignite.testtools.Iron$3, alias=org.apache.ignite.testtools.Iron$3] [13:09:17,223][INFO][disco-event-worker-#46%null%][GridDeploymentPerVersionStore] Class was undeployed in SHARED or CONT INUOUS mode [cls=class org.apache.ignite.testtools.Iron$2, alias=org.apache.ignite.testtools.Iron$2] [13:09:17,244][INFO][disco-event-worker-#46%null%][GridDeploymentPerVersionStore] Class was undeployed in SHARED or CONT INUOUS mode [cls=class org.apache.ignite.testtools.Iron$StringBinaryObjectIgniteBiPredicate, alias=org.apache.ignite.tes ttools.Iron$StringBinaryObjectIgniteBiPredicate] [13:09:57,574][INFO][grid-timeout-worker-#33%null%][IgniteKernal] Metrics for local node (to disable set 'metricsLogFrequency' to 0) ^-- Node [id=86d4d306, name=null] ^-- H/N/C [hosts=1, nodes=1, CPUs=8] ^-- CPU [cur=0%, avg=4,06%, GC=0%] ^-- Heap [used=350MB, free=82,48%, comm=2001MB] ^-- Public thread pool [active=0, idle=16, qSize=0] ^-- System thread pool [active=0, idle=16, qSize=0] ^-- Outbound messages queue [size=0] [13:10:18,266][SEVERE][exchange-worker-#49%null%][GridDhtPartitionsExchangeFuture] Failed to reinitialize local partitio ns (preloading will be stopped): GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], n odeId=d37d89a5, evt=NODE_LEFT] class org.apache.ignite.IgniteException: Failed to get value for key: 352. This can happen due to a long GC pause. at org.apache.ignite.internal.processors.query.h2.opt.GridH2AbstractKeyValueRow.getValue(GridH2AbstractKeyValueR ow.java:253) at org.apache.ignite.internal.processors.query.h2.opt.GridH2AbstractKeyValueRow.getValue(GridH2AbstractKeyValueR ow.java:284) at org.apache.ignite.internal.processors.query.h2.opt.GridH2IndexBase.compareRows(GridH2IndexBase.java:119) at org.apache.ignite.internal.processors.query.h2.opt.GridH2TreeIndex.compare(GridH2TreeIndex.java:248) at org.apache.ignite.internal.processors.query.h2.opt.GridH2TreeIndex.compare(GridH2TreeIndex.java:49) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap$2.compareTo(GridOffHeapSnapTreeMap.java :1350) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap$2.compareTo(GridOffHeapSnapTreeMap.java :1346) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja va:2102) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja va:2217) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja va:2217) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja va:2217) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja va:2217) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja va:2217) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja va:2217) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja va:2217) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja va:2217) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja va:2217) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja va:2217) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.updateUnderRoot(GridOffHeapSnapTreeMap. java:2034) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.update(GridOffHeapSnapTreeMap.java:1915 ) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.remove(GridOffHeapSnapTreeMap.java:1884 ) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.remove(GridOffHeapSnapTreeMap.java:108) at org.apache.ignite.internal.processors.query.h2.opt.GridH2TreeIndex.remove(GridH2TreeIndex.java:408) at org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.doUpdate(GridH2Table.java:434) at org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.update(GridH2Table.java:340) at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.remove(IgniteH2Indexing.java:586) at org.apache.ignite.internal.processors.query.GridQueryProcessor.remove(GridQueryProcessor.java:1014) at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.remove(GridCacheQueryManager.java:431 ) at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.clearIndex(GridCacheMapEntry.java:3760) at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.clear(GridCacheMapEntry.java:2615) at org.apache.ignite.internal.processors.cache.GridCacheAdapter.clearLocally(GridCacheAdapter.java:1247) at org.apache.ignite.internal.processors.cache.GridCacheDeploymentManager.onUndeploy0(GridCacheDeploymentManager .java:265) at org.apache.ignite.internal.processors.cache.GridCacheDeploymentManager.access$300(GridCacheDeploymentManager. java:66) at org.apache.ignite.internal.processors.cache.GridCacheDeploymentManager$3.apply(GridCacheDeploymentManager.jav a:237) at org.apache.ignite.internal.processors.cache.GridCacheDeploymentManager.unwind(GridCacheDeploymentManager.java :208) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPreloader.unwindUndeploys(GridDh tPreloader.java:748) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(Gr idDhtPartitionsExchangeFuture.java:574) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePa rtitionExchangeManager.java:1297) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:745) [13:10:18,465][SEVERE][exchange-worker-#49%null%][GridCachePartitionExchangeManager] Failed to wait for completion of pa rtition map exchange (preloading will not start): GridDhtPartitionsExchangeFuture [dummy=false, forcePreload=false, reas sign=false, discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=d37d89a5-d829-4d99-8da4-6c257d577d95, addrs=[0:0:0:0:0 :0:0:1, 127.0.0.1, 192.168.1.112, 2001:0:9d38:6ab8:2c8e:661e:a1b7:c399], sockAddrs=[work-pc/192.168.1.112:0, /0:0:0:0:0: 0:0:1:0, work-pc/192.168.1.112:0, /127.0.0.1:0, /192.168.1.112:0, /2001:0:9d38:6ab8:2c8e:661e:a1b7:c399:0], discPort=0, order=2, intOrder=2, lastExchangeTime=1455098845119, loc=false, ver=1.5.6#20160130-sha1:b146c0f5, isClient=true], topVer =3, nodeId8=86d4d306, msg=Node left: TcpDiscoveryNode [id=d37d89a5-d829-4d99-8da4-6c257d577d95, addrs=[0:0:0:0:0:0:0:1, 127.0.0.1, 192.168.1.112, 2001:0:9d38:6ab8:2c8e:661e:a1b7:c399], sockAddrs=[work-pc/192.168.1.112:0, /0:0:0:0:0:0:0:1:0, work-pc/192.168.1.112:0, /127.0.0.1:0, /192.168.1.112:0, /2001:0:9d38:6ab8:2c8e:661e:a1b7:c399:0], discPort=0, order=2, intOrder=2, lastExchangeTime=1455098845119, loc=false, ver=1.5.6#20160130-sha1:b146c0f5, isClient=true], type=NODE_LEFT , tstamp=1455098957188], rcvdIds=GridConcurrentHashSet [elements=[]], rmtIds=null, exchId=GridDhtPartitionExchangeId [to pVer=AffinityTopologyVersion [topVer=3, minorTopVer=0], nodeId=d37d89a5, evt=NODE_LEFT], init=true, ready=false, replied =false, added=true, initFut=GridFutureAdapter [resFlag=2, res=false, startTime=1455098957260, endTime=1455099018457, ign oreInterrupts=false, lsnr=null, state=DONE], topSnapshot=null, lastVer=null, partReleaseFut=null, skipPreload=false, cli entOnlyExchange=false, initTs=1455098957260, oldest=86d4d306-0be1-4cc6-9134-c39556225a56, oldestOrder=1, evtLatch=0, rem aining=[], super=GridFutureAdapter [resFlag=1, res=class o.a.i.IgniteException: Failed to get value for key: 352. This c an happen due to a long GC pause., startTime=1455098957260, endTime=1455099018457, ignoreInterrupts=false, lsnr=null, st ate=DONE]] class org.apache.ignite.IgniteCheckedException: Failed to get value for key: 352. This can happen due to a long GC pause . at org.apache.ignite.internal.util.IgniteUtils.cast(IgniteUtils.java:7005) at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:166) at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:115) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePa rtitionExchangeManager.java:1299) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:110) at java.lang.Thread.run(Thread.java:745) Caused by: class org.apache.ignite.IgniteException: Failed to get value for key: 352. This can happen due to a long GC p ause. at org.apache.ignite.internal.processors.query.h2.opt.GridH2AbstractKeyValueRow.getValue(GridH2AbstractKeyValueR ow.java:253) at org.apache.ignite.internal.processors.query.h2.opt.GridH2AbstractKeyValueRow.getValue(GridH2AbstractKeyValueR ow.java:284) at org.apache.ignite.internal.processors.query.h2.opt.GridH2IndexBase.compareRows(GridH2IndexBase.java:119) at org.apache.ignite.internal.processors.query.h2.opt.GridH2TreeIndex.compare(GridH2TreeIndex.java:248) at org.apache.ignite.internal.processors.query.h2.opt.GridH2TreeIndex.compare(GridH2TreeIndex.java:49) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap$2.compareTo(GridOffHeapSnapTreeMap.java :1350) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap$2.compareTo(GridOffHeapSnapTreeMap.java :1346) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja va:2102) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja va:2217) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja va:2217) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja va:2217) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja va:2217) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja va:2217) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja va:2217) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja va:2217) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja va:2217) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja va:2217) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.attemptUpdate(GridOffHeapSnapTreeMap.ja va:2217) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.updateUnderRoot(GridOffHeapSnapTreeMap. java:2034) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.update(GridOffHeapSnapTreeMap.java:1915 ) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.remove(GridOffHeapSnapTreeMap.java:1884 ) at org.apache.ignite.internal.util.offheap.unsafe.GridOffHeapSnapTreeMap.remove(GridOffHeapSnapTreeMap.java:108) at org.apache.ignite.internal.processors.query.h2.opt.GridH2TreeIndex.remove(GridH2TreeIndex.java:408) at org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.doUpdate(GridH2Table.java:434) at org.apache.ignite.internal.processors.query.h2.opt.GridH2Table.update(GridH2Table.java:340) at org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.remove(IgniteH2Indexing.java:586) at org.apache.ignite.internal.processors.query.GridQueryProcessor.remove(GridQueryProcessor.java:1014) at org.apache.ignite.internal.processors.cache.query.GridCacheQueryManager.remove(GridCacheQueryManager.java:431 ) at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.clearIndex(GridCacheMapEntry.java:3760) at org.apache.ignite.internal.processors.cache.GridCacheMapEntry.clear(GridCacheMapEntry.java:2615) at org.apache.ignite.internal.processors.cache.GridCacheAdapter.clearLocally(GridCacheAdapter.java:1247) at org.apache.ignite.internal.processors.cache.GridCacheDeploymentManager.onUndeploy0(GridCacheDeploymentManager .java:265) at org.apache.ignite.internal.processors.cache.GridCacheDeploymentManager.access$300(GridCacheDeploymentManager. java:66) at org.apache.ignite.internal.processors.cache.GridCacheDeploymentManager$3.apply(GridCacheDeploymentManager.jav a:237) at org.apache.ignite.internal.processors.cache.GridCacheDeploymentManager.unwind(GridCacheDeploymentManager.java :208) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPreloader.unwindUndeploys(GridDh tPreloader.java:748) at org.apache.ignite.internal.processors.cache.distributed.dht.preloader.GridDhtPartitionsExchangeFuture.init(Gr idDhtPartitionsExchangeFuture.java:574) at org.apache.ignite.internal.processors.cache.GridCachePartitionExchangeManager$ExchangeWorker.body(GridCachePa rtitionExchangeManager.java:1297) ... 2 more {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)