[
https://issues.apache.org/jira/browse/IGNITE-732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14558163#comment-14558163
]
Ilya Suntsov edited comment on IGNITE-732 at 5/25/15 12:22 PM:
---------------------------------------------------------------
Steps for reproduction:
1. Start 3 nodes with ExampleNodeStartup
2. Start StreamWords
3. Start QueryWords
I observed the following behavior:
{noformat}
[14:26:26] __________ ________________
[14:26:26] / _/ ___/ |/ / _/_ __/ __/
[14:26:26] _/ // (7 7 // / / / / _/
[14:26:26] /___/\___/_/|_/___/ /_/ /___/
[14:26:26]
[14:26:26] ver. 1.0.7-QAIS2501#20150522-sha1:050f4292
[14:26:26] 2015 Copyright(C) Apache Software Foundation
[14:26:26]
[14:26:26] Quiet mode.
[14:26:26] ^-- Logging to file
'/Users/gridgain/Downloads/gridgain-enterprise-fabric-7.0.7-QAIS2501/work/log/ignite-364df547.log'
[14:26:26] ^-- To see **FULL** console log here add -DIGNITE_QUIET=false or
"-v" to ignite.{sh|bat}
[14:26:26]
[14:26:26] Initial heap size is 192MB (should be no less than 512MB, use
-Xms512m -Xmx512m).
[14:26:26] Configured plugins:
[14:26:26] ^-- GridGain 7.0.7-QAIS2501#20150525-sha1:e7f977ac
[14:26:26] ^-- 2015 Copyright(C) GridGain Systems
[14:26:26]
[14:26:30] Attempting to start more nodes than physical RAM available on
current host (this can cause significant slowdown)
[14:26:32,975][ERROR][main][GridEntLicenseProcessor] License bursting ==>
Maximum number of nodes (2) is exceeded.
[14:26:32,975][ERROR][main][GridEntLicenseProcessor] ^-- Contact
sa...@gridgain.com for further assistance. Make sure to include your license
ID: 14177F1A-5B5A-4F9C-BC55-F7335715F311
[14:26:32,976][ERROR][main][GridEntLicenseProcessor] License grace/burst period
- left 1 hour.
[14:26:32] Performance suggestions for grid (fix if possible)
[14:26:32] To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
[14:26:32] ^-- Disable peer class loading (set 'peerClassLoadingEnabled' to
false)
[14:26:32] ^-- Disable grid events (remove 'includeEventTypes' from
configuration)
[14:26:32]
[14:26:32] To start Console Management & Monitoring run ignitevisorcmd.{sh|bat}
[14:26:32]
[14:26:32] Ignite node started OK (id=364df547)
[14:26:33] Topology snapshot [ver=5, nodes=5, CPUs=8, heap=13.0GB]
[14:26:36] New version is available at
http://www.gridgain.com/download/editions/#enterprise-edition: 7.0.5
Query results [avg=4.81, min=1, max=1730]
(in,673)
(it,585)
(to,526)
(a,431)
(I,422)
(that,371)
(she,337)
(said,288)
(or,246)
(for,245)
Query results [avg=4.50, min=1, max=429]
(the,2230)
(and,1065)
(of,831)
(to,796)
(a,662)
(she,535)
(said,424)
(was,347)
(Alice,276)
(you,273)
Query results [avg=2.95, min=1, max=266]
...
(the,3219)
(to,1681)
(and,1549)
(a,1474)
(of,1248)
(she,1047)
(said,959)
(in,876)
(it,801)
(was,714)
[14:27:22] Topology snapshot [ver=6, nodes=4, CPUs=8, heap=11.0GB]
[14:27:22] Ignite node stopped OK [uptime=00:00:49:513]
Exception in thread "main" javax.cache.CacheException: Failed to run map query
remotely.
at
org.apache.ignite.internal.processors.query.h2.twostep.GridReduceQueryExecutor.query(GridReduceQueryExecutor.java:343)
at
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.queryTwoStep(IgniteH2Indexing.java:775)
at
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.queryTwoStep(IgniteH2Indexing.java:866)
at
org.apache.ignite.internal.processors.query.GridQueryProcessor.queryTwoStep(GridQueryProcessor.java:561)
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxy.query(IgniteCacheProxy.java:513)
at
org.apache.ignite.examples.streaming.wordcount.QueryWords.main(QueryWords.java:67)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)
Caused by: javax.cache.CacheException: Failed to execute map query on the node:
30a59fd5-ac86-46a6-9c01-1307260a0e6c, Node left the topology.
at
org.apache.ignite.internal.processors.query.h2.twostep.GridReduceQueryExecutor.fail(GridReduceQueryExecutor.java:204)
at
org.apache.ignite.internal.processors.query.h2.twostep.GridReduceQueryExecutor.access$300(GridReduceQueryExecutor.java:61)
at
org.apache.ignite.internal.processors.query.h2.twostep.GridReduceQueryExecutor$2.onEvent(GridReduceQueryExecutor.java:147)
at
org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:745)
at
org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:730)
at
org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.record(GridEventStorageManager.java:270)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.recordEvent(GridDiscoveryManager.java:1597)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body0(GridDiscoveryManager.java:1787)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body(GridDiscoveryManager.java:1636)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:108)
at java.lang.Thread.run(Thread.java:745)
{noformat}
Is it correct or not?
was (Author: ustas):
Steps for reproduction:
1. Start 3 nodes with ExampleNodeStartup
2. Start StreamWords
3. Start QueryWords
I observed the following behavior:
{noformat}
[14:26:26] __________ ________________
[14:26:26] / _/ ___/ |/ / _/_ __/ __/
[14:26:26] _/ // (7 7 // / / / / _/
[14:26:26] /___/\___/_/|_/___/ /_/ /___/
[14:26:26]
[14:26:26] ver. 1.0.7-QAIS2501#20150522-sha1:050f4292
[14:26:26] 2015 Copyright(C) Apache Software Foundation
[14:26:26]
[14:26:26] Quiet mode.
[14:26:26] ^-- Logging to file
'/Users/gridgain/Downloads/gridgain-enterprise-fabric-7.0.7-QAIS2501/work/log/ignite-364df547.log'
[14:26:26] ^-- To see **FULL** console log here add -DIGNITE_QUIET=false or
"-v" to ignite.{sh|bat}
[14:26:26]
[14:26:26] Initial heap size is 192MB (should be no less than 512MB, use
-Xms512m -Xmx512m).
[14:26:26] Configured plugins:
[14:26:26] ^-- GridGain 7.0.7-QAIS2501#20150525-sha1:e7f977ac
[14:26:26] ^-- 2015 Copyright(C) GridGain Systems
[14:26:26]
[14:26:30] Attempting to start more nodes than physical RAM available on
current host (this can cause significant slowdown)
[14:26:32,975][ERROR][main][GridEntLicenseProcessor] License bursting ==>
Maximum number of nodes (2) is exceeded.
[14:26:32,975][ERROR][main][GridEntLicenseProcessor] ^-- Contact
sa...@gridgain.com for further assistance. Make sure to include your license
ID: 14177F1A-5B5A-4F9C-BC55-F7335715F311
[14:26:32,976][ERROR][main][GridEntLicenseProcessor] License grace/burst period
- left 1 hour.
[14:26:32] Performance suggestions for grid (fix if possible)
[14:26:32] To disable, set -DIGNITE_PERFORMANCE_SUGGESTIONS_DISABLED=true
[14:26:32] ^-- Disable peer class loading (set 'peerClassLoadingEnabled' to
false)
[14:26:32] ^-- Disable grid events (remove 'includeEventTypes' from
configuration)
[14:26:32]
[14:26:32] To start Console Management & Monitoring run ignitevisorcmd.{sh|bat}
[14:26:32]
[14:26:32] Ignite node started OK (id=364df547)
[14:26:33] Topology snapshot [ver=5, nodes=5, CPUs=8, heap=13.0GB]
[14:26:36] New version is available at
http://www.gridgain.com/download/editions/#enterprise-edition: 7.0.5
Query results [avg=4.81, min=1, max=1730]
(in,673)
(it,585)
(to,526)
(a,431)
(I,422)
(that,371)
(she,337)
(said,288)
(or,246)
(for,245)
Query results [avg=4.50, min=1, max=429]
(the,2230)
(and,1065)
(of,831)
(to,796)
(a,662)
(she,535)
(said,424)
(was,347)
(Alice,276)
(you,273)
Query results [avg=2.95, min=1, max=266]
...
(the,3219)
(to,1681)
(and,1549)
(a,1474)
(of,1248)
(she,1047)
(said,959)
(in,876)
(it,801)
(was,714)
[14:27:22] Topology snapshot [ver=6, nodes=4, CPUs=8, heap=11.0GB]
[14:27:22] Ignite node stopped OK [uptime=00:00:49:513]
Exception in thread "main" javax.cache.CacheException: Failed to run map query
remotely.
at
org.apache.ignite.internal.processors.query.h2.twostep.GridReduceQueryExecutor.query(GridReduceQueryExecutor.java:343)
at
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.queryTwoStep(IgniteH2Indexing.java:775)
at
org.apache.ignite.internal.processors.query.h2.IgniteH2Indexing.queryTwoStep(IgniteH2Indexing.java:866)
at
org.apache.ignite.internal.processors.query.GridQueryProcessor.queryTwoStep(GridQueryProcessor.java:561)
at
org.apache.ignite.internal.processors.cache.IgniteCacheProxy.query(IgniteCacheProxy.java:513)
at
org.apache.ignite.examples.streaming.wordcount.QueryWords.main(QueryWords.java:67)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)
Caused by: javax.cache.CacheException: Failed to execute map query on the node:
30a59fd5-ac86-46a6-9c01-1307260a0e6c, Node left the topology.
at
org.apache.ignite.internal.processors.query.h2.twostep.GridReduceQueryExecutor.fail(GridReduceQueryExecutor.java:204)
at
org.apache.ignite.internal.processors.query.h2.twostep.GridReduceQueryExecutor.access$300(GridReduceQueryExecutor.java:61)
at
org.apache.ignite.internal.processors.query.h2.twostep.GridReduceQueryExecutor$2.onEvent(GridReduceQueryExecutor.java:147)
at
org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:745)
at
org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:730)
at
org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.record(GridEventStorageManager.java:270)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.recordEvent(GridDiscoveryManager.java:1597)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body0(GridDiscoveryManager.java:1787)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body(GridDiscoveryManager.java:1636)
at
org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:108)
at java.lang.Thread.run(Thread.java:745)
{noformat}
Is it correct or not?
> IgniteCache.size() should not fail in case of topology changes.
> ---------------------------------------------------------------
>
> Key: IGNITE-732
> URL: https://issues.apache.org/jira/browse/IGNITE-732
> Project: Ignite
> Issue Type: Bug
> Components: cache
> Reporter: Dmitriy Setrakyan
> Assignee: Yakov Zhdanov
> Priority: Blocker
> Fix For: sprint-5
>
>
> I believe that cache.size() method fails if topology changes during the
> execution. I got the following error when executing word-count {{QueryWords}}
> class and removing one of the server nodes.
> {code}
> Query results [total=1003726, avg=4.34, min=1, max=1314]
> (the,20)
> (and,9)
> (a,8)
> (in,8)
> (of,8)
> (that,5)
> (said,4)
> (on,4)
> (for,4)
> (to,4)
> [17:35:54] Topology snapshot [ver=5, nodes=3, CPUs=4, heap=4.0GB]
> [17:35:54,772][ERROR][disco-event-worker-#46%null%][GridTaskWorker] Failed to
> obtain remote job result policy for result from ComputeTask.result(..) method
> (will fail the whole task): GridJobResultImpl [job=C1 [arg=null],
> sib=GridJobSiblingImpl
> [sesId=9a42d9fac41-8437ac61-27ef-4e4b-951d-a29b18346bfe,
> jobId=ba42d9fac41-833cae7a-295f-4578-a6b0-f4b6d3a1bf50,
> nodeId=833cae7a-295f-4578-a6b0-f4b6d3a1bf50, isJobDone=false],
> jobCtx=GridJobContextImpl
> [jobId=ba42d9fac41-833cae7a-295f-4578-a6b0-f4b6d3a1bf50, attrs={}],
> node=TcpDiscoveryNode [id=833cae7a-295f-4578-a6b0-f4b6d3a1bf50,
> addrs=[192.168.2.10, 0:0:0:0:0:0:0:1, 127.0.0.1],
> sockAddrs=[/192.168.2.10:47500, /0:0:0:0:0:0:0:1:47500, /127.0.0.1:47500],
> discPort=47500, order=1, intOrder=1, loc=false,
> ver=1.0.0#19691231-sha1:00000000, isClient=false], ex=class
> o.a.i.cluster.ClusterTopologyException: Node has left grid:
> 833cae7a-295f-4578-a6b0-f4b6d3a1bf50, hasRes=true, isCancelled=false,
> isOccupied=true]
> class org.apache.ignite.cluster.ClusterTopologyException: Node has left grid:
> 833cae7a-295f-4578-a6b0-f4b6d3a1bf50
> at
> org.apache.ignite.internal.processors.task.GridTaskWorker.onNodeLeft(GridTaskWorker.java:1204)
> at
> org.apache.ignite.internal.processors.task.GridTaskProcessor$TaskDiscoveryListener.onEvent(GridTaskProcessor.java:1180)
> at
> org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:739)
> at
> org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:724)
> at
> org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.record(GridEventStorageManager.java:272)
> at
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.recordEvent(GridDiscoveryManager.java:1533)
> at
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body0(GridDiscoveryManager.java:1720)
> at
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body(GridDiscoveryManager.java:1572)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:108)
> at java.lang.Thread.run(Thread.java:745)
> [17:35:55] Ignite node stopped OK [uptime=02:45:01:791]
> Exception in thread "main" javax.cache.CacheException: class
> org.apache.ignite.IgniteCheckedException: Node has left grid:
> 833cae7a-295f-4578-a6b0-f4b6d3a1bf50
> at
> org.apache.ignite.internal.processors.cache.GridCacheUtils.convertToCacheException(GridCacheUtils.java:1808)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheProxy.cacheException(IgniteCacheProxy.java:1463)
> at
> org.apache.ignite.internal.processors.cache.IgniteCacheProxy.size(IgniteCacheProxy.java:594)
> at
> org.apache.ignite.examples.streaming.wordcount.QueryWords.main(QueryWords.java:69)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)
> Caused by: class org.apache.ignite.IgniteCheckedException: Node has left
> grid: 833cae7a-295f-4578-a6b0-f4b6d3a1bf50
> at
> org.apache.ignite.internal.util.IgniteUtils.cast(IgniteUtils.java:6715)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:124)
> at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter$68.applyx(GridCacheAdapter.java:3757)
> at
> org.apache.ignite.internal.processors.cache.GridCacheAdapter$68.applyx(GridCacheAdapter.java:3754)
> at
> org.apache.ignite.internal.util.lang.IgniteClosureX.apply(IgniteClosureX.java:37)
> at
> org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:54)
> at
> org.apache.ignite.internal.util.future.GridFutureChainListener.apply(GridFutureChainListener.java:28)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListener(GridFutureAdapter.java:241)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.notifyListeners(GridFutureAdapter.java:229)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:359)
> at
> org.apache.ignite.internal.util.future.GridFutureAdapter.onDone(GridFutureAdapter.java:325)
> at
> org.apache.ignite.internal.processors.task.GridTaskWorker.finishTask(GridTaskWorker.java:1323)
> at
> org.apache.ignite.internal.processors.task.GridTaskWorker.finishTask(GridTaskWorker.java:1291)
> at
> org.apache.ignite.internal.processors.task.GridTaskWorker$3.apply(GridTaskWorker.java:871)
> at
> org.apache.ignite.internal.processors.task.GridTaskWorker$3.apply(GridTaskWorker.java:822)
> at
> org.apache.ignite.internal.util.IgniteUtils.wrapThreadLoader(IgniteUtils.java:6139)
> at
> org.apache.ignite.internal.processors.task.GridTaskWorker.result(GridTaskWorker.java:822)
> at
> org.apache.ignite.internal.processors.task.GridTaskWorker.onResponse(GridTaskWorker.java:718)
> at
> org.apache.ignite.internal.processors.task.GridTaskWorker.onNodeLeft(GridTaskWorker.java:1223)
> at
> org.apache.ignite.internal.processors.task.GridTaskProcessor$TaskDiscoveryListener.onEvent(GridTaskProcessor.java:1180)
> at
> org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:739)
> at
> org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.notifyListeners(GridEventStorageManager.java:724)
> at
> org.apache.ignite.internal.managers.eventstorage.GridEventStorageManager.record(GridEventStorageManager.java:272)
> at
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.recordEvent(GridDiscoveryManager.java:1533)
> at
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body0(GridDiscoveryManager.java:1720)
> at
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$DiscoveryWorker.body(GridDiscoveryManager.java:1572)
> at
> org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:108)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: class org.apache.ignite.cluster.ClusterTopologyException: Node has
> left grid: 833cae7a-295f-4578-a6b0-f4b6d3a1bf50
> at
> org.apache.ignite.internal.processors.task.GridTaskWorker.onNodeLeft(GridTaskWorker.java:1204)
> ... 9 more
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
