Hello, The issue with grid segmented: The reasons of this may be long garbage collector pause, long pause of process of user layer due to process of system kernal work, network problem (like break).
By steps: 1) Update ignite version, because the 1.5.0 is very old. 1.7.0 is already available. 2) Collect logs work garbage collector. You can use the article [1]. 3) Check your system, using utils top, dstat, ping It the issue continue to repeat, provide logs from server and client, garbage collector logs and output the utils at the time of the crash. On Fri, Aug 12, 2016 at 5:08 AM, 波板仔 <[email protected]> wrote: > My ignite cluster is deploy on 2 machines. Each machine has 4 ignite > process. like: > > $jps > 30678 CommandLineStartup > 30723 CommandLineStartup > 52461 Jps > 30698 CommandLineStartup > 30681 CommandLineStartup > > When there is little data in the ignite cluster, I can use > IgniteDataStreamer to write data into the ignite cluster. But if there have > been more than 200G data in the ignite cluster, I can't use > IgniteDataStreamer to write more data into the cluster. The memory and disk > space are both enough. and the CPU load average is 2.09. The client log is : > > > [17:04:00,407][INFO][main][IgniteKernal%grid_recmd] Daemon mode: off > [17:04:00,407][INFO][main][IgniteKernal%grid_recmd] OS: Linux > [17:04:00,407][INFO][main][IgniteKernal%grid_recmd] OS user: mqq > [17:04:00,408][INFO][main][IgniteKernal%grid_recmd] Language runtime: > Java Platform API Specification ver. 1.7 > [17:04:00,408][INFO][main][IgniteKernal%grid_recmd] VM information: > Java(TM) SE Runtime Environment 1.7.0_72-b14 Oracle Corporation Java > HotSpot(TM) 64-Bit Server VM 24.72-b04 > [17:04:00,409][INFO][main][IgniteKernal%grid_recmd] VM total memory: > 2.9GB > [17:04:00,409][INFO][main][IgniteKernal%grid_recmd] Remote Management > [restart: off, REST: on, JMX (remote: off)] > [17:04:00,409][INFO][main][IgniteKernal%grid_recmd] > IGNITE_HOME=/data1/home/igni1.5 > [17:04:00,410][INFO][main][IgniteKernal%grid_recmd] VM arguments: > [-Xms3g, -Xmx3g, -XX:-UseGCOverheadLimit, -XX:+AggressiveOpts, > -XX:MaxPermSize=256m, -XX:+UseParNewGC, -XX:+UseConcMarkSweepGC, > -XX:+UseTLAB, -DIGNITE_QUIET=false] > [17:04:00,418][INFO][main][IgniteKernal%grid_recmd] Configured caches > ['ignite-marshaller-sys-cache', 'ignite-sys-cache', > 'ignite-atomics-sys-cache', 'cache_recmd'] > [17:04:00,422][INFO][main][IgniteKernal%grid_recmd] 3-rd party > licenses can be found at: /data1/home/igni1.5/libs/licenses > [17:04:00,518][INFO][main][IgniteKernal%grid_recmd] Non-loopback > local IPs: 10.0.0.1 > [17:04:00,519][INFO][main][IgniteKernal%grid_recmd] Enabled local > MACs: 9017ACC1492D > [17:04:00,530][INFO][main][IgnitePluginProcessor] Configured plugins: > [17:04:00,530][INFO][main][IgnitePluginProcessor] ^-- None > [17:04:00,530][INFO][main][IgnitePluginProcessor] > [17:04:00,574][WARNING][main][TcpCommunicationSpi] Failure detection > timeout will be ignored (one of SPI parameters has been set explicitly) > [17:04:00,627][INFO][main][TcpCommunicationSpi] IPC shared memory > server endpoint started [port=48104, tokDir=/data1/home/igni1.5/ > work/ipc/shmem/e052dc07-8992-4834-87fd-abf84c0f644a-14472] > [17:04:00,628][INFO][main][TcpCommunicationSpi] Successfully bound > shared memory communication to TCP port [port=48104, locHost= > 0.0.0.0/0.0.0.0] > [17:04:00,652][INFO][main][TcpCommunicationSpi] Successfully bound to > TCP port [port=47106, locHost=0.0.0.0/0.0.0.0] > [17:04:00,685][WARNING][main][NoopCheckpointSpi] Checkpoints are > disabled (to enable configure any GridCheckpointSpi implementation) > [17:04:00,749][WARNING][main][GridCollisionManager] Collision > resolution is disabled (all jobs will be activated upon arrival). > [17:04:00,755][WARNING][main][NoopSwapSpaceSpi] Swap space is > disabled. To enable use FileSwapSpaceSpi. > [17:04:00,757][INFO][main][IgniteKernal%grid_recmd] Security status > [authentication=off, tls/ssl=off] > [17:04:00,861][WARNING][main][GridCacheProcessor] Deployment mode for > cache is not CONTINUOUS or SHARED (it is recommended that you change > deployment mode and restart): ISOLATED > [17:04:01,040][INFO][main][GridTcpRestProtocol] Command protocol > successfully started [name=TCP binary, host=0.0.0.0/0.0.0.0, port=11217] > > [17:04:01,138][SEVERE][tcp-client-disco-msg-worker-#4%grid_recmd%][TcpDiscoverySpi] > Exception on joining: Connection refused > java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) > at java.net.AbstractPlainSocketImpl.doConnect( > AbstractPlainSocketImpl.java:339) > at java.net.AbstractPlainSocketImpl.connectToAddress( > AbstractPlainSocketImpl.java:200) > at java.net.AbstractPlainSocketImpl.connect( > AbstractPlainSocketImpl.java:182) > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > at java.net.Socket.connect(Socket.java:579) > at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi. > openSocket(TcpDiscoverySpi.java:1266) > at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi. > openSocket(TcpDiscoverySpi.java:1241) > at org.apache.ignite.spi.discovery.tcp.ClientImpl. > sendJoinRequest(ClientImpl.java:579) > at org.apache.ignite.spi.discovery.tcp.ClientImpl. > joinTopology(ClientImpl.java:505) > at org.apache.ignite.spi.discovery.tcp.ClientImpl. > access$900(ClientImpl.java:118) > at org.apache.ignite.spi.discovery.tcp.ClientImpl$ > MessageWorker.tryJoin(ClientImpl.java:1508) > at org.apache.ignite.spi.discovery.tcp.ClientImpl$ > MessageWorker.body(ClientImpl.java:1327) > at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) > > [17:04:01,139][SEVERE][tcp-client-disco-msg-worker-#4%grid_recmd%][TcpDiscoverySpi] > Exception on joining: Connection refused > java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) > at java.net.AbstractPlainSocketImpl.doConnect( > AbstractPlainSocketImpl.java:339) > at java.net.AbstractPlainSocketImpl.connectToAddress( > AbstractPlainSocketImpl.java:200) > at java.net.AbstractPlainSocketImpl.connect( > AbstractPlainSocketImpl.java:182) > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > at java.net.Socket.connect(Socket.java:579) > at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi. > openSocket(TcpDiscoverySpi.java:1266) > at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi. > openSocket(TcpDiscoverySpi.java:1241) > at org.apache.ignite.spi.discovery.tcp.ClientImpl. > sendJoinRequest(ClientImpl.java:579) > at org.apache.ignite.spi.discovery.tcp.ClientImpl. > joinTopology(ClientImpl.java:505) > at org.apache.ignite.spi.discovery.tcp.ClientImpl. > access$900(ClientImpl.java:118) > at org.apache.ignite.spi.discovery.tcp.ClientImpl$ > MessageWorker.tryJoin(ClientImpl.java:1508) > at org.apache.ignite.spi.discovery.tcp.ClientImpl$ > MessageWorker.body(ClientImpl.java:1327) > at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) > > [17:04:01,140][SEVERE][tcp-client-disco-msg-worker-#4%grid_recmd%][TcpDiscoverySpi] > Exception on joining: Connection refused > java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) > at java.net.AbstractPlainSocketImpl.doConnect( > AbstractPlainSocketImpl.java:339) > at java.net.AbstractPlainSocketImpl.connectToAddress( > AbstractPlainSocketImpl.java:200) > at java.net.AbstractPlainSocketImpl.connect( > AbstractPlainSocketImpl.java:182) > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > at java.net.Socket.connect(Socket.java:579) > at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi. > openSocket(TcpDiscoverySpi.java:1266) > at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi. > openSocket(TcpDiscoverySpi.java:1241) > at org.apache.ignite.spi.discovery.tcp.ClientImpl. > sendJoinRequest(ClientImpl.java:579) > at org.apache.ignite.spi.discovery.tcp.ClientImpl. > joinTopology(ClientImpl.java:505) > at org.apache.ignite.spi.discovery.tcp.ClientImpl. > access$900(ClientImpl.java:118) > at org.apache.ignite.spi.discovery.tcp.ClientImpl$ > MessageWorker.tryJoin(ClientImpl.java:1508) > at org.apache.ignite.spi.discovery.tcp.ClientImpl$ > MessageWorker.body(ClientImpl.java:1327) > at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) > > [17:04:01,141][SEVERE][tcp-client-disco-msg-worker-#4%grid_recmd%][TcpDiscoverySpi] > Exception on joining: Connection refused > java.net.ConnectException: Connection refused > at java.net.PlainSocketImpl.socketConnect(Native Method) > at java.net.AbstractPlainSocketImpl.doConnect( > AbstractPlainSocketImpl.java:339) > at java.net.AbstractPlainSocketImpl.connectToAddress( > AbstractPlainSocketImpl.java:200) > at java.net.AbstractPlainSocketImpl.connect( > AbstractPlainSocketImpl.java:182) > at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) > at java.net.Socket.connect(Socket.java:579) > at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi. > openSocket(TcpDiscoverySpi.java:1266) > at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi. > openSocket(TcpDiscoverySpi.java:1241) > at org.apache.ignite.spi.discovery.tcp.ClientImpl. > sendJoinRequest(ClientImpl.java:579) > at org.apache.ignite.spi.discovery.tcp.ClientImpl. > joinTopology(ClientImpl.java:505) > at org.apache.ignite.spi.discovery.tcp.ClientImpl. > access$900(ClientImpl.java:118) > at org.apache.ignite.spi.discovery.tcp.ClientImpl$ > MessageWorker.tryJoin(ClientImpl.java:1508) > at org.apache.ignite.spi.discovery.tcp.ClientImpl$ > MessageWorker.body(ClientImpl.java:1327) > at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) > [17:05:01,179][SEVERE][main][IgniteKernal%grid_recmd] Failed to start > manager: GridManagerAdapter [enabled=true, name=o.a.i.i.managers. > discovery.GridDiscoveryManager] > class org.apache.ignite.IgniteCheckedException: Failed to start SPI: > TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, ackTimeout=5000, > reconCnt=10, maxAckTimeout=600000, forceSrvMode=false, > clientReconnectDisabled=false] > at org.apache.ignite.internal.managers.GridManagerAdapter. > startSpi(GridManagerAdapter.java:258) > at org.apache.ignite.internal.managers.discovery. > GridDiscoveryManager.start(GridDiscoveryManager.java:660) > at org.apache.ignite.internal.IgniteKernal.startManager( > IgniteKernal.java:1505) > at org.apache.ignite.internal.IgniteKernal.start( > IgniteKernal.java:917) > at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0( > IgnitionEx.java:1688) > at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start( > IgnitionEx.java:1547) > at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1003) > at org.apache.ignite.internal.IgnitionEx.startConfigurations( > IgnitionEx.java:929) > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:815) > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:714) > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:584) > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:554) > at org.apache.ignite.Ignition.start(Ignition.java:347) > at IgniteOnlineWriteTest.actPutBatchs(IgniteOnlineWriteTest.java:30) > at IgniteOnlineWriteTest.main(IgniteOnlineWriteTest.java:12) > Caused by: class org.apache.ignite.spi.IgniteSpiException: Join > process timed out, did not receive response for join request (consider > increasing 'joinTimeout' configuration property) [joinTimeout=60000, > sock=Socket[addr=/10.0.0.1,port=47500,localport=35836]] > at org.apache.ignite.spi.discovery.tcp.ClientImpl$ > MessageWorker.body(ClientImpl.java:1334) > at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) > [17:05:01,180][SEVERE][main][IgniteKernal%grid_recmd] Got exception > while starting (will rollback startup routine). > class org.apache.ignite.IgniteCheckedException: Failed to start > manager: GridManagerAdapter [enabled=true, name=org.apache.ignite. > internal.managers.discovery.GridDiscoveryManager] > at org.apache.ignite.internal.IgniteKernal.startManager( > IgniteKernal.java:1510) > at org.apache.ignite.internal.IgniteKernal.start( > IgniteKernal.java:917) > at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0( > IgnitionEx.java:1688) > at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start( > IgnitionEx.java:1547) > at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1003) > at org.apache.ignite.internal.IgnitionEx.startConfigurations( > IgnitionEx.java:929) > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:815) > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:714) > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:584) > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:554) > at org.apache.ignite.Ignition.start(Ignition.java:347) > at IgniteOnlineWriteTest.actPutBatchs(IgniteOnlineWriteTest.java:30) > at IgniteOnlineWriteTest.main(IgniteOnlineWriteTest.java:12) > Caused by: class org.apache.ignite.IgniteCheckedException: Failed to > start SPI: TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, > ackTimeout=5000, reconCnt=10, maxAckTimeout=600000, forceSrvMode=false, > clientReconnectDisabled=false] > at org.apache.ignite.internal.managers.GridManagerAdapter. > startSpi(GridManagerAdapter.java:258) > at org.apache.ignite.internal.managers.discovery. > GridDiscoveryManager.start(GridDiscoveryManager.java:660) > at org.apache.ignite.internal.IgniteKernal.startManager( > IgniteKernal.java:1505) > ... 12 more > Caused by: class org.apache.ignite.spi.IgniteSpiException: Join > process timed out, did not receive response for join request (consider > increasing 'joinTimeout' configuration property) [joinTimeout=60000, > sock=Socket[addr=/10.0.0.1,port=47500,localport=35836]] > at org.apache.ignite.spi.discovery.tcp.ClientImpl$ > MessageWorker.body(ClientImpl.java:1334) > at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) > [17:05:01,193][INFO][main][GridTcpRestProtocol] Command protocol > successfully stopped: TCP binary > > [17:05:01,201][WARNING][grid-nio-worker-0-#58%grid_recmd%][TcpCommunicationSpi] > Local node is not started or fully initialized [isStopping=true] > > [17:05:01,201][WARNING][grid-nio-worker-1-#59%grid_recmd%][TcpCommunicationSpi] > Local node is not started or fully initialized [isStopping=true] > > [17:05:01,843][SEVERE][tcp-client-disco-sock-reader-#3%grid_recmd%][TcpDiscoverySpi] > Failed to read message > [sock=Socket[addr=/10.0.0.1,port=47500,localport=35836], > locNodeId=e052dc07-8992-4834-87fd-abf84c0f644a, > rmtNodeId=198bc1a5-1b1d-4b65-bccb-b2eb5ad1ec9b] > class org.apache.ignite.IgniteCheckedException: Failed to deserialize > object with given class loader: sun.misc.Launcher$AppClassLoader@3415ddf5 > at org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal( > JdkMarshaller.java:105) > at org.apache.ignite.spi.discovery.tcp.ClientImpl$ > SocketReader.body(ClientImpl.java:888) > at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) > Caused by: java.io.IOException: Stream closed. > at java.net.AbstractPlainSocketImpl.available( > AbstractPlainSocketImpl.java:458) > at java.net.SocketInputStream.available(SocketInputStream.java:245) > at java.io.BufferedInputStream.read(BufferedInputStream.java:342) > at org.apache.ignite.marshaller.jdk.JdkMarshallerInputStreamWrappe > r.read(JdkMarshallerInputStreamWrapper.java:53) > at java.io.ObjectInputStream$PeekInputStream.read( > ObjectInputStream.java:2310) > at java.io.ObjectInputStream$PeekInputStream.readFully( > ObjectInputStream.java:2323) > at java.io.ObjectInputStream$BlockDataInputStream. > readUTFBody(ObjectInputStream.java:3063) > at java.io.ObjectInputStream$BlockDataInputStream.readUTF( > ObjectInputStream.java:2864) > at java.io.ObjectInputStream.readString(ObjectInputStream.java:1638) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1341) > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) > at org.apache.ignite.internal.util.IgniteUtils.readMap( > IgniteUtils.java:4890) > at org.apache.ignite.spi.discovery.tcp.internal. > TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:559) > at java.io.ObjectInputStream.readExternalData( > ObjectInputStream.java:1837) > at java.io.ObjectInputStream.readOrdinaryObject( > ObjectInputStream.java:1796) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) > at java.util.ArrayList.readObject(ArrayList.java:771) > at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at java.io.ObjectStreamClass.invokeReadObject( > ObjectStreamClass.java:1017) > at java.io.ObjectInputStream.readSerialData( > ObjectInputStream.java:1893) > at java.io.ObjectInputStream.readOrdinaryObject( > ObjectInputStream.java:1798) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at java.io.ObjectInputStream.defaultReadFields( > ObjectInputStream.java:1990) > at java.io.ObjectInputStream.readSerialData( > ObjectInputStream.java:1915) > at java.io.ObjectInputStream.readOrdinaryObject( > ObjectInputStream.java:1798) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) > at java.util.TreeMap.buildFromSorted(TreeMap.java:3302) > at java.util.TreeMap.readObject(TreeMap.java:3234) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke( > NativeMethodAccessorImpl.java:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at java.io.ObjectStreamClass.invokeReadObject( > ObjectStreamClass.java:1017) > at java.io.ObjectInputStream.readSerialData( > ObjectInputStream.java:1893) > at java.io.ObjectInputStream.readOrdinaryObject( > ObjectInputStream.java:1798) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at java.io.ObjectInputStream.defaultReadFields( > ObjectInputStream.java:1990) > at java.io.ObjectInputStream.readSerialData( > ObjectInputStream.java:1915) > at java.io.ObjectInputStream.readOrdinaryObject( > ObjectInputStream.java:1798) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) > at org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal( > JdkMarshaller.java:102) > ... 2 more > > [17:05:01,845][SEVERE][tcp-client-disco-sock-reader-#3%grid_recmd%][TcpDiscoverySpi] > Connection failed [sock=Socket[addr=/10.0.0.1,port=47500,localport=35836], > locNodeId=e052dc07-8992-4834-87fd-abf84c0f644a] > java.io.IOException: Stream closed. > at java.net.AbstractPlainSocketImpl.available( > AbstractPlainSocketImpl.java:458) > at java.net.SocketInputStream.available(SocketInputStream.java:245) > at java.io.BufferedInputStream.read(BufferedInputStream.java:342) > at org.apache.ignite.marshaller.jdk.JdkMarshallerInputStreamWrappe > r.read(JdkMarshallerInputStreamWrapper.java:53) > at java.io.ObjectInputStream$PeekInputStream.read( > ObjectInputStream.java:2310) > at java.io.ObjectInputStream$PeekInputStream.readFully( > ObjectInputStream.java:2323) > at java.io.ObjectInputStream$BlockDataInputStream. > readUTFBody(ObjectInputStream.java:3063) > at java.io.ObjectInputStream$BlockDataInputStream.readUTF( > ObjectInputStream.java:2864) > at java.io.ObjectInputStream.readString(ObjectInputStream.java:1638) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1341) > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) > at org.apache.ignite.internal.util.IgniteUtils.readMap( > IgniteUtils.java:4890) > at org.apache.ignite.spi.discovery.tcp.internal. > TcpDiscoveryNode.readExternal(TcpDiscoveryNode.java:559) > at java.io.ObjectInputStream.readExternalData( > ObjectInputStream.java:1837) > at java.io.ObjectInputStream.readOrdinaryObject( > ObjectInputStream.java:1796) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) > at java.util.ArrayList.readObject(ArrayList.java:771) > at sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at java.io.ObjectStreamClass.invokeReadObject( > ObjectStreamClass.java:1017) > at java.io.ObjectInputStream.readSerialData( > ObjectInputStream.java:1893) > at java.io.ObjectInputStream.readOrdinaryObject( > ObjectInputStream.java:1798) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at java.io.ObjectInputStream.defaultReadFields( > ObjectInputStream.java:1990) > at java.io.ObjectInputStream.readSerialData( > ObjectInputStream.java:1915) > at java.io.ObjectInputStream.readOrdinaryObject( > ObjectInputStream.java:1798) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) > at java.util.TreeMap.buildFromSorted(TreeMap.java:3302) > at java.util.TreeMap.readObject(TreeMap.java:3234) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke( > NativeMethodAccessorImpl.java:57) > at sun.reflect.DelegatingMethodAccessorImpl.invoke( > DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at java.io.ObjectStreamClass.invokeReadObject( > ObjectStreamClass.java:1017) > at java.io.ObjectInputStream.readSerialData( > ObjectInputStream.java:1893) > at java.io.ObjectInputStream.readOrdinaryObject( > ObjectInputStream.java:1798) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at java.io.ObjectInputStream.defaultReadFields( > ObjectInputStream.java:1990) > at java.io.ObjectInputStream.readSerialData( > ObjectInputStream.java:1915) > at java.io.ObjectInputStream.readOrdinaryObject( > ObjectInputStream.java:1798) > at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350) > at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370) > at org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal( > JdkMarshaller.java:102) > at org.apache.ignite.spi.discovery.tcp.ClientImpl$ > SocketReader.body(ClientImpl.java:888) > at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) > [17:05:01,876][INFO][main][IgniteKernal%grid_recmd] > > >>> +----------------------------------------------------------- > ----------------------------+ > >>> Ignite ver. 1.5.0-final#20151229-sha1: > f1f8cda2f3f62231f42a59951bf34c39577c1bec stopped OK > >>> +----------------------------------------------------------- > ----------------------------+ > >>> Grid name: grid_recmd > >>> Grid uptime: 00:01:02:580 > > > class org.apache.ignite.IgniteException: Failed to start manager: > GridManagerAdapter [enabled=true, name=org.apache.ignite. > internal.managers.discovery.GridDiscoveryManager] > at org.apache.ignite.internal.util.IgniteUtils. > convertException(IgniteUtils.java:882) > at org.apache.ignite.Ignition.start(Ignition.java:350) > at IgniteOnlineWriteTest.actPutBatchs(IgniteOnlineWriteTest.java:30) > at IgniteOnlineWriteTest.main(IgniteOnlineWriteTest.java:12) > Caused by: class org.apache.ignite.IgniteCheckedException: Failed to > start manager: GridManagerAdapter [enabled=true, name=org.apache.ignite. > internal.managers.discovery.GridDiscoveryManager] > at org.apache.ignite.internal.IgniteKernal.startManager( > IgniteKernal.java:1510) > at org.apache.ignite.internal.IgniteKernal.start( > IgniteKernal.java:917) > at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0( > IgnitionEx.java:1688) > at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start( > IgnitionEx.java:1547) > at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1003) > at org.apache.ignite.internal.IgnitionEx.startConfigurations( > IgnitionEx.java:929) > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:815) > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:714) > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:584) > at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:554) > at org.apache.ignite.Ignition.start(Ignition.java:347) > ... 2 more > Caused by: class org.apache.ignite.IgniteCheckedException: Failed to > start SPI: TcpDiscoverySpi [addrRslvr=null, sockTimeout=5000, > ackTimeout=5000, reconCnt=10, maxAckTimeout=600000, forceSrvMode=false, > clientReconnectDisabled=false] > at org.apache.ignite.internal.managers.GridManagerAdapter. > startSpi(GridManagerAdapter.java:258) > at org.apache.ignite.internal.managers.discovery. > GridDiscoveryManager.start(GridDiscoveryManager.java:660) > at org.apache.ignite.internal.IgniteKernal.startManager( > IgniteKernal.java:1505) > ... 12 more > Caused by: class org.apache.ignite.spi.IgniteSpiException: Join > process timed out, did not receive response for join request (consider > increasing 'joinTimeout' configuration property) [joinTimeout=60000, > sock=Socket[addr=/10.0.0.1,port=47500,localport=35836]] > at org.apache.ignite.spi.discovery.tcp.ClientImpl$ > MessageWorker.body(ClientImpl.java:1334) > at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62) > > the ignite process's JVM_OPTS is : > > export JVM_OPTS="-Xms5g -Xmx5g -Xmn2g -server -XX:-UseGCOverheadLimit > -XX:+AggressiveOpts -XX:MaxPermSize=256m -XX:+UseP > arNewGC -XX:+UseConcMarkSweepGC -XX:+UseTLAB" > > the srv.xml is > > <bean id="cacheA" class="org.apache.ignite.configuration. > CacheConfiguration"> > <property name="name" value="cache_recmd"/> > <property name="cacheMode" value="PARTITIONED"/><!-- partition > --> > <property name="affinity"> > <bean class="org.apache.ignite.cache.affinity.rendezvous. > RendezvousAffinityFunction"> > <property name="hashIdResolver" > > <bean class="org.apache.ignite.cache.affinity. > AffinityNodeAddressHashResolver"/> > </property> > <property name="partitions" value="280" /> > </bean> > </property> > <property name="backups" value="0"/> > <property name="writeSynchronizationMode" > value="PRIMARY_SYNC"/> > <property name="atomicityMode" value="ATOMIC"/> > <property name="atomicWriteOrderMode" value="PRIMARY"/> > <property name="memoryMode" value="ONHEAP_TIERED"/><!-- memory > tire --> > <property name="startSize" value="#{30 * 1024 * 1024}"/> > <property name="offHeapMaxMemory" value="#{512 * 1024 * > 1024}"/> > <property name="swapEnabled" value="true"/> > <property name="evictionPolicy"> > <bean class="org.apache.ignite.cache.eviction.lru. > LruEvictionPolicy"> > <property name="maxMemorySize" value="#{1024 * 1024 * > 1024}"/> > </bean> > </property> > <property name="rebalanceMode" value="ASYNC"/> > <property name="rebalanceBatchSize" value="#{1024 * 1024}"/> > <property name="rebalanceThrottle" value="100"/> > </bean> > > <bean id="igniA" > class="org.apache.ignite.configuration.IgniteConfiguration" > scope="singleton"> > <property name="gridName" value="grid_recmd"/> > <property name="clientMode" value="false"/> > <property name="metricsLogFrequency" value="0"/><!-- 0 means > disable metrics log --> > <property name="deploymentMode" value="ISOLATED"/> > <property name="discoverySpi"> > <bean class="org.apache.ignite.spi. > discovery.tcp.TcpDiscoverySpi"> > <property name="localPort" value="47500"/> > <property name="localPortRange" value="20"/> > <property name="ipFinder"> > <bean class="org.apache.ignite.spi. > discovery.tcp.ipfinder.vm.TcpDiscoveryVmIpFinder"> > <property name="shared" value="true"/> > <property name="addresses"> > <list> > <value>10.0.0.1:47500..47520</value> > <value>10.0.0.2:47500..47520</value> > </list> > </property> > </bean> > </property> > <property name="joinTimeout" value="60000"/> > </bean> > </property> > <property name="communicationSpi"> > <bean class="org.apache.ignite.spi.communication.tcp. > TcpCommunicationSpi"> > <property name="socketWriteTimeout" value="30000"/> > <property name="connectTimeout" value="50000"/> > </bean> > </property> > <property name="failureDetectionTimeout" value="60000"/> > <property name="swapSpaceSpi"> > <bean class="org.apache.ignite.spi.swapspace.file. > FileSwapSpaceSpi"> > <property name="baseDirectory" > value="/data1/home/igni1.5/my/swp"/> > </bean> > </property> > <property name="cacheConfiguration"> > <list> > <ref bean="cacheA"/> > </list> > </property> > <property name="includeEventTypes"> > <list> > </list> > </property> > </bean> > > -- Vladislav Pyatkov
