Re: Re: Cannot connect the ignite server after running one or two days

2018-02-01 Thread xiang jie 
OK.  I’ll try it when the problem appears next time. 

 

Thanks.

 

发件人: Evgenii Zhuravlev [mailto:e.zhuravlev...@gmail.com] 
发送时间: 2018年2月1日 16:13
收件人: user@ignite.apache.org
主题: Re: 答复: Re: Cannot connect the ignite server after running one or two days

 

" Failed to deserialize object " is just a consequence of the cause exception, 
as you see from stacktrace - Caused by: java.net.SocketException: Socket closed.

Could you run netstat -apnt on server when you face this problem and share 
results here?

 

 

 

2018-02-01 11:04 GMT+03:00 xiang jie <xia...@cosconetwork.com.cn 
<mailto:xia...@cosconetwork.com.cn> >:

But from server ping client is OK. And from log file, It seems communication
is OK before " Failed to deserialize object " error. It really confuses me.


[2018-01-30
07:40:41,646][DEBUG][exchange-worker-#42%igniteCosco%][GridDhtPartitionDeman
der] Adding partition assignments: GridDhtPreloaderAssignments
[exchangeId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=2364, minorTopVer=0], discoEvt=DiscoveryEvent
[evtNode=TcpDiscoveryNode [id=b07edbd2-3eaa-4b3b-acfc-4159f2bc047d,
addrs=[127.0.0.1, 172.41.6.81], sockAddrs=[/172.41.6.81:0 
<http://172.41.6.81:0> , /127.0.0.1:0 <http://127.0.0.1:0> ],
discPort=0, order=2364, intOrder=1185, lastExchangeTime=1517269240080,
loc=false, ver=2.3.0#20171028-sha1:8add7fd5, isClient=true], topVer=2364,
nodeId8=f290fb24, msg=Node joined: TcpDiscoveryNode
[id=b07edbd2-3eaa-4b3b-acfc-4159f2bc047d, addrs=[127.0.0.1, 172.41.6.81],
sockAddrs=[/172.41.6.81:0 <http://172.41.6.81:0> , /127.0.0.1:0 
<http://127.0.0.1:0> ], discPort=0, order=2364,
intOrder=1185, lastExchangeTime=1517269240080, loc=false,
ver=2.3.0#20171028-sha1:8add7fd5, isClient=true], type=NODE_JOINED,
tstamp=1517269240130], nodeId=b07edbd2, evt=NODE_JOINED],
topVer=AffinityTopologyVersion [topVer=2364, minorTopVer=0],
cancelled=false, exchId=GridDhtPartitionExchangeId
[topVer=AffinityTopologyVersion [topVer=2364, minorTopVer=0],
discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
[id=b07edbd2-3eaa-4b3b-acfc-4159f2bc047d, addrs=[127.0.0.1, 172.41.6.81],
sockAddrs=[/172.41.6.81:0 <http://172.41.6.81:0> , /127.0.0.1:0 
<http://127.0.0.1:0> ], discPort=0, order=2364,
intOrder=1185, lastExchangeTime=1517269240080, loc=false,
ver=2.3.0#20171028-sha1:8add7fd5, isClient=true], topVer=2364,
nodeId8=f290fb24, msg=Node joined: TcpDiscoveryNode
[id=b07edbd2-3eaa-4b3b-acfc-4159f2bc047d, addrs=[127.0.0.1, 172.41.6.81],
sockAddrs=[/172.41.6.81:0 <http://172.41.6.81:0> , /127.0.0.1:0 
<http://127.0.0.1:0> ], discPort=0, order=2364,
intOrder=1185, lastExchangeTime=1517269240080, loc=false,
ver=2.3.0#20171028-sha1:8add7fd5, isClient=true], type=NODE_JOINED,
tstamp=1517269240130], nodeId=b07edbd2, evt=NODE_JOINED], super={}]
..
..
[sock=Socket[addr=/10.10.11.69 <http://10.10.11.69> 
,port=49474,localport=47500],
locNodeId=f290fb24-94ad-4aba-8a6c-78119ad5dd74,
rmtNodeId=b20f0495-ea0f-43be-b92f-f55803767a6b]
class org.apache.ignite.IgniteCheckedException: Failed to deserialize object
with given class loader: sun.misc.Launcher$AppClassLoader@764c12b6 
<mailto:sun.misc.Launcher$AppClassLoader@764c12b6> 
at
org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java
:129)
at
org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(Abstr
actNodeNameAwareMarshaller.java:94)
at
org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:9740)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.
java:5946)
at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
Caused by: java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at org.apache.ignite.marshaller.jdk.JdkMarshallerInputStreamWrapper.
read(JdkMarshallerInputStreamWrapper.java:53)
at java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.
java:2653)
at
java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2
669)
at
java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.j
ava:3146)
at
java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:858)
at java.io.ObjectInputStream.(ObjectInputStream.java:354)
at
org.apache.ignite.marshaller.jdk.JdkMarshallerObjectInputStream.(JdkMa
rshallerObjectInputStream.java:39)
at
org.apach

Re: 答复: Re: Cannot connect the ignite server after running one or two days

2018-02-01 Thread Evgenii Zhuravlev 
" Failed to deserialize object " is just a consequence of the cause
exception, as you see from stacktrace - Caused by:
java.net.SocketException: Socket closed.
Could you run netstat -apnt on server when you face this problem and share
results here?



2018-02-01 11:04 GMT+03:00 xiang jie <xia...@cosconetwork.com.cn>:

> But from server ping client is OK. And from log file, It seems
> communication
> is OK before " Failed to deserialize object " error. It really confuses me.
>
>
> [2018-01-30
> 07:40:41,646][DEBUG][exchange-worker-#42%igniteCosco%][
> GridDhtPartitionDeman
> der] Adding partition assignments: GridDhtPreloaderAssignments
> [exchangeId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
> [topVer=2364, minorTopVer=0], discoEvt=DiscoveryEvent
> [evtNode=TcpDiscoveryNode [id=b07edbd2-3eaa-4b3b-acfc-4159f2bc047d,
> addrs=[127.0.0.1, 172.41.6.81], sockAddrs=[/172.41.6.81:0, /127.0.0.1:0],
> discPort=0, order=2364, intOrder=1185, lastExchangeTime=1517269240080,
> loc=false, ver=2.3.0#20171028-sha1:8add7fd5, isClient=true], topVer=2364,
> nodeId8=f290fb24, msg=Node joined: TcpDiscoveryNode
> [id=b07edbd2-3eaa-4b3b-acfc-4159f2bc047d, addrs=[127.0.0.1, 172.41.6.81],
> sockAddrs=[/172.41.6.81:0, /127.0.0.1:0], discPort=0, order=2364,
> intOrder=1185, lastExchangeTime=1517269240080, loc=false,
> ver=2.3.0#20171028-sha1:8add7fd5, isClient=true], type=NODE_JOINED,
> tstamp=1517269240130], nodeId=b07edbd2, evt=NODE_JOINED],
> topVer=AffinityTopologyVersion [topVer=2364, minorTopVer=0],
> cancelled=false, exchId=GridDhtPartitionExchangeId
> [topVer=AffinityTopologyVersion [topVer=2364, minorTopVer=0],
> discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
> [id=b07edbd2-3eaa-4b3b-acfc-4159f2bc047d, addrs=[127.0.0.1, 172.41.6.81],
> sockAddrs=[/172.41.6.81:0, /127.0.0.1:0], discPort=0, order=2364,
> intOrder=1185, lastExchangeTime=1517269240080, loc=false,
> ver=2.3.0#20171028-sha1:8add7fd5, isClient=true], topVer=2364,
> nodeId8=f290fb24, msg=Node joined: TcpDiscoveryNode
> [id=b07edbd2-3eaa-4b3b-acfc-4159f2bc047d, addrs=[127.0.0.1, 172.41.6.81],
> sockAddrs=[/172.41.6.81:0, /127.0.0.1:0], discPort=0, order=2364,
> intOrder=1185, lastExchangeTime=1517269240080, loc=false,
> ver=2.3.0#20171028-sha1:8add7fd5, isClient=true], type=NODE_JOINED,
> tstamp=1517269240130], nodeId=b07edbd2, evt=NODE_JOINED], super={}]
> ..
> ..
> [sock=Socket[addr=/10.10.11.69,port=49474,localport=47500],
> locNodeId=f290fb24-94ad-4aba-8a6c-78119ad5dd74,
> rmtNodeId=b20f0495-ea0f-43be-b92f-f55803767a6b]
> class org.apache.ignite.IgniteCheckedException: Failed to deserialize
> object
> with given class loader: sun.misc.Launcher$AppClassLoader@764c12b6
> at
> org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(
> JdkMarshaller.java
> :129)
> at
> org.apache.ignite.marshaller.AbstractNodeNameAwareMarshalle
> r.unmarshal(Abstr
> actNodeNameAwareMarshaller.java:94)
> at
> org.apache.ignite.internal.util.IgniteUtils.unmarshal(
> IgniteUtils.java:9740)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl$
> SocketReader.body(ServerImpl.
> java:5946)
> at
> org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
> Caused by: java.net.SocketException: Socket closed
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.socketRead(SocketInputStream.
> java:116)
> at java.net.SocketInputStream.read(SocketInputStream.java:171)
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
> at org.apache.ignite.marshaller.jdk.JdkMarshallerInputStreamWrappe
> r.
> read(JdkMarshallerInputStreamWrapper.java:53)
> at java.io.ObjectInputStream$PeekInputStream.read(
> ObjectInputStream.
> java:2653)
> at
> java.io.ObjectInputStream$PeekInputStream.readFully(
> ObjectInputStream.java:2
> 669)
> at
> java.io.ObjectInputStream$BlockDataInputStream.
> readShort(ObjectInputStream.j
> ava:3146)
> at
> java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:858)
> at java.io.ObjectInputStream.(ObjectInputStream.java:354)
> at
> org.apache.ignite.marshaller.jdk.JdkMarshallerObjectInputStream
> .(JdkMa
> rshallerObjectInputStream.java:39)
>     at
> org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(
> JdkMarshaller.java
> :119)
> ... 4 more
>
> -邮件原件-
> 发件人: ezhuravlev [mailto:e.zhuravlev.

答复: Re: Cannot connect the ignite server after running one or two days

2018-02-01 Thread xiang jie 
But from server ping client is OK. And from log file, It seems communication
is OK before " Failed to deserialize object " error. It really confuses me. 


[2018-01-30
07:40:41,646][DEBUG][exchange-worker-#42%igniteCosco%][GridDhtPartitionDeman
der] Adding partition assignments: GridDhtPreloaderAssignments
[exchangeId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion
[topVer=2364, minorTopVer=0], discoEvt=DiscoveryEvent
[evtNode=TcpDiscoveryNode [id=b07edbd2-3eaa-4b3b-acfc-4159f2bc047d,
addrs=[127.0.0.1, 172.41.6.81], sockAddrs=[/172.41.6.81:0, /127.0.0.1:0],
discPort=0, order=2364, intOrder=1185, lastExchangeTime=1517269240080,
loc=false, ver=2.3.0#20171028-sha1:8add7fd5, isClient=true], topVer=2364,
nodeId8=f290fb24, msg=Node joined: TcpDiscoveryNode
[id=b07edbd2-3eaa-4b3b-acfc-4159f2bc047d, addrs=[127.0.0.1, 172.41.6.81],
sockAddrs=[/172.41.6.81:0, /127.0.0.1:0], discPort=0, order=2364,
intOrder=1185, lastExchangeTime=1517269240080, loc=false,
ver=2.3.0#20171028-sha1:8add7fd5, isClient=true], type=NODE_JOINED,
tstamp=1517269240130], nodeId=b07edbd2, evt=NODE_JOINED],
topVer=AffinityTopologyVersion [topVer=2364, minorTopVer=0],
cancelled=false, exchId=GridDhtPartitionExchangeId
[topVer=AffinityTopologyVersion [topVer=2364, minorTopVer=0],
discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode
[id=b07edbd2-3eaa-4b3b-acfc-4159f2bc047d, addrs=[127.0.0.1, 172.41.6.81],
sockAddrs=[/172.41.6.81:0, /127.0.0.1:0], discPort=0, order=2364,
intOrder=1185, lastExchangeTime=1517269240080, loc=false,
ver=2.3.0#20171028-sha1:8add7fd5, isClient=true], topVer=2364,
nodeId8=f290fb24, msg=Node joined: TcpDiscoveryNode
[id=b07edbd2-3eaa-4b3b-acfc-4159f2bc047d, addrs=[127.0.0.1, 172.41.6.81],
sockAddrs=[/172.41.6.81:0, /127.0.0.1:0], discPort=0, order=2364,
intOrder=1185, lastExchangeTime=1517269240080, loc=false,
ver=2.3.0#20171028-sha1:8add7fd5, isClient=true], type=NODE_JOINED,
tstamp=1517269240130], nodeId=b07edbd2, evt=NODE_JOINED], super={}]
..
..
[sock=Socket[addr=/10.10.11.69,port=49474,localport=47500],
locNodeId=f290fb24-94ad-4aba-8a6c-78119ad5dd74,
rmtNodeId=b20f0495-ea0f-43be-b92f-f55803767a6b]
class org.apache.ignite.IgniteCheckedException: Failed to deserialize object
with given class loader: sun.misc.Launcher$AppClassLoader@764c12b6
at
org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java
:129)
at
org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(Abstr
actNodeNameAwareMarshaller.java:94)
at
org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:9740)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.
java:5946)
at
org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
Caused by: java.net.SocketException: Socket closed
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at org.apache.ignite.marshaller.jdk.JdkMarshallerInputStreamWrapper.
read(JdkMarshallerInputStreamWrapper.java:53)
at java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.
java:2653)
at
java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2
669)
at
java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.j
ava:3146)
at
java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:858)
at java.io.ObjectInputStream.(ObjectInputStream.java:354)
at
org.apache.ignite.marshaller.jdk.JdkMarshallerObjectInputStream.(JdkMa
rshallerObjectInputStream.java:39)
at
org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java
:119)
... 4 more  

-邮件原件-
发件人: ezhuravlev [mailto:e.zhuravlev...@gmail.com] 
发送时间: 2018年1月31日 21:50
收件人: user@ignite.apache.org
主题: Re: Re: Cannot connect the ignite server after running one or two days

 >There are several clients connected by VPN,  is it possible to the
client's restart regularly causing ignite socket communication to a certain
degree of obstruction and becoming more and more serious as time goes by? 

Does this mean that it's impossible to connect from server to client nodes
directly? If so, it can definitely be the cause of this behavior.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Re: Cannot connect the ignite server after running one or two days

2018-01-31 Thread ezhuravlev 
 >There are several clients connected by VPN,  is it possible to the client's
restart regularly causing ignite socket communication to a certain degree of
obstruction and becoming more and more serious as time goes by? 

Does this mean that it's impossible to connect from server to client nodes
directly? If so, it can definitely be the cause of this behavior.



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Re: Cannot connect the ignite server after running one or two days

2018-01-30 Thread xiang jie 
Hi, Evgenii

Thank you for your reply. 

The logs files are too big and same mostly, so collect the logs at the
moment of connection. 

Because clients cannot connect to cluster any more and always show these
error messages when we try to connect them again, I don't think it's the
problem of that moment for long GC pause.  

At the same time, there are clients still can connect but after restarting
these clients several times,  they may cannot connect to cluster either.  As
time goes on, all clients cannot connect to cluster.  If we restart one
server node is still no use, clients cannot connect. When restart all server
nodes and load data again, everything is OK.  Do these circumstances
indicate that it is not a network problem? 

There are several clients connected by VPN,  is it possible to the client's
restart regularly causing ignite socket communication to a certain degree of
obstruction and becoming more and more serious as time goes by?


Thanks

-邮件原件-
发件人: ezhuravlev [mailto:e.zhuravlev...@gmail.com] 
发送时间: 2018年1月30日 23:00
收件人: user@ignite.apache.org
主题: Re: Re: Cannot connect the ignite server after running one or two days

Hi,

Looks like logs from the server is still not full. If you've checked them
and you sure that you don't have any exceptions in it before witnessing this
problem, then, I think that you could have some connection problems or a
long GC pause. Do you have any network monitoring? Also, I would recommend
checking GC logs at this moment(or share it with community) on all nodes.

Also, Why did you set "queryThreadPoolSize" value="32" while you have only
24 CPUs for all 5 hosts? It's definitely will reduce performance due to a
lot of context-switching.

If everything okay with GC logs, it's possible to check TCP dumps, just to
understand where the connection breaks off.

Evgenii



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Re: Cannot connect the ignite server after running one or two days

2018-01-30 Thread ezhuravlev 
Hi,

Looks like logs from the server is still not full. If you've checked them
and you sure that you don't have any exceptions in it before witnessing this
problem, then, I think that you could have some connection problems or a
long GC pause. Do you have any network monitoring? Also, I would recommend
checking GC logs at this moment(or share it with community) on all nodes.

Also, Why did you set "queryThreadPoolSize" value="32" while you have only
24 CPUs for all 5 hosts? It's definitely will reduce performance due to a
lot of context-switching.

If everything okay with GC logs, it's possible to check TCP dumps, just to
understand where the connection breaks off.

Evgenii



--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/

Re: Cannot connect the ignite server after running one or two days

2018-01-23 Thread Evgenii Zhuravlev 
Hi,

I see that you use both ipv4 and ipv6 stacks, I think it can somehow affect
this behavior, so, I would recommend enabling -Djava.net.preferIPv4Stack=true
for all nodes in the cluster(including clients). Also, why do you decided
to set such parameters to the communicationSPI:






If after suggested change you will still witness this problem, please
provide full logs from all nodes in cluster.

Evgenii


2018-01-23 9:35 GMT+03:00 xiang jie :

> Hi,
>
>
>
> We have deployed ignite 2.3 and created about 100 caches and load data
> from oracle into these caches.
>
>
>
> But clients(started in developing tomcat web apps, we often start and stop
> these apps) often cannot connect the ignite server after one or two days.
> By searching logs, we found below errors in server side：
>
>
>
>
>
> [2018-01-23 12:43:30,979][WARN 
> ][grid-timeout-worker-#23%igniteCosco%][TcpDiscoverySpi]
> Socket write has timed out (consider increasing
> 'IgniteConfiguration.failureDetectionTimeout' configuration property)
> [failureDetectionTimeout=1, rmtAddr=/172.41.27.66:3871, rmtPort=3871,
> sockTimeout=5000]
>
> [2018-01-23 
> 12:43:30,980][ERROR][tcp-disco-sock-reader-#63%igniteCosco%][TcpDiscoverySpi]
> Caught exception on message read 
> [sock=Socket[addr=/172.41.27.66,port=3871,localport=47500],
> locNodeId=89865e33-c722-4989-b663-a75c936b068f,
> rmtNodeId=ed9d5cc9-11d3-4bfd-a648-1ae718ac16f9]
>
> class org.apache.ignite.IgniteCheckedException: Failed to deserialize
> object with given class loader: sun.misc.Launcher$AppClassLoader@7390d1e8
>
>   at org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(Jd
> kMarshaller.java:129)
>
>   at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller
> .unmarshal(AbstractNodeNameAwareMarshaller.java:94)
>
>   at org.apache.ignite.internal.util.IgniteUtils.unmarshal(Ignite
> Utils.java:9740)
>
>   at org.apache.ignite.spi.discovery.tcp.ServerImpl$SocketReader.
> body(ServerImpl.java:5946)
>
>   at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
>
> Caused by: java.net.SocketException: Socket closed
>
>   at java.net.SocketInputStream.socketRead0(Native Method)
>
>   at java.net.SocketInputStream.read(SocketInputStream.java:152)
>
>   at java.net.SocketInputStream.read(SocketInputStream.java:122)
>
>   at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
>
>   at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
>
>   at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
>
>   at org.apache.ignite.marshaller.jdk.JdkMarshallerInputStreamWra
> pper.read(JdkMarshallerInputStreamWrapper.java:53)
>
>   at java.io.ObjectInputStream$PeekInputStream.read(ObjectInputSt
> ream.java:2310)
>
>   at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectIn
> putStream.java:2323)
>
>   at java.io.ObjectInputStream$BlockDataInputStream.readShort(
> ObjectInputStream.java:2794)
>
>   at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream
> .java:801)
>
>   at java.io.ObjectInputStream.(ObjectInputStream.java:299)
>
>   at org.apache.ignite.marshaller.jdk.JdkMarshallerObjectInputStr
> eam.(JdkMarshallerObjectInputStream.java:39)
>
>   at org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(Jd
> kMarshaller.java:119)
>
>   ... 4 more
>
> [2018-01-23 
> 12:43:30,980][DEBUG][tcp-disco-sock-reader-#63%igniteCosco%][TcpDiscoverySpi]
> Client connection failed 
> [sock=Socket[addr=/172.41.27.66,port=3871,localport=47500],
> locNodeId=89865e33-c722-4989-b663-a75c936b068f,
> rmtNodeId=ed9d5cc9-11d3-4bfd-a648-1ae718ac16f9]
>
> [2018-01-23 12:43:30,981][INFO 
> ][tcp-disco-sock-reader-#63%igniteCosco%][TcpDiscoverySpi]
> Finished serving remote node connection [rmtAddr=/172.41.27.66:3871,
> rmtPort=3871
>
> [2018-01-23 
> 12:43:30,981][DEBUG][tcp-disco-sock-reader-#63%igniteCosco%][TcpDiscoverySpi]
> Grid runnable finished normally: tcp-disco-sock-reader-#63%igniteCosco%
>
> [2018-01-23 
> 12:43:30,981][ERROR][tcp-disco-client-message-worker-#64%igniteCosco%][TcpDiscoverySpi]
> Client connection failed 
> [sock=Socket[addr=/172.41.27.66,port=3871,localport=47500],
> locNodeId=89865e33-c722-4989-b663-a75c936b068f,
> rmtNodeId=ed9d5cc9-11d3-4bfd-a648-1ae718ac16f9,
> msg=TcpDiscoveryNodeAddedMessage [node=TcpDiscoveryNode
> [id=ed9d5cc9-11d3-4bfd-a648-1ae718ac16f9, addrs=[0:0:0:0:0:0:0:1,
> 127.0.0.1, 172.41.27.66, 192.168.126.1, 192.168.81.1,
> 2001:0:9d38:6ab8:8aa:3888:2532:627e], 
> sockAddrs=[/2001:0:9d38:6ab8:8aa:3888:2532:627e:0,
> /192.168.81.1:0, /127.0.0.1:0, /0:0:0:0:0:0:0:1:0, /192.168.126.1:0, /
> 172.41.27.66:0], discPort=0, order=0, intOrder=340,
> lastExchangeTime=1516682580894, loc=false, ver=2.3.0#20171028-sha1:8add7fd5,
> isClient=true], dataPacket=o.a.i.spi.discovery
> .tcp.internal.DiscoveryDataPacket@4d671e9d, discardMsgId=null,
> discardCustomMsgId=null, top=[TcpDiscoveryNode
> [id=fbf2344b-652b-4c5c-ad78-a470ec8c62e8,