Thanks Andrey, For the moment it works correctly with preferIPv4Stack. The issue first appeared on 2.0, we run the same test suite since 1.6, and we have never seen it.
Thanks again, Juan From: Andrey Mashenkov [mailto:[email protected]] Sent: Wednesday, May 10, 2017 12:32 PM To: [email protected] Subject: Re: How to correctly shut down Ignite Application Hi Juan, Looks like there is a bug, sometimes ignite choose ipv6 protocol to communicate with other nodes that causes strange communication errors. On Wed, May 10, 2017 at 1:01 PM, Juan Barani <[email protected]<mailto:[email protected]>> wrote: Hi, Not necessarily is the case that the client cannot connect to the server. I experienced yesterday with ignite 2.0, that when a client is forced with: l_discoverySpi.setForceServerMode(true); it comes as isClient=false in the exception message, even if setClientMode=true. Since updating to 2.0, we experienced random NPEs when publishing a message in a topic during the interval a server went down, but failed to notify other nodes due to a hard stop => ERROR 2017-05-10T09:57:58,282 - de.uplanet.test.integration.RemoteTestServiceBean[pool-4-thread-1] Failed to send message to remote node: TcpDiscoveryNode [id=ef626cb1-3880-418e-a9d1-68fd692771fd, addrs=[0:0:0:0:0:0:0:1%lo, 10.0.2.15, 127.0.0.1, 172.17.0.1], sockAddrs=[/172.17.0.1:0<http://172.17.0.1:0>, 0:0:0:0:0:0:0:1%lo:0, /127.0.0.1:0<http://127.0.0.1:0>, /10.0.2.15:0<http://10.0.2.15:0>], discPort=0, order=3, intOrder=3, lastExchangeTime=1494410235152, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6, isClient=true] org.apache.ignite.spi.IgniteSpiException: Failed to send message to remote node: TcpDiscoveryNode [id=ef626cb1-3880-418e-a9d1-68fd692771fd, addrs=[0:0:0:0:0:0:0:1%lo, 10.0.2.15, 127.0.0.1, 172.17.0.1], sockAddrs=[/172.17.0.1:0<http://172.17.0.1:0>, 0:0:0:0:0:0:0:1%lo:0, /127.0.0.1:0<http://127.0.0.1:0>, /10.0.2.15:0<http://10.0.2.15:0>], discPort=0, order=3, intOrder=3, lastExchangeTime=1494410235152, loc=false, ver=2.0.0#20170430-sha1:d4eef3c6, isClient=true] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2483) ~[ignite-core-2.0.0.jar:2.0.0] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage(TcpCommunicationSpi.java:2419) ~[ignite-core-2.0.0.jar:2.0.0] at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1329) ~[ignite-core-2.0.0.jar:2.0.0] at org.apache.ignite.internal.managers.communication.GridIoManager.send(GridIoManager.java:1698) ~[ignite-core-2.0.0.jar:2.0.0] at org.apache.ignite.internal.managers.communication.GridIoManager.sendOrderedMessageToGridTopic(GridIoManager.java:1473) ~[ignite-core-2.0.0.jar:2.0.0] at org.apache.ignite.internal.managers.communication.GridIoManager.sendUserMessage(GridIoManager.java:1588) ~[ignite-core-2.0.0.jar:2.0.0] at org.apache.ignite.internal.IgniteMessagingImpl.sendOrdered(IgniteMessagingImpl.java:165) ~[ignite-core-2.0.0.jar:2.0.0] at de.uplanet.lucy.server.distributed.cloud.datagrid.ignite.IgniteGridTopic.publish(IgniteGridTopic.java:58) ~[update/:?] at de.uplanet.test.integration.RemoteTestServiceBean.lambda$3(RemoteTestServiceBean.java:123) ~[update/:?] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_92] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_92] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_92] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_92] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_92] Caused by: org.apache.ignite.IgniteCheckedException: java.lang.NullPointerException at org.apache.ignite.internal.util.IgniteUtils.cast(IgniteUtils.java:7242) ~[ignite-core-2.0.0.jar:2.0.0] at org.apache.ignite.internal.util.future.GridFutureAdapter.resolve(GridFutureAdapter.java:258) ~[ignite-core-2.0.0.jar:2.0.0] at org.apache.ignite.internal.util.future.GridFutureAdapter.get0(GridFutureAdapter.java:170) ~[ignite-core-2.0.0.jar:2.0.0] at org.apache.ignite.internal.util.future.GridFutureAdapter.get(GridFutureAdapter.java:139) ~[ignite-core-2.0.0.jar:2.0.0] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2630) ~[ignite-core-2.0.0.jar:2.0.0] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2455) ~[ignite-core-2.0.0.jar:2.0.0] ... 13 more Caused by: java.util.concurrent.ExecutionException: java.lang.NullPointerException at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_92] at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_92] at org.apache.ignite.internal.util.IgniteUtils.filterReachable(IgniteUtils.java:1895) ~[ignite-core-2.0.0.jar:2.0.0] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:2891) ~[ignite-core-2.0.0.jar:2.0.0] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:2702) ~[ignite-core-2.0.0.jar:2.0.0] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2594) ~[ignite-core-2.0.0.jar:2.0.0] at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.sendMessage0(TcpCommunicationSpi.java:2455) ~[ignite-core-2.0.0.jar:2.0.0] ... 13 more Caused by: java.lang.NullPointerException at org.apache.ignite.internal.util.IgniteUtils.reachable(IgniteUtils.java:2102) ~[ignite-core-2.0.0.jar:2.0.0] at org.apache.ignite.internal.util.IgniteUtils$18.run(IgniteUtils.java:1884) ~[ignite-core-2.0.0.jar:2.0.0] ... 5 more These errors disappeared after setting: -Djava.net.preferIPv4Stack=true I came under the impression that something is working different under ipv4 and ipv6, but cannot tell why the NPE was coming. Probably the errors are related. Thanks! Juan From: Andrey Mashenkov [mailto:[email protected]<mailto:[email protected]>] Sent: Tuesday, May 9, 2017 7:49 PM To: [email protected]<mailto:[email protected]> Subject: Re: How to correctly shut down Ignite Application Looks like client node can't connect to server [id=2b404f04-cad5-4209-a615-ab3b9639b8e4, addrs=[0:0:0:0:0:0:0:1%1, 127.0.0.1, 172.30.32.89].. isClient=false] Would you please attach full logs? On Tue, May 9, 2017 at 2:55 AM, baozipu <[email protected]<mailto:[email protected]>> wrote: Hi, Thanks for the reply. Both the applications are launched in Embedded mode in Spark. I am not sure if the flag of preferIPv4Stack still matters. Both of the applications runs perfect when they are alive. It is only when the "client application" (actually another application in server mode) closed, exit from the topology, that those error messages pop up. I couldn't understand why the server application reports such error. It should be normal that some connections disconnected.... Could you give me more detail for diagnosing the problem? -- View this message in context: http://apache-ignite-users.70518.x6.nabble.com/How-to-correctly-shut-down-Ignite-Application-tp12548p12553.html Sent from the Apache Ignite Users mailing list archive at Nabble.com. -- Best regards, Andrey V. Mashenkov -- Best regards, Andrey V. Mashenkov
