[
https://issues.apache.org/jira/browse/IGNITE-10388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Karpushin updated IGNITE-10388:
--------------------------------------
Description:
I'm creating this issue as suggested on
[stackoverflow|https://stackoverflow.com/questions/53433749/how-to-prevent-ignite-from-dying-if-first-attempt-to-join-the-cluster-failed]
I've also created a simple [github
project|https://github.com/skarpushin/issue20181122] to help reproduce the
issue (please check README file before running the project).
I'm using {{docker-compose}} to start several containers with java web
application with embedded Ignite 2.6.
I configured Ignite so it will use SSL.
If I start node1 and wait it to fully start, and then start node2, they will
join cluster no problem.
But if I start them simultaneously (that's how it works by default with
docker-compose) they both will fail with exception and will never retry or get
back.
{{{{class org.apache.ignite.IgniteException: Unable to establish secure
connection. Was remote cluster configured with SSL? [rmtAddr=/10.9.9.1:47501,
errMsg="Remote host closed connection during handshake"] at
org.apache.ignite.spi.discovery.tcp.ServerImpl.sendMessageDirectly(ServerImpl.java:1293)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl.sendJoinRequestMessage(ServerImpl.java:1046)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:890)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:373) at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:1948)
at
org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:915)
at
org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1721) at
org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1028) at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2014)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1723)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1151) at
org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:649) at
org.apache.ignite.IgniteSpring.start(IgniteSpring.java:66) at
org.apache.ignite.IgniteSpringBean.afterSingletonsInstantiated(IgniteSpringBean.java:172)
at
org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:777)
at
org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:869)
at
org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550)
at
org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:409)
at
org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:291)
at
org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:103)
at
org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4853)
at
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5314)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) at
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:753)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:729) at
org.apache.catalina.core.StandardHost.addChild(StandardHost.java:717) at
org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:976) at
org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1853) at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at
java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748) Caused by:
javax.net.ssl.SSLHandshakeException: Remote host closed connection during
handshake at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1002)
at
sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:757) at
sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123) at
java.io.OutputStream.write(OutputStream.java:75) at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.writeToSocket(TcpDiscoverySpi.java:1501)
at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1452)
at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1413)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl.sendMessageDirectly(ServerImpl.java:1199)
... 33 more Caused by: java.io.EOFException: SSL peer shut down incorrectly at
sun.security.ssl.InputRecord.read(InputRecord.java:505) at
sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983) ... 41 more }}
{{2018-11-22 14:47:06.727 INFO localhost-startStop-1 Log4JLogger Command
protocol successfully stopped: TCP binary }}
{{2018-11-22 14:47:11.733 INFO localhost-startStop-1 Log4JLogger No
verification for local node leave has been received from coordinator (will stop
node anyway).}}}}
was:
I'm creating this issue as suggested on
[stackoverflow|https://stackoverflow.com/questions/53433749/how-to-prevent-ignite-from-dying-if-first-attempt-to-join-the-cluster-failed]
I've also created a simple [github
project|https://github.com/skarpushin/issue20181122] to help reproduce the
issue (please check README file before running the project).
I'm using {{docker-compose}} to start several containers with java web
application with embedded Ignite 2.6.
I configured Ignite so it will use SSL.
If I start node1 and wait it to fully start, and then start node2, they will
join cluster no problem.
But if I start them simultaneously (that's how it works by default with
docker-compose) they both will fail with exception and will never retry or get
back.
{{{{class org.apache.ignite.IgniteException: Unable to establish secure
connection. Was remote cluster configured with SSL? [rmtAddr=/10.9.9.1:47501,
errMsg="Remote host closed connection during handshake"] at
org.apache.ignite.spi.discovery.tcp.ServerImpl.sendMessageDirectly(ServerImpl.java:1293)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl.sendJoinRequestMessage(ServerImpl.java:1046)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:890)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:373) at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:1948)
at
org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297)
at
org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:915)
at
org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1721) at
org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1028) at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2014)
at
org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1723)
at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1151) at
org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:649) at
org.apache.ignite.IgniteSpring.start(IgniteSpring.java:66) at
org.apache.ignite.IgniteSpringBean.afterSingletonsInstantiated(IgniteSpringBean.java:172)
at
org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:777)
at
org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:869)
at
org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550)
at
org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:409)
at
org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:291)
at
org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:103)
at
org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4853)
at
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5314)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) at
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:753)
at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:729) at
org.apache.catalina.core.StandardHost.addChild(StandardHost.java:717) at
org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:976) at
org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1853) at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at
java.util.concurrent.FutureTask.run(FutureTask.java:266) at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748) Caused by:
javax.net.ssl.SSLHandshakeException: Remote host closed connection during
handshake at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1002)
at
sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:757) at
sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123) at
java.io.OutputStream.write(OutputStream.java:75) at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.writeToSocket(TcpDiscoverySpi.java:1501)
at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1452)
at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1413)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl.sendMessageDirectly(ServerImpl.java:1199)
... 33 more Caused by: java.io.EOFException: SSL peer shut down incorrectly at
sun.security.ssl.InputRecord.read(InputRecord.java:505) at
sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983) ... 41 more
2018-11-22 14:47:06.727 INFO localhost-startStop-1 Log4JLogger Command protocol
successfully stopped: TCP binary 2018-11-22 14:47:11.733 INFO
localhost-startStop-1 Log4JLogger No verification for local node leave has been
received from coordinator (will stop node anyway).}}}}
> Prevent Ignite from "dying" if first attempt to join the cluster failed
> -----------------------------------------------------------------------
>
> Key: IGNITE-10388
> URL: https://issues.apache.org/jira/browse/IGNITE-10388
> Project: Ignite
> Issue Type: Bug
> Components: general
> Affects Versions: 2.6
> Environment: * Linux OS (I'm using Manjaro 1.18)
> * Jdk 1.8
> * Docker-compose 3.7
> Reporter: Sergey Karpushin
> Priority: Major
>
> I'm creating this issue as suggested on
> [stackoverflow|https://stackoverflow.com/questions/53433749/how-to-prevent-ignite-from-dying-if-first-attempt-to-join-the-cluster-failed]
> I've also created a simple [github
> project|https://github.com/skarpushin/issue20181122] to help reproduce the
> issue (please check README file before running the project).
> I'm using {{docker-compose}} to start several containers with java web
> application with embedded Ignite 2.6.
> I configured Ignite so it will use SSL.
> If I start node1 and wait it to fully start, and then start node2, they will
> join cluster no problem.
> But if I start them simultaneously (that's how it works by default with
> docker-compose) they both will fail with exception and will never retry or
> get back.
> {{{{class org.apache.ignite.IgniteException: Unable to establish secure
> connection. Was remote cluster configured with SSL? [rmtAddr=/10.9.9.1:47501,
> errMsg="Remote host closed connection during handshake"] at
> org.apache.ignite.spi.discovery.tcp.ServerImpl.sendMessageDirectly(ServerImpl.java:1293)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl.sendJoinRequestMessage(ServerImpl.java:1046)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl.joinTopology(ServerImpl.java:890)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl.spiStart(ServerImpl.java:373)
> at
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.spiStart(TcpDiscoverySpi.java:1948)
> at
> org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:297)
> at
> org.apache.ignite.internal.managers.discovery.GridDiscoveryManager.start(GridDiscoveryManager.java:915)
> at
> org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1721)
> at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1028) at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2014)
> at
> org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1723)
> at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1151) at
> org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:649) at
> org.apache.ignite.IgniteSpring.start(IgniteSpring.java:66) at
> org.apache.ignite.IgniteSpringBean.afterSingletonsInstantiated(IgniteSpringBean.java:172)
> at
> org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:777)
> at
> org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:869)
> at
> org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550)
> at
> org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:409)
> at
> org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:291)
> at
> org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:103)
> at
> org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4853)
> at
> org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5314)
> at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145) at
> org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:753)
> at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:729)
> at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:717) at
> org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:976) at
> org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1853) at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at
> java.util.concurrent.FutureTask.run(FutureTask.java:266) at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748) Caused by:
> javax.net.ssl.SSLHandshakeException: Remote host closed connection during
> handshake at
> sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1002) at
> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
> at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:757) at
> sun.security.ssl.AppOutputStream.write(AppOutputStream.java:123) at
> java.io.OutputStream.write(OutputStream.java:75) at
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.writeToSocket(TcpDiscoverySpi.java:1501)
> at
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1452)
> at
> org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.openSocket(TcpDiscoverySpi.java:1413)
> at
> org.apache.ignite.spi.discovery.tcp.ServerImpl.sendMessageDirectly(ServerImpl.java:1199)
> ... 33 more Caused by: java.io.EOFException: SSL peer shut down incorrectly
> at sun.security.ssl.InputRecord.read(InputRecord.java:505) at
> sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983) ... 41 more
> }}
> {{2018-11-22 14:47:06.727 INFO localhost-startStop-1 Log4JLogger Command
> protocol successfully stopped: TCP binary }}
> {{2018-11-22 14:47:11.733 INFO localhost-startStop-1 Log4JLogger No
> verification for local node leave has been received from coordinator (will
> stop node anyway).}}}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
