I'm getting this error when the nodes are shutting down.
What are the possible causes for this?
A bug was marked for this a year ago or so, but was marked as resolved, it
seems?
http://apache-ignite-issues.70530.x6.nabble.com/jira-Updated-IGNITE-9826-Ignite-node-with-TcpDiscoveryS3IpFinder-can-hang-while-stopping-td75612.html
Ignite cache is empty. Shutting down...
2019-10-02 08:44:37 [main] INFO
org.apache.ignite.internal.processors.rest.protocols.tcp.GridTcpRestProtocol.info(117)
- Command protocol successfully stopped: TCP binary
2019-10-02 08:44:37 [main] INFO
org.eclipse.jetty.server.AbstractConnector.doStop(332) - Stopped
ServerConnector@70101687{HTTP/1.1,[http/1.1]}{0.0.0.0:8080}
2019-10-02 08:44:37 [main] INFO
org.apache.ignite.internal.processors.rest.protocols.http.jetty.GridJettyRestProtocol.info(117)
- Command protocol successfully stopped: Jetty REST
2019-10-02 08:44:37 [grid-timeout-worker-#79] INFO
org.apache.ignite.internal.IgniteKernal.info(117) -
Metrics for local node (to disable set 'metricsLogFrequency' to 0)
^-- Node [id=c51b8462, uptime=00:15:00.073]
^-- H/N/C [hosts=2, nodes=2, CPUs=72]
^-- CPU [cur=-100%, avg=-99.73%, GC=0%]
^-- PageMemory [pages=1226]
^-- Heap [used=112MB, free=99.59%, comm=516MB]
^-- Off-heap [used=4MB, free=99.97%, comm=336MB]
^-- sysMemPlc region [used=0MB, free=99.21%, comm=40MB]
^-- default region [used=4MB, free=99.97%, comm=256MB]
^-- TxLog region [used=0MB, free=100%, comm=40MB]
^-- Outbound messages queue [size=0]
^-- Public thread pool [active=0, idle=0, qSize=0]
^-- System thread pool [active=0, idle=6, qSize=0]
2019-10-02 08:44:37 [tcp-disco-sock-reader-#7] INFO
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.info(117) - Finished
serving remote node connection [rmtAddr=/10.0.11.180:49151, rmtPort=49151
2019-10-02 08:44:37 [tcp-disco-sock-reader-#4] INFO
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.info(117) - Finished
serving remote node connection [rmtAddr=/10.0.31.134:42413, rmtPort=42413
2019-10-02 08:44:37 [tcp-disco-sock-reader-#6] INFO
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.info(117) - Finished
serving remote node connection [rmtAddr=/10.0.21.167:60763, rmtPort=60763
2019-10-02 08:44:37 [tcp-disco-ip-finder-cleaner-#5] ERROR
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.error(137) - Failed to
clean IP finder up.
org.apache.ignite.spi.IgniteSpiException: Failed to list objects in the
bucket: ignite-configurations-production
at
org.apache.ignite.spi.discovery.tcp.ipfinder.s3.TcpDiscoveryS3IpFinder.getRegisteredAddresses(TcpDiscoveryS3IpFinder.java:192)
at
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.registeredAddresses(TcpDiscoverySpi.java:1900)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$IpFinderCleaner.cleanIpFinder(ServerImpl.java:1998)
at
org.apache.ignite.spi.discovery.tcp.ServerImpl$IpFinderCleaner.body(ServerImpl.java:1973)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
Caused by: com.amazonaws.SdkClientException: Failed to sanitize XML
document destined for handler class
com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser$ListBucketHandler
at
com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.sanitizeXmlDocument(XmlResponsesSaxParser.java:214)
at
com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.parseListBucketObjectsResponse(XmlResponsesSaxParser.java:298)
at
com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:70)
at
com.amazonaws.services.s3.model.transform.Unmarshallers$ListObjectsUnmarshaller.unmarshall(Unmarshallers.java:59)
at
com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:62)
at
com.amazonaws.services.s3.internal.S3XmlResponseHandler.handle(S3XmlResponseHandler.java:31)
at
com.amazonaws.http.response.AwsResponseHandlerAdapter.handle(AwsResponseHandlerAdapter.java:70)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleResponse(AmazonHttpClient.java:1554)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1272)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1056)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:743)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:717)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:699)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:667)
at
com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:649)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:513)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4137)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:4079)
at
com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:819)
at
com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:791)
at
org.apache.ignite.spi.discovery.tcp.ipfinder.s3.TcpDiscoveryS3IpFinder.getRegisteredAddresses(TcpDiscoveryS3IpFinder.java:146)
... 4 common frames omitted
Caused by: com.amazonaws.AbortedException:
at
com.amazonaws.internal.SdkFilterInputStream.abortIfNeeded(SdkFilterInputStream.java:53)
at
com.amazonaws.internal.SdkFilterInputStream.read(SdkFilterInputStream.java:81)
at
com.amazonaws.event.ProgressInputStream.read(ProgressInputStream.java:180)
at java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
at java.base/java.io.InputStreamReader.read(InputStreamReader.java:185)
at java.base/java.io.BufferedReader.read1(BufferedReader.java:210)
at java.base/java.io.BufferedReader.read(BufferedReader.java:287)
at java.base/java.io.Reader.read(Reader.java:229)
at
com.amazonaws.services.s3.model.transform.XmlResponsesSaxParser.sanitizeXmlDocument(XmlResponsesSaxParser.java:186)
... 24 common frames omitted
2019-10-02 08:44:40 [tcp-disco-msg-worker-#2] WARN
org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.warning(127) - Local
node has detected failed nodes and started cluster-wide procedure. To speed
up failure detection please see 'Failure Detection' section under javadoc
for 'TcpDiscoverySpi'
