Lucasgranet opened a new issue #4455:
URL: https://github.com/apache/cloudstack/issues/4455
<!--
Verify first that your issue/request is not already reported on GitHub.
Also test if the latest release and master branch are affected too.
Always add information AFTER of these HTML comments, but no need to delete
the comments.
-->
##### ISSUE TYPE
<!-- Pick one below and delete the rest -->
* Bug Report
##### COMPONENT NAME
<!--
Categorize the issue, e.g. API, VR, VPN, UI, etc.
-->
~~~
API
~~~
##### CLOUDSTACK VERSION
<!--
New line separated list of affected versions, commit ID for issues on master
branch.
-->
~~~
4.15-SNAPSHOT 5f8289ffe90fd829493bf4e0d23c64ef50313627 (master - 11/09/2020)
~~~
##### CONFIGURATION
<!--
Information about the configuration if relevant, e.g. basic network,
advanced networking, etc. N/A otherwise
-->
Built on a CentOS8.2 (up to date) - Deployed on a CentOS8.2 (up to date)
##### OS / ENVIRONMENT
<!--
Information about the environment if relevant, N/A otherwise
-->
##### SUMMARY
<!-- Explain the problem/feature briefly -->
I cannot start ASC 4.15-SNAPSHOT on a CentOS8.2.
- I have already start ASC 4.15 on C8. I wanted to fetch last updates to
start a new cluster but a SSL connection cannot be established to the
management server.
The service retry in loop and cause a very high CPU usage.
- All APIs are very very slow (and not sure they are working well)
- The issue seems to be caused by a bad certificate generation (bad usage -
see below)
##### STEPS TO REPRODUCE
<!--
For bugs, show exactly how to reproduce the problem, using a minimal
test-case. Use Screenshots if accurate.
For new features, show how the feature would be used.
-->
Build from master, deploy on a C8.
<!-- Paste example playbooks or commands between quotes below -->
<!-- You can also paste gist.github.com links for larger files -->
##### EXPECTED RESULTS
<!-- What did you expect to happen when running the steps above? -->
~~~
No major log issue
~~~
##### ACTUAL RESULTS
<!-- What actually happened? -->
<!-- Paste verbatim command output between quotes below -->
Log from the service
~~~
nov. 09 20:42:36 cs.iaasm.lgr.fr java[3568]: INFO
[o.a.c.s.l.CloudStackExtendedLifeCycle] (main:null) (logid:) Starting
CloudStack Components
nov. 09 20:42:36 cs.iaasm.lgr.fr java[3568]: INFO
[o.a.c.s.l.CloudStackExtendedLifeCycle] (main:null) (logid:) Done Starting
CloudStack Components
nov. 09 20:42:37 cs.iaasm.lgr.fr java[3568]: INFO
[o.a.c.s.l.CloudStackExtendedLifeCycle] (main:null) (logid:) Configuring
CloudStack Components
nov. 09 20:42:37 cs.iaasm.lgr.fr java[3568]: INFO
[o.a.c.s.l.CloudStackExtendedLifeCycle] (main:null) (logid:) Done Configuring
CloudStack Components
nov. 09 20:42:38 cs.iaasm.lgr.fr java[3568]: INFO [c.c.u.LogUtils]
(main:null) (logid:) log4j configuration found at
/etc/cloudstack/management/log4j-cloud.xml
nov. 09 20:42:39 cs.iaasm.lgr.fr java[3568]: WARN [c.c.u.n.Link]
(AgentManager-SSLHandshakeHandler-39:null) (logid:) This SSL engine was forced
to close inbound due to end of stream.
nov. 09 20:42:39 cs.iaasm.lgr.fr java[3568]: javax.net.ssl.SSLException:
closing inbound before receiving peer's close_notify
nov. 09 20:42:39 cs.iaasm.lgr.fr java[3568]: at
java.base/sun.security.ssl.Alert.createSSLException(Alert.java:133)
nov. 09 20:42:39 cs.iaasm.lgr.fr java[3568]: at
java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
nov. 09 20:42:39 cs.iaasm.lgr.fr java[3568]: at
java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:337)
nov. 09 20:42:39 cs.iaasm.lgr.fr java[3568]: at
java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:293)
nov. 09 20:42:39 cs.iaasm.lgr.fr java[3568]: at
java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:284)
nov. 09 20:42:39 cs.iaasm.lgr.fr java[3568]: at
java.base/sun.security.ssl.SSLEngineImpl.closeInbound(SSLEngineImpl.java:733)
nov. 09 20:42:39 cs.iaasm.lgr.fr java[3568]: at
com.cloud.utils.nio.Link.doHandshakeUnwrap(Link.java:490)
nov. 09 20:42:39 cs.iaasm.lgr.fr java[3568]: at
com.cloud.utils.nio.Link.doHandshake(Link.java:618)
nov. 09 20:42:39 cs.iaasm.lgr.fr java[3568]: at
com.cloud.utils.nio.NioConnection$1.run(NioConnection.java:216)
nov. 09 20:42:39 cs.iaasm.lgr.fr java[3568]: at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
nov. 09 20:42:39 cs.iaasm.lgr.fr java[3568]: at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
nov. 09 20:42:39 cs.iaasm.lgr.fr java[3568]: at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
nov. 09 20:42:39 cs.iaasm.lgr.fr java[3568]: at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
nov. 09 20:42:39 cs.iaasm.lgr.fr java[3568]: at
java.base/java.lang.Thread.run(Thread.java:834)
nov. 09 20:42:40 cs.iaasm.lgr.fr java[3568]: WARN [c.c.a.AlertManagerImpl]
(Cluster-Notification-1:ctx-f38888de) (logid:9d1cfe62) AlertType:: 14 |
dataCenterId:: 0 | podId:: 0 | clusterId:: null | message:: Management server
node 10.2.112.14 is up
nov. 09 20:42:41 cs.iaasm.lgr.fr java[3568]: INFO
[c.c.a.m.AgentManagerImpl] (AgentManager-Handler-2:null) (logid:) Connection
from /10.2.112.177 closed but no cleanup was done.
nov. 09 20:42:43 cs.iaasm.lgr.fr java[3568]: ERROR [c.c.u.n.Link]
(AgentManager-SSLHandshakeHandler-72:null) (logid:) SSL error caught during
wrap data: Unexpected handshake message: client_hello, for local
address=/10.2.112.14:8250, remote address=/10.2.112.177:50496.
nov. 09 20:42:45 cs.iaasm.lgr.fr java[3568]: ERROR [c.c.u.n.Link]
(AgentManager-SSLHandshakeHandler-37:null) (logid:) SSL error caught during
wrap data: Unexpected handshake message: client_hello, for local
address=/10.2.112.14:8250, remote address=/10.2.112.177:50388.
nov. 09 20:42:46 cs.iaasm.lgr.fr java[3568]: INFO
[c.c.a.m.AgentManagerImpl] (AgentManager-Handler-5:null) (logid:) Connection
from /10.2.112.177 closed but no cleanup was done.
nov. 09 20:42:48 cs.iaasm.lgr.fr java[3568]: INFO
[c.c.a.m.AgentManagerImpl] (AgentManager-Handler-6:null) (logid:) Connection
from /10.2.112.177 closed but no cleanup was done.
nov. 09 20:42:59 cs.iaasm.lgr.fr java[3568]: ERROR [c.c.u.n.Link]
(AgentManager-SSLHandshakeHandler-42:null) (logid:) SSL error caught during
wrap data: Unexpected handshake message: client_hello, for local
address=/10.2.112.14:8250, remote address=/10.2.112.177:50398.
nov. 09 20:43:03 cs.iaasm.lgr.fr java[3568]: WARN [c.c.u.n.Link]
(AgentManager-SSLHandshakeHandler-3:null) (logid:) SSL Handshake has taken more
than 30s to connect to: /10.2.112.177:50320. Please investigate this connection.
nov. 09 20:43:03 cs.iaasm.lgr.fr java[3568]: WARN [c.c.u.n.Link]
(AgentManager-SSLHandshakeHandler-6:null) (logid:) SSL Handshake has taken more
than 30s to connect to: /10.2.112.177:50326. Please investigate this connection.
nov. 09 20:43:05 cs.iaasm.lgr.fr java[3568]: WARN [c.c.u.n.Link]
(AgentManager-SSLHandshakeHandler-11:null) (logid:) SSL Handshake has taken
more than 30s to connect to: /10.2.112.177:50336. Please investigate this
connection.
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: WARN [c.c.u.n.Link]
(AgentManager-SSLHandshakeHandler-9:null) (logid:) SSL Handshake has taken more
than 30s to connect to: /10.2.112.177:50332. Please investigate this connection.
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: WARN [c.c.u.n.Link]
(AgentManager-SSLHandshakeHandler-4:null) (logid:) SSL Handshake has taken more
than 30s to connect to: /10.2.112.177:50322. Please investigate this connection.
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: WARN [c.c.u.n.Link]
(AgentManager-SSLHandshakeHandler-1:null) (logid:) SSL Handshake has taken more
than 30s to connect to: /10.2.112.177:50316. Please investigate this connection.
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: WARN [c.c.u.n.Link]
(AgentManager-SSLHandshakeHandler-7:null) (logid:) SSL Handshake has taken more
than 30s to connect to: /10.2.112.177:50328. Please investigate this connection.
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: WARN [c.c.u.n.Link]
(AgentManager-SSLHandshakeHandler-8:null) (logid:) SSL Handshake has taken more
than 30s to connect to: /10.2.112.177:50330. Please investigate this connection.
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: WARN [c.c.u.n.Link]
(AgentManager-SSLHandshakeHandler-10:null) (logid:) SSL Handshake has taken
more than 30s to connect to: /10.2.112.177:50334. Please investigate this
connection.
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: WARN [c.c.u.n.Link]
(AgentManager-SSLHandshakeHandler-15:null) (logid:) SSL Handshake has taken
more than 30s to connect to: /10.2.112.177:50344. Please investigate this
connection.
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: WARN [c.c.u.n.Link]
(AgentManager-SSLHandshakeHandler-13:null) (logid:) SSL Handshake has taken
more than 30s to connect to: /10.2.112.177:50340. Please investigate this
connection.
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: WARN [c.c.u.n.Link]
(AgentManager-SSLHandshakeHandler-108:null) (logid:) This SSL engine was forced
to close inbound due to end of stream.
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: javax.net.ssl.SSLException:
closing inbound before receiving peer's close_notify
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: at
java.base/sun.security.ssl.Alert.createSSLException(Alert.java:133)
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: at
java.base/sun.security.ssl.Alert.createSSLException(Alert.java:117)
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: at
java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:337)
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: at
java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:293)
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: at
java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:284)
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: at
java.base/sun.security.ssl.SSLEngineImpl.closeInbound(SSLEngineImpl.java:733)
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: at
com.cloud.utils.nio.Link.doHandshakeUnwrap(Link.java:490)
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: at
com.cloud.utils.nio.Link.doHandshake(Link.java:618)
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: at
com.cloud.utils.nio.NioConnection$1.run(NioConnection.java:216)
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: at
java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: at
java.base/java.lang.Thread.run(Thread.java:834)
nov. 09 20:43:06 cs.iaasm.lgr.fr java[3568]: WARN [c.c.u.n.Link]
(AgentManager-SSLHandshakeHandler-12:null) (logid:) SSL Handshake has taken
more than 30s to connect to: /10.2.112.177:50338. Please investigate this
connection.
~~~
I performed an OpenSSL connection to the management service - TCP Link
cannot be established, OpenSSL refused the certificate. Java must not like too.
~~~
-> # openssl s_client -connect 10.2.112.14:8250
CONNECTED(00000003)
Can't use SSL_get_servername
depth=0 CN = ca.cloudstack.apache.org
verify error:num=18:self signed certificate
verify return:1
depth=0 CN = ca.cloudstack.apache.org
verify error:num=26:unsupported certificate purpose
verify return:1
depth=0 CN = ca.cloudstack.apache.org
verify return:1
---
Certificate chain
0 s:CN = ca.cloudstack.apache.org
i:CN = ca.cloudstack.apache.org
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFDTCCAvWgAwIBAgIJAMtEGDhOFq8iMA0GCSqGSIb3DQEBCwUAMCMxITAfBgNV
BAMMGGNhLmNsb3Vkc3RhY2suYXBhY2hlLm9yZzAgFw0yMDExMDkwNzMzNTJaGA8y
MDUwMTEwMjE5MzM1MlowIzEhMB8GA1UEAwwYY2EuY2xvdWRzdGFjay5hcGFjaGUu
b3JnMIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAq2Z0gJIKjv3lLIFQ
bV7ARjqQBwAQd4hl674mTknzTBMvq2rnw9wSlnIcngK+8OsSHXLHIoD9wWr5C/+h
OB43rRUoYBJ2KjFg9I/HF8tSiaRPMIZ0DVfyZSmttdo7kQBhKgxBPRpfegD8ry/p
tx8YZb75ADAZKVPwM8B4O1+cgYSYG92b54K/pa6W5YV2eXcplOYKlo4zXMGvEQ3z
CwjT74WBub+u284/B4FZHkwZY+Pq90gzFkeEbUzsnXVfRZr4sPuzLiza07DfExFA
GzVwMpKIeCu8YRHL+iTnzQKoKeaeXgIRQcviCf5N+zk2tnqG/2O5CcFTtVfYYq52
iUPRmAEpUYQl9LEWcOwmJiNhNPG/EBsIjoe8ZxsB3gH4sqe17ieOAWZvZCTmvPvH
dryGoSYafICHpMLrTD24zb8uC6mrpyqF7v1M/aoXxxoWE2Y3hD5UnThWU2w2ZIrx
62UeF9XZGsq6s2/oFC8Saeg+cJcW/an42VnVJ0Mp6SNEGlU04Gr0fmQIRcFkI79Y
uQvsUUnxaBGKivTf+KbySBZ7pa7Tn2v8tlwAXIqQqcWZ4qqpTu9R0PApmSkoZCKl
HKPQJpSHrIkE7BWcjViO9RS0xed1J9jyNsFMsynqItZT29gETQokFRdIOWti5ALw
hDpnC/NxOZpOOgClu0gVgUq4g3UCAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAO
BgNVHQ8BAf8EBAMCAQYwHQYDVR0OBBYEFPb7eAs0X8PkriAswvCuW3Xfkhn2MA0G
CSqGSIb3DQEBCwUAA4ICAQAWWLimV3S/iB8tvbx56dn1u2HyG9Q49pSD/TFMB0jJ
agiX5hbOYn7o2SgHJQDOBuPF31fCLMTU3R+O3WwXjh15AWm03KLDnZs5wgSGzE0g
+XD47hhh4wWEx48NACToMqUHMcPIR+G9dV4w9npe8LSzO1HzuObAXwmD1JqbVQL0
63ZMGG0zD5Pqx2/ykWu5tA0oR3SAlgoKB8B+vuEmyyGxKtsBKT5U2f69diW/upH6
mzgRIHTnx965k9e4xt8NHZNa0jHBD/fV9hkne0wt7PSM2GotZmGcaQw5TqFmxC5p
uelakiUdMW2x9Ij2KGA1rTGaqDG8n9tWV7DaMDRTfm7/3Qg4ae6uSaQh07ELnzT0
Ixm9MLZk6AjpKUO0AcoL+vAuWuinNManAuuGoU9XZVUsUIyYdtUVDp1M4lPms4ut
8AboxSoPWSXXnQXaGnG6HcP3sn0KNGvB9j7OHLMFfE+EOtE2kp6Khm2zFUVu+IM7
CtuamA+1nb2koFh9BxD3WkiDf9wHBW/lyA1nn8vd5qK/Mn4MsUxKdW1mOfIISak/
Bvkm+lQH9LapjmNsT+iocD0a6ysH9nJN6/R94mxc1PrTo+T8sQqtffqEjfKZIS7J
qmL0FZQD8O0rRPi0TCudknt5TvTeEfZiB89I8/Doc3zJvN9o41QalhRQy0AKFO1N
fA==
-----END CERTIFICATE-----
subject=CN = ca.cloudstack.apache.org
issuer=CN = ca.cloudstack.apache.org
---
No client certificate CA names sent
Requested Signature Algorithms:
ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA1:RSA+SHA1
Shared Requested Signature Algorithms:
ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA-PSS+SHA256:RSA-PSS+SHA384:RSA-PSS+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512
Peer signing digest: SHA256
Peer signature type: RSA-PSS
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 2387 bytes and written 846 bytes
Verification error: unsupported certificate purpose
---
New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Server public key is 4096 bit
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 26 (unsupported certificate purpose)
---
140036465194816:error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad
certificate:ssl/record/rec_layer_s3.c:1543:SSL alert number 42
~~~
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
