UPDATE: The mgmt. logs show that the cloudmanagementserver.keystore is being 
generated with password “vmops.com”. When we test it, the password is correct.

keytool -list -keystore cloudmanagementserver.keystore -storepass vmops.com

Keystore type: JKS
Keystore provider: SUN

Your keystore contains 1 entry

mykey, May 29, 2017, PrivateKeyEntry,
Certificate fingerprint (SHA1): 
9C:91:BA:06:BB:4B:D1:B4:24:3F:8A:13:03:FD:6B:76:4E:9F:EA:29

LOGS

```1181097 2017-05-29 19:56:20,258 INFO  [c.c.s.ConfigurationServerImpl] 
(main:null) Processing updateSSLKeyStore
1181104 2017-05-29 19:56:20,261 INFO  [c.c.s.ConfigurationServerImpl] 
(main:null) SSL keystore located at 
/etc/cloudstack/management/cloudmanagementserver.keystore
1181105 2017-05-29 19:56:20,268 DEBUG [c.c.u.s.Script] (main:null) Executing: 
sudo keytool -genkey -keystore 
/etc/cloudstack/management/cloudmanagementserver.keystore -storepass vmops.com 
-keypass vmops.com -keyalg RSA -validity 3650 -dn1181105 ame cn="Cloudstack 
User",ou="localhost",o="localhost",c="Unknown"
1181106 2017-05-29 19:56:21,511 DEBUG [c.c.u.s.Script] (main:null) Execution is 
successful.
1181107 2017-05-29 19:56:21,512 INFO  [c.c.s.ConfigurationServerImpl] 
(main:null) Generated SSL keystore.
1181111 2017-05-29 19:56:21,552 TRACE [c.c.u.d.T.Statement] (main:null) 
Preparing: INSERT INTO configuration (configuration.instance, 
configuration.component, configuration.name, configuration.value, 
configuration.default_value, configur1181111 ation.description, 
configuration.category, configuration.is_dynamic, configuration.scope, 
configuration.updated) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
1181112 2017-05-29 19:56:21,556 TRACE [c.c.u.d.T.Transaction] (main:null) txn: 
DB Changes committed. Time = 6
1181113 2017-05-29 19:56:21,557 TRACE [c.c.u.d.T.Statement] (main:null) 
Closing: com.mysql.jdbc.JDBC4PreparedStatement@29f7a353: INSERT INTO 
configuration (configuration.instance, configuration.component, 
configuration.name, configuratio1181113 n.value, configuration.default_value, 
configuration.description, configuration.category, configuration.is_dynamic, 
configuration.scope, configuration.updated) VALUES (_binary'DEFAULT', 
_binary'management-server', _binary'ssl.keys1181113 tore', 
_binary'WKcZHRCDXhtqcLDK3ZSTyUYoZ3N81oiYbuool9WU2glPddGLrDZ4FyfQuThHXS4fH1fWPbKsjo9LgbvYUfFeAi4mdrfwOHXbCZPozlxPnBPp6gQh2eAKwB2eGytwZITxp0x9fqTv1bk0wWkOoEAEH37e6h+lWPvmv6eYvL85XGBcAJFVxRRHJwXl6u3r9dUobWF25lb6LLvi0P9kBpRFtb1181113
 
iv/p4DnCZgDaWrEwQCdtETwE6fMCVPxrUm/uEV4t0rbYApLU3ttpQolrR2USsBFlxFIxK4vYI9NeQK5dHK252s7pJriuQzwA14AyI36HLJZRkl/qVTu/ttIMqT9UcWvC5UZFC6iuv8E+eH5KFQhlx3HY/nvyQ8ADDgopmv0svmCBEHXP7mTN+K7JuPN8qBq9gaTNRNMAR33ma0r6lWD4K1+c7luKQS3hAMDMD1181113
 
KlkN09QTReNjydySuLnF6tlQQ1I1FPfa+eFE1VSj75ktd0nKgbxVnps79DceCFx428w6Jh9M7GRMvXykSHxI4SxrqXUeo4h1pX618r1kGLoG6SYPfuawRQqiC7ZouScyBK/z1egxR+xajXhMWxBtjfPxRx6fgwG4ouFT9RH6YhtFA+ppug+fhJvL2x9EZ78o6zDYpVmOOehLffHRdZHDtwtFg7srdmsNBHHg31181113
 
ukIYlF1MWvhnyUE0+nDu6HAgl4p+lt8lohxEFUoMa4bWaHVYOLMvgtqecdCU6CKwzGy3TiL/FI0/e1jUX5n9Y9FY5w7cjMnl2T95VKGXlnrBAyjo08Xt3MPM7xQduHM2kRYJsDXpJtokv+zWPmLUlPjTvnfLbobsBcNdrpU1ZPDPFXjPto5Ud3oPHzD1nu1DsK/Mfae6Lo8C0rl+I2kfCi0BZEKCJ0um8MzqZ1181113
 
0u32cbSXS/T7YU4jJNaiAujS3BDCb9L+u7irvhFTqe2PzDizV/YJa4y2YpQOpLv3m9JCsG0A1gvdZfOwDVUmg580TpomjPcmeH9bsOP8ESXa3d4Sm8tPHsetZrhnEtss0+OERTtNabFohAcjSKABIvcodq5HEqBzQtpyQN5MJA3YfarQMnPnAfIbjXszvqI04euBbQ2gyvDWoRIrcPjCjsUh9jRSUOMHaLVDg1181113
 
Cr4gK02+LhnGQeCXWjFo2i4FXCEajDgVpMHwYFhD1Uhdf3ttHBQNTaZ1ObTmP7E27H1VRpDvCP4omdFpxGV250ls5mj8uwMxrPchX7S7BlRbUT8XMyOm9RZiavRenOJp0xGWh+/YKKQFhDWTUyV3MiKah4iEPJ02fSpZ8pB+y4IYDO9ya+apLv8E4Bi90AZYOpWSYQr6t0wGcXpmY0JMyDoUxidh24HqO+/xJ1181113
 
6npD7SvEkoyFuXFAUaoamZ0vn/thxF5ZlTpA3Gfpny1j/4SLyV+GZWTK7wRhLjbZOonghXm2N6jpBYX/2dfdvoogTkEaxJHNitNyiZYwJm8fC7iaVmSB/OsJPs3Nd1ZmeljUCX6xsmQuiI47qDRmp1ZseQQM2HdGQqrK1X3EEcPa4BpgmVeOVRZ+uZ6gxAFYqQFAeKz6tUO6vQsOO/GAXwNwFSQDtBPUzC0ic1181113
 
P01cVkeUVlj56BaAp0Ia/tUbSDe5LhEdWo/C5innVeltJ//4Evhk3zHuyT1aiMJgWEwzZLgYuF1jHqK//ybSquXZJOyjrun+LVZs1vpp/qtRg3fk9qMjglnP0WIN44Kh/vnc7dDCrsyr/Ezu5Bh9oM+miNXVv5KH+2fMFYVzElB7Q0+kVmKkeucjtaBIlgtRjHFgK+uQzb+OZJSHByGx4Ci0jkDAxBNP4OsLf1181113
 
SC4pv8i8IppmcTwZGUbOye4x7QtvCo4JVzD5eZydzTD+GYFXgoeeFLveSm5DU44jeVjc3tKfcrhPeBc8pM3oGuSyvalYs7LGbf/9ozeg2vJWmQwEamPqHtq6ceg4qVRs2cIzaelc+yGT9BBOcUnHzCgr+gOC/F3lCJHbjUmyH52j4SLHZ3sZ6LB6CCVnkJ2Z7QYpCXe7lEIx3iX13+xBMSGwE/p4jKwJYBW0O1181113
 
k5dgTUG/QOEWhQVtdn33+uOH8X2/MMFMofJzaF7XC/mOCWtzSucTq1Obbng8ySAvp5ESIYfBtGeG3jgN+Pda7SzyDv8kQbyHFulyK+wzhYq/3lqOLnXP4P4nShHn6td7MunMJ6yPUX/swx5Pa35vpH/6IGxWHFMjJtzH/y8sliyy9SE3o4z6R3kVK+AWG7GvHRCdS9BNIJJyrjuEUuBcRlFpWPjEMw4n4ZSy81181113
 
+JJ0wlDpk8mlHflOLcuB72RAXyuHgsqi/8gIc5eM8Jf2968Av5WOeW0V4UHop0/wXOyXZf0bQisNzU0iBcRbszemlHPS3wCNmgkDIUgq2fkRZaLQO2hyurDzsEARQUE90XIKg7JWzkPXOMXZ6XR57INVL8kB3dPLkwzNOtC62YjbS8670SfWFjochnJSjlhAqFtVkuY8etyZ5mJRzrLl9b3MYlsmxHmdCCP5b1181113
 
JfC/Nq7ugKLq5RO/ACaz9Y934Q68K1lJYY2mHJHVnbeK9PewVfTjRG8A5NDhDBIduABoZi0DIAsz3lmEZGWG2V2GkIfGfha/fyvz9sasgO/Wrx4+Xpkt2aT+IU9f289ak9w1Bo/5DzdxmAZIHdonOUlb5TET5Eel114iW8mVO42snhKukWPMSayZVFPH9aCvdv3+5v7q5z8HPsnvSLpEiQ04LhVcGRHGV9+Qn1181113
 
EjcfFBTTIFjTFXb5d93+LPjoM6061kyqe60TqaTp1T0nxg4czhxTU1lpRkAFl88dyhJC4ZlJBsOp7tltzjyEa19Dw2nCySnyJ0dGVbZQ4vrIWkoQS4XxxbcWtP7gGLjmIL+bm3hEKSpotuiflAZlfbmqYILqHFzjkloYxSYjjJCoYzGaKCNCBT/ChGpDZamChdOVd2usCcSOqWZGa2dlQKp9kDkXQAMpefa8c1181113
 
YW0DRsgdqHLgXujEtHYbTrKxGhQGb4/yamDbZ+V4XvErxfAinlCUotrYUm7B71MSGgndXBTcq+/+oETKbaRdkx2qgbEqOZXUtv6uMrXIEvL3l+msb3VG1gz7d8cqWXs9VY/6HHCxNDKeldGMrGqTQPYznMzRiiCXyp+u+nOTf+gOzTfSl2BUm94jv/Hcjpj4zXnL1TV3KSjwBNRJFCIhh1sdqhvt0Xi9QJn2R1181113
 
QaLJ9y2TrBDGYEAmMgmXDVTa5CXBau0BwdiNiwXFvJkt7Vsvn7dRXTFyJ1YTO4c0H3n5B69QiWYaD3V7iRVmRzuQg8nSXTGh35Ol5Qh5lNI8byroC1gDMbxCjdJwR0fpUbn5fq4QFBgHDr9agFFf1kR2T4if+m4WFEGCP8bTwek5FL/K8CQLJ1JgCccWZy+2JZh2WQbVUvL7Bq5zcADUlM3ikHg4CrRZR8A3p1181113
 
4PM95HN/fu9+f2ZRj6ZF5TYDxk0weLJBVHbnph8mOLB0li5iPdUytNuKTCFJc1G1QUjB7fuV9g6l5mTG46wg0xU1mYJ9LB1oFix1lysnXBBaSYmnEzfiJbYqKqcvXJ+ycMPPAfmNDEyrzo0LTQtvyat7M2UGiAQ5cn0ahXEcZzSqmog/VE8xGJkBi1coP7Unp0lscuLgRxD7ev5Yru9iv6at0WSuvpZskVajq1181113
 
+B+R2XVEBMPFTjVynSr53fI46lhqr/YxzCEjVSuz0JDs5IBaahzT1U/wVKHbY6it2ZFNAwsvAG46CoYtJF06rYsRNu7Y+WbJC97xJciY2rpEq+Y2ZMo7Z20/hTSnBCg0IqUNxO+G/p9GxGRqRPA440Mcs5wz9ye7sYz0Z2uyZ5UUgLiNqL5lyw+dd9IxcPq5qaxJYRscsiL2T7IlYAJLrBn/9iz2gHVFEvZQF1181113
 1QRDJWQ3ApqmaQrT2ZyZINY4pDOkNshRnBOjyEFosp3DEvdQ==', null, _binary'SSL 
Keystore for the management servers', _binary'Hidden', 0, null, null)
1181114 2017-05-29 19:56:21,557 TRACE [c.c.u.d.T.Connection] (main:null) 
Closing DB connection: dbconn2026396576
1181115 2017-05-29 19:56:21,558 TRACE [c.c.u.d.T.Connection] (main:null) 
Creating a DB connection with  no txn:  for 0: dbconn1496793545. Stack: 
-TransactionLegacy.prepareStatement:467-TransactionLegacy.prepareAutoCloseStatement:460-Gene1181115
 
ricDaoBase.findById:1022-GenericDaoBase.findByIdIncludingRemoved:987-GenericDaoBase.persist:1435-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:57-DelegatingMethodAccessorImpl.invoke:43-Method.invoke:606-AopU1181115
 
tils.invokeJoinpointUsingReflection:317-ReflectiveMethodInvocation.invokeJoinpoint:183-ReflectiveMethodInvocation.proceed:150
1181116 2017-05-29 19:56:21,559 TRACE [c.c.u.d.T.Statement] (main:null) 
Preparing: SELECT configuration.instance, configuration.component, 
configuration.name, configuration.value, configuration.default_value, 
configuration.description, c1181116 onfiguration.category, 
configuration.is_dynamic, configuration.scope, configuration.updated FROM 
configuration WHERE configuration.name = ?
1181117 2017-05-29 19:56:21,562 TRACE [c.c.u.d.T.Statement] (main:null) 
Closing: com.mysql.jdbc.JDBC4PreparedStatement@3ac020e1: SELECT 
configuration.instance, configuration.component, configuration.name, 
configuration.value, configurati1181117 on.default_value, 
configuration.description, configuration.category, configuration.is_dynamic, 
configuration.scope, configuration.updated FROM configuration WHERE 
configuration.name = _binary'ssl.keystore'
1181118 2017-05-29 19:56:21,562 TRACE [c.c.u.d.T.Connection] (main:null) 
Closing DB connection: dbconn1496793545
1181119 2017-05-29 19:56:21,562 TRACE [c.c.u.d.T.Transaction] (main:null) 
Transaction is done
1181120 2017-05-29 19:56:21,562 INFO  [c.c.s.ConfigurationServerImpl] 
(main:null) Stored SSL keystore to database.

On 29/5/17, 8:12 pm, "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:

    Hi Rohit,
    Here’s the telnet responses for all scenarios are the same: The connection 
is closed immediately. 
    
    We found these in the trace logs
    
    1187417 2017-05-29 19:56:41,471 TRACE [c.c.u.n.NioConnection] 
(pool-3-thread-1:null) Keys Processing: 1
    1187419 2017-05-29 19:56:41,476 TRACE [c.c.u.n.NioConnection] 
(pool-3-thread-1:null) Connection accepted for 
Socket[addr=/192.168.12.63,port=35673,localport=8250]
    1187426 2017-05-29 19:56:41,498 TRACE [c.c.u.n.NioConnection] 
(pool-3-thread-1:null) Connection closed due to failure: Keystore was tampered 
with, or password was incorrect
    1187427 2017-05-29 19:56:41,499 TRACE [c.c.u.n.NioConnection] 
(pool-3-thread-1:null) Keys Done Processing.
    1187428 2017-05-29 19:56:41,499 TRACE [c.c.u.n.NioConnection] 
(pool-3-thread-1:null) Keys Processing: 0
    1187429 2017-05-29 19:56:41,499 TRACE [c.c.u.n.NioConnection] 
(pool-3-thread-1:null) Keys Done Processing.
    
    Is this referring to the cloudmanagementserver.keystore file and 
mysql.cloud.configuration ssl.keystore entry? These have been recreated many 
times and permissions appear correct.
    
    Hopefully this helps.
    
    Regards, Jason
    
    
    
    
    On 29/5/17, 5:35 pm, "Rohit Yadav" <rohit.ya...@shapeblue.com> wrote:
    
        Hi Jason,
        
        
        - Ensure that the 'host' global setting is the IP of the mgmt server or 
the VIP/LB in front of the mgmt server(s)
        
        - If you need to change the host global setting, restart mgmt server 
and destroy old SSVMs
        
        - If you try telnet on mgmt server (host) IP, on port 8250, does telnet 
immediately stop:
        
        telnet localhost 8250
        Trying 127.0.0.1...
        Connected to localhost.
        Escape character is '^]'.
        
        Try telnetting both from the mgmt server host (i.e. use localhost) and 
from the ssvm.
        
        - If telnet exists immediately, that means mgmt server host is unable 
to accept client connections on port 8250 because of which SSL handshake fails 
leading to the exceptions in the logs. The error message "Failed to send 
server's CLOSE message due to socket channel's failure." suggests that mgmt 
server was not able to accept the connection. The possible reasons for such a 
behaviour can be: (a) there is an issue with keystore setup in the mgmt server 
(the server component NioServer handles accept(), during which SSL handshake is 
performed and it fails), i.e. the SSLContext is not properly initialized so any 
client ssl handshake fails; (b) iptables or network/firewall rules blocking 
connections to port 8250 on the mgmt server host, either flush the 'filter' 
table or add ALLOW rules for 0.0.0.0/0 on port 8250.
        
        - In case the issue is with keystore setup, delete any *.keystore file 
from the mgmt server's classpath, i.e. search and remove any .keystore file 
from /etc/cloudstack/management and from /usr/share/cloudstack-management/, and 
restart the management server.
        
        Regards.
        
        
        ________________________________
        From: Jason Kinsella <ja...@cloudpeople.com.au>
        Sent: 29 May 2017 05:23:09
        To: users@cloudstack.apache.org
        Subject: Re: SSVM NIO SSL Handshake error
        
        UPDATE: I manually propagated a new systemvm.iso to secstorage and I 
can now ssh to the ssvm.
        
        Unfortunately same handshake errors in logs.
        
        On 28/5/17, 9:44 pm, "Jason Kinsella" <ja...@cloudpeople.com.au> wrote:
        
            Hi Rohit,
            I completed the test. The results are as follows:
        
            The ssh.publickey and ssh.privatekey deletions in DB triggered the 
keys to be recreated, but not in the /var/lib/cloudstack/management/.ssh 
folder. They were recreated in /var/cloudstack/management/.ssh folder.
        
            Here’s the log entry:
        
            2017-05-28 20:53:28,508 DEBUG [c.c.u.s.Script] 
(localhost-startStop-1:null) (logid:) Executing: /bin/bash -c if [ -f 
/var/cloudstack/management/.ssh/id_rsa ]; then rm -f 
/var/cloudstack/management/.ssh/id_rsa; fi; ssh-keygen -t rsa -N '' -f 
/var/cloudstack/management/.ssh/id_rsa –q
        
            I then destroyed the ssvm and after recreation could not ssh to it 
from management server due to key publickey error. I have always previously 
been able to ssh to the ssvm using the var/cloudstack/management/.ssh keys.
        
            I checked the management server logs and found entry about id_rsa 
files differ – injecting into systemvm.iso
        
            Hopefully this is helpful.
        
            Regards,
            Jason
        
        
            On 28/5/17, 5:29 pm, "Rohit Yadav" <rohit.ya...@shapeblue.com> 
wrote:
        
                Hi Jason,
        
        
                In your test environment that uses the same db, can you try to 
do a workaround-experiment from [1]:
        
        
                0) chmod +r and chown cloud:cloud relevant file and locations.
        
        
                1) Stop Management Server
        
                2) Delete SSH Keys in mysql Database: delete from configuration 
where name = "ssh.publickey" ; delete from configuration where name = 
"ssh.privatekey" ;
        
                3) Delete the SSH Keys rm 
/var/lib/cloudstack/management/.ssh/id_rsa.pub rm 
/var/lib/cloudstack/management/.ssh/id_rsa
        
                4) Start the Management Server - SSH Keys are generated and 
mysql entries inserted
        
        
        
                [1] http://markmail.org/message/zfjyd7s22itg7t7q
        
        
                Regards.
        
                ________________________________
                From: Jason Kinsella <ja...@cloudpeople.com.au>
                Sent: 27 May 2017 05:33:43
                To: users@cloudstack.apache.org
                Subject: Re: SSVM NIO SSL Handshake error
        
                Files are linked here.
        
                
https://dl.dropboxusercontent.com/u/10588206/acs492/managmenet-server-logs.tar.gz
                
https://dl.dropboxusercontent.com/u/10588206/acs492/systemvm.tar.gz
        
                Today we did a couple of additional tests that proved 
interesting. We’ve got a prod and a dev server. Both were upgraded last month. 
The prod has the error, but the dev is working. Everything was the same 
including CentOS 6.5.
        
                We restored the dev DB into the fresh CentOS7 box and it 
displayed the same problem. This would suggest an OS issue. Therefore, the 
converse should work. We restored the prod DB into the dev server and it 
continues to exhibit the problem.
        
                This suggests that we may have missed something in the 
migration between servers. Here’s steps:
        
                    Stop cloudstack-man service on broken box
                    Dump DB
                    Copy to new and restore
                    Copy db.properties & key files and update IP entry in 
db.properties
                    Update DB entry host to new IP
                    Delete DB ssl.keystore and keystore file
                    Destroy systemVMs in Vmware
                    Start cloudstack-man on new box
        
                    The /var/cloudstack/management/.ssh/ files are referenced 
when we ssh to ssvm from MS so they are correct. What about ssh.public and 
ssh.private in db.cloud.configuration table?
        
                    Regards,
                    Jason
        
                    On 25/5/17, 7:51 pm, "Rohit Yadav" 
<rohit.ya...@shapeblue.com> wrote:
        
                        Hi Jason,
        
        
                        Thanks for sharing the details. Yes, with the new setup 
please share with us the mgmt server logs and ssvm logs with TRACE enabled in 
the log4j configuration.
        
        
                        Regards.
        
                        ________________________________
                        From: Jason Kinsella <ja...@cloudpeople.com.au>
                        Sent: 25 May 2017 12:49:50
                        To: users@cloudstack.apache.org
                        Subject: Re: SSVM NIO SSL Handshake error
        
                        Hi Rohit,
                        API login – fixed.
        
                        Latest systemvmtemplate (shapeblue new) in place – no 
improvement
        
                        No loadbalancer or known service on MS port 8250
        
                        I am doing my testing now on a fresh install of CentOS7 
using shapeblue noredist with DB restored.
        
                        Hypervisor = vmware vsphere 6.5 with ESX 6.5
        
                        Systemvm.iso is dated today
        
                        All systemvms are exhibiting same behaviour.
        
                        Would any other logs help?
        
                        Regards,
                        Jason
        
                        On 25/5/17, 4:55 pm, "Rohit Yadav" 
<rohit.ya...@shapeblue.com> wrote:
        
                            Hi Jason,
        
        
                            The API login issue can be fixed by following this, 
which I believe you have already fixed: 
http://docs.cloudstack.apache.org/projects/cloudstack-administration/en/4.9/accounts.html#using-dynamic-roles
        
        
                            If not already in-use, can you try using the latest 
systemvmtemplate (for 4.6-4.9) from 
http://packages.shapeblue.com/systemvmtemplate/4.6/new.
        
        
                            Do you have a load-balancer on port 8250 on the 
management server(s), or any script/service that may be trying to perform a 
tcp-connect on mgmt server's port 8250?
        
        
                            When you upgrade can you make sure that both 
cloudstack-common and cloudstack-management packages are upgraded to 4.9.2.0? 
Also, what hypervisor(s) are you using?
        
        
                            The following error may hint that the jars on 
systemvms may not be updated, as one of the exception classes are missing:
        
        
                                2017-05-23 11:58:22,468 INFO  
[utils.exception.CSExceptionErrorCode] (main:null) Could not find exception: 
com.cloud.utils.exception.NioConnectionException in error code list for 
exceptions
        
        
                            Can you check that systemvm.iso are synced across 
hosts: (1) make sure cloudstack-common package is upgraded/updated to the same 
version as cloudstack-management (4.9.2.0), (2) if you're using vmware, delete 
this from the secondary storage, (3) for xenserver force reconnect on the host 
(from ui/api) or manually copy the scripts to xenserver host(s), (4) for kvm 
upgrade the cloudstack-common package.
        
        
                            Destroy all other systemvms and see if you can 
reproduce the issue?
        
        
                            Regards.
        
                            ________________________________
                            From: Jason Kinsella <ja...@cloudpeople.com.au>
                            Sent: 25 May 2017 09:32:25
                            To: users@cloudstack.apache.org
                            Subject: Re: SSVM NIO SSL Handshake error
        
                            Also, just wanted to mention that the symptoms we 
have with systemvms not connecting is described in the mail-list
        
                            CS 4.9 NIO Selector wait time PR-1601 - 
https://www.mail-archive.com/dev@cloudstack.apache.org/msg69154.html
        
                            The only difference is that this thread refers to 
KVM hosts not connecting.
        
                            I’ve tried most suggestions in this thread.
        
                            On 25/5/17, 1:51 pm, "Jason Kinsella" 
<ja...@cloudpeople.com.au> wrote:
        
                                Java versions are as follows:
        
                                MS: 1.7.0_141
                                SSVM: 1.7.0_85
        
                                Deleted keystore files (again) and restarted 
MS, then recreated the SSVM.
        
                                Errors from SSVM:/var/log/cloud.log
        
                                2017-05-25 03:01:28,757 INFO  
[utils.nio.NioClient] (main:null) Connecting to 192.168.12.5:8250
                                2017-05-25 03:01:29,293 WARN  [utils.nio.Link] 
(main:null) This SSL engine was forced to close inbound due to end of stream.
                                2017-05-25 03:01:29,293 ERROR [utils.nio.Link] 
(main:null) Failed to send server's CLOSE message due to socket channel's 
failure.
                                2017-05-25 03:01:29,294 ERROR 
[utils.nio.NioClient] (main:null) SSL Handshake failed while connecting to 
host: 192.168.12.5 port: 8250
                                2017-05-25 03:01:29,294 ERROR 
[utils.nio.NioConnection] (main:null) Unable to initialize the threads.
                                java.io.IOException: SSL Handshake failed while 
connecting to host: 192.168.12.5 port: 8250
                                     at 
com.cloud.utils.nio.NioClient.init(NioClient.java:67)
                                     at 
com.cloud.utils.nio.NioConnection.start(NioConnection.java:88)
                                     at 
com.cloud.agent.Agent.start(Agent.java:228)
                                     at 
com.cloud.agent.AgentShell.launchAgent(AgentShell.java:399)
                                     at 
com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:367)
                                     at 
com.cloud.agent.AgentShell.launchAgent(AgentShell.java:351)
                                     at 
com.cloud.agent.AgentShell.start(AgentShell.java:456)
                                     at 
com.cloud.agent.AgentShell.main(AgentShell.java:491)
        
                                Same SSL engine forced to close inbound due to 
end of stream
        
        
        
        
        
        
        
                                On 25/5/17, 1:51 am, "Rajani Karuturi" 
<raj...@apache.org> wrote:
        
                                    Can you check java version? Set the default 
java to 1.7 and delete keystore
                                    files and restart MS
        
                                    ~Rajani
        
                                    Sent from phone.
        
                                    On 24 May 2017 9:15 p.m., "Jason Kinsella" 
<ja...@cloudpeople.com.au> wrote:
        
                                    > I have now moved management server to a 
fresh CentOS7 server. But
                                    > unfortunately I’m getting the exact same 
SSL handshake error. Back to
                                    > square one.
                                    >
                                    > On 24/5/17, 11:40 pm, "Jason Kinsella" 
<ja...@cloudpeople.com.au> wrote:
                                    >
                                    >     Hi All,
                                    >     Based on the feedback it seems like 
the issue is related to CentOS
                                    > version, so I’ve built a new CentOS7 
Management server using Blueshape
                                    > noredist. I’ve restored the 4.9.2.0 DB 
into this server and
                                    > management-server.logs look clean on 
boot. The only problem is that I can’t
                                    > log into the webUI.
                                    >
                                    >     The logs show a successful login 
(user = kinsja), but the the API
                                    > command either is not allowed or doesn’t 
exist for the user. This means the
                                    > UI doesn’t load.
                                    >
                                    >     Anyone seen this with a restored DB?
                                    >
                                    >     2017-05-24 09:26:08,239 DEBUG 
[c.c.u.AccountManagerImpl]
                                    > (catalina-exec-17:ctx-ee2c5e26) 
(logid:a8ca5ee5) User: kinsja in domain 1
                                    > has successfully logged in
                                    >     2017-05-24 09:26:08,246 INFO  
[c.c.a.ApiServer] (catalina-exec-17:ctx-ee2c5e26)
                                    > (logid:a8ca5ee5) Current user logged in 
under  timezone
                                    >     2017-05-24 09:26:08,246 INFO  
[c.c.a.ApiServer] (catalina-exec-17:ctx-ee2c5e26)
                                    > (logid:a8ca5ee5) Timezone offset from UTC 
is: 0.0
                                    >     2017-05-24 09:26:08,251 DEBUG 
[c.c.a.ApiServlet] (catalina-exec-17:ctx-ee2c5e26)
                                    > (logid:a8ca5ee5) ===END===  192.168.10.38 
-- POST
                                    >     2017-05-24 09:26:08,320 DEBUG 
[c.c.a.ApiServlet] (catalina-exec-13:ctx-a1d38347)
                                    > (logid:3404c663) ===START===  
192.168.10.38 -- GET
                                    > 
command=listCapabilities&response=json&_=1495632368256
                                    >     2017-05-24 09:26:08,325 DEBUG 
[c.c.a.ApiServer]
                                    > (catalina-exec-13:ctx-a1d38347 
ctx-960796a5) (logid:3404c663) The user with
                                    > id:31 is not allowed to request the API 
command or the API command does not
                                    > exist: listCapabilities
                                    >
                                    >     Thanks
                                    >     Jason
                                    >
                                    >     From: Jason Kinsella 
<ja...@cloudpeople.com.au>
                                    >     Date: Tuesday, 23 May 2017 at 10:11 pm
                                    >     To: "users@cloudstack.apache.org" 
<users@cloudstack.apache.org>
                                    >     Subject: SSVM NIO SSL Handshake error
                                    >
                                    >     Hi,
                                    >     We recently upgraded from 4.5.0 to 
4.9.2.0 and encountered a problem
                                    > with the SSVM and Console Proxy. They 
cannot connect to the management
                                    > server. The SSVM cloud.log repeats this 
error every couple of seconds.
                                    >
                                    >     2017-05-23 11:58:22,461 INFO  
[utils.nio.NioClient] (main:null)
                                    > Connecting to 192.168.12.1:8250
                                    >     2017-05-23 11:58:22,465 WARN  
[utils.nio.Link] (main:null) This SSL
                                    > engine was forced to close inbound due to 
end of stream.
                                    >     2017-05-23 11:58:22,465 ERROR 
[utils.nio.Link] (main:null) Failed to
                                    > send server's CLOSE message due to socket 
channel's failure.
                                    >     2017-05-23 11:58:22,466 ERROR 
[utils.nio.NioClient] (main:null) SSL
                                    > Handshake failed while connecting to 
host: 192.168.12.1 port: 8250
                                    >     2017-05-23 11:58:22,466 ERROR 
[utils.nio.NioConnection] (main:null)
                                    > Unable to initialize the threads.
                                    >     java.io.IOException: SSL Handshake 
failed while connecting to host:
                                    > 192.168.12.1 port: 8250
                                    >                     at 
com.cloud.utils.nio.NioClient.
                                    > init(NioClient.java:67)
                                    >                     at 
com.cloud.utils.nio.NioConnection.start(
                                    > NioConnection.java:88)
                                    >                     at 
com.cloud.agent.Agent.start(Agent.java:237)
                                    >                     at 
com.cloud.agent.AgentShell.
                                    > launchAgent(AgentShell.java:399)
                                    >                     at 
com.cloud.agent.AgentShell.
                                    > 
launchAgentFromClassInfo(AgentShell.java:367)
                                    >                     at 
com.cloud.agent.AgentShell.
                                    > launchAgent(AgentShell.java:351)
                                    >                     at 
com.cloud.agent.AgentShell.
                                    > start(AgentShell.java:456)
                                    >                     at 
com.cloud.agent.AgentShell.
                                    > main(AgentShell.java:491)
                                    >     2017-05-23 11:58:22,468 INFO  
[utils.exception.CSExceptionErrorCode]
                                    > (main:null) Could not find exception: 
com.cloud.utils.exception.NioConnectionException
                                    > in error code list for exceptions
                                    >     2017-05-23 11:58:22,468 WARN  
[cloud.agent.Agent] (main:null) NIO
                                    > Connection Exception  
com.cloud.utils.exception.NioConnectionException:
                                    > SSL Handshake failed while connecting to 
host: 192.168.12.1 port: 8250
                                    >
                                    >     The setup is very simple. Single 
management server and ports are open.
                                    >
                                    >     Things checked / tried:
                                    >
                                    >     ·         Destroyed SSVM multiple 
times – still same problem.
                                    >
                                    >     ·         SSH to SSVM from MS using 
ssh -i /var/cloudstack/management/.ssh/id_rsa
                                    > -p 3922 root@IPADDRESS – PASS
                                    >
                                    >     ·         SSVM telnet on 8250 to MS – 
PASS
                                    >
                                    >     I’ve also tested a restore of the DB 
into our working development
                                    > 4.9.2.0 server. It also exhibits the 
handshake errors, so most likely DB
                                    > related.
                                    >
                                    >     I’ve used up all my skills. Please 
help
                                    >
                                    >     Regards,
                                    >     Jason
                                    >
                                    >
                                    >
                                    >
        
        
        
        
        
                            rohit.ya...@shapeblue.com
                            www.shapeblue.com<http://www.shapeblue.com>
                            53 Chandos Place, Covent Garden, London  WC2N 4HSUK
                            @shapeblue
        
        
        
        
        
        
                        rohit.ya...@shapeblue.com
                        www.shapeblue.com<http://www.shapeblue.com>
                        53 Chandos Place, Covent Garden, London  WC2N 4HSUK
                        @shapeblue
        
        
        
        
        
        
        
        
                rohit.ya...@shapeblue.com
                www.shapeblue.com<http://www.shapeblue.com>
                53 Chandos Place, Covent Garden, London  WC2N 4HSUK
                @shapeblue
        
        
        
        
        
        
        
        
        rohit.ya...@shapeblue.com 
        www.shapeblue.com
        53 Chandos Place, Covent Garden, London  WC2N 4HSUK
        @shapeblue
          
         
        
        
    
    

Reply via email to