could you try this command from the ssvm? openssl s_client -connect <mgmt ip>:8250
-- Erik On Mon, May 29, 2017 at 12:26 PM, Jason Kinsella <ja...@cloudpeople.com.au> wrote: > UPDATE: The mgmt. logs show that the cloudmanagementserver.keystore is being > generated with password “vmops.com”. When we test it, the password is correct. > > keytool -list -keystore cloudmanagementserver.keystore -storepass vmops.com > > Keystore type: JKS > Keystore provider: SUN > > Your keystore contains 1 entry > > mykey, May 29, 2017, PrivateKeyEntry, > Certificate fingerprint (SHA1): > 9C:91:BA:06:BB:4B:D1:B4:24:3F:8A:13:03:FD:6B:76:4E:9F:EA:29 > > LOGS > > ```1181097 2017-05-29 19:56:20,258 INFO [c.c.s.ConfigurationServerImpl] > (main:null) Processing updateSSLKeyStore > 1181104 2017-05-29 19:56:20,261 INFO [c.c.s.ConfigurationServerImpl] > (main:null) SSL keystore located at > /etc/cloudstack/management/cloudmanagementserver.keystore > 1181105 2017-05-29 19:56:20,268 DEBUG [c.c.u.s.Script] (main:null) Executing: > sudo keytool -genkey -keystore > /etc/cloudstack/management/cloudmanagementserver.keystore -storepass > vmops.com -keypass vmops.com -keyalg RSA -validity 3650 -dn1181105 ame > cn="Cloudstack User",ou="localhost",o="localhost",c="Unknown" > 1181106 2017-05-29 19:56:21,511 DEBUG [c.c.u.s.Script] (main:null) Execution > is successful. > 1181107 2017-05-29 19:56:21,512 INFO [c.c.s.ConfigurationServerImpl] > (main:null) Generated SSL keystore. > 1181111 2017-05-29 19:56:21,552 TRACE [c.c.u.d.T.Statement] (main:null) > Preparing: INSERT INTO configuration (configuration.instance, > configuration.component, configuration.name, configuration.value, > configuration.default_value, configur1181111 ation.description, > configuration.category, configuration.is_dynamic, configuration.scope, > configuration.updated) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) > 1181112 2017-05-29 19:56:21,556 TRACE [c.c.u.d.T.Transaction] (main:null) > txn: DB Changes committed. Time = 6 > 1181113 2017-05-29 19:56:21,557 TRACE [c.c.u.d.T.Statement] (main:null) > Closing: com.mysql.jdbc.JDBC4PreparedStatement@29f7a353: INSERT INTO > configuration (configuration.instance, configuration.component, > configuration.name, configuratio1181113 n.value, configuration.default_value, > configuration.description, configuration.category, configuration.is_dynamic, > configuration.scope, configuration.updated) VALUES (_binary'DEFAULT', > _binary'management-server', _binary'ssl.keys1181113 tore', > _binary'WKcZHRCDXhtqcLDK3ZSTyUYoZ3N81oiYbuool9WU2glPddGLrDZ4FyfQuThHXS4fH1fWPbKsjo9LgbvYUfFeAi4mdrfwOHXbCZPozlxPnBPp6gQh2eAKwB2eGytwZITxp0x9fqTv1bk0wWkOoEAEH37e6h+lWPvmv6eYvL85XGBcAJFVxRRHJwXl6u3r9dUobWF25lb6LLvi0P9kBpRFtb1181113 > > iv/p4DnCZgDaWrEwQCdtETwE6fMCVPxrUm/uEV4t0rbYApLU3ttpQolrR2USsBFlxFIxK4vYI9NeQK5dHK252s7pJriuQzwA14AyI36HLJZRkl/qVTu/ttIMqT9UcWvC5UZFC6iuv8E+eH5KFQhlx3HY/nvyQ8ADDgopmv0svmCBEHXP7mTN+K7JuPN8qBq9gaTNRNMAR33ma0r6lWD4K1+c7luKQS3hAMDMD1181113 > > KlkN09QTReNjydySuLnF6tlQQ1I1FPfa+eFE1VSj75ktd0nKgbxVnps79DceCFx428w6Jh9M7GRMvXykSHxI4SxrqXUeo4h1pX618r1kGLoG6SYPfuawRQqiC7ZouScyBK/z1egxR+xajXhMWxBtjfPxRx6fgwG4ouFT9RH6YhtFA+ppug+fhJvL2x9EZ78o6zDYpVmOOehLffHRdZHDtwtFg7srdmsNBHHg31181113 > > ukIYlF1MWvhnyUE0+nDu6HAgl4p+lt8lohxEFUoMa4bWaHVYOLMvgtqecdCU6CKwzGy3TiL/FI0/e1jUX5n9Y9FY5w7cjMnl2T95VKGXlnrBAyjo08Xt3MPM7xQduHM2kRYJsDXpJtokv+zWPmLUlPjTvnfLbobsBcNdrpU1ZPDPFXjPto5Ud3oPHzD1nu1DsK/Mfae6Lo8C0rl+I2kfCi0BZEKCJ0um8MzqZ1181113 > > 0u32cbSXS/T7YU4jJNaiAujS3BDCb9L+u7irvhFTqe2PzDizV/YJa4y2YpQOpLv3m9JCsG0A1gvdZfOwDVUmg580TpomjPcmeH9bsOP8ESXa3d4Sm8tPHsetZrhnEtss0+OERTtNabFohAcjSKABIvcodq5HEqBzQtpyQN5MJA3YfarQMnPnAfIbjXszvqI04euBbQ2gyvDWoRIrcPjCjsUh9jRSUOMHaLVDg1181113 > > Cr4gK02+LhnGQeCXWjFo2i4FXCEajDgVpMHwYFhD1Uhdf3ttHBQNTaZ1ObTmP7E27H1VRpDvCP4omdFpxGV250ls5mj8uwMxrPchX7S7BlRbUT8XMyOm9RZiavRenOJp0xGWh+/YKKQFhDWTUyV3MiKah4iEPJ02fSpZ8pB+y4IYDO9ya+apLv8E4Bi90AZYOpWSYQr6t0wGcXpmY0JMyDoUxidh24HqO+/xJ1181113 > > 6npD7SvEkoyFuXFAUaoamZ0vn/thxF5ZlTpA3Gfpny1j/4SLyV+GZWTK7wRhLjbZOonghXm2N6jpBYX/2dfdvoogTkEaxJHNitNyiZYwJm8fC7iaVmSB/OsJPs3Nd1ZmeljUCX6xsmQuiI47qDRmp1ZseQQM2HdGQqrK1X3EEcPa4BpgmVeOVRZ+uZ6gxAFYqQFAeKz6tUO6vQsOO/GAXwNwFSQDtBPUzC0ic1181113 > > P01cVkeUVlj56BaAp0Ia/tUbSDe5LhEdWo/C5innVeltJ//4Evhk3zHuyT1aiMJgWEwzZLgYuF1jHqK//ybSquXZJOyjrun+LVZs1vpp/qtRg3fk9qMjglnP0WIN44Kh/vnc7dDCrsyr/Ezu5Bh9oM+miNXVv5KH+2fMFYVzElB7Q0+kVmKkeucjtaBIlgtRjHFgK+uQzb+OZJSHByGx4Ci0jkDAxBNP4OsLf1181113 > > SC4pv8i8IppmcTwZGUbOye4x7QtvCo4JVzD5eZydzTD+GYFXgoeeFLveSm5DU44jeVjc3tKfcrhPeBc8pM3oGuSyvalYs7LGbf/9ozeg2vJWmQwEamPqHtq6ceg4qVRs2cIzaelc+yGT9BBOcUnHzCgr+gOC/F3lCJHbjUmyH52j4SLHZ3sZ6LB6CCVnkJ2Z7QYpCXe7lEIx3iX13+xBMSGwE/p4jKwJYBW0O1181113 > > k5dgTUG/QOEWhQVtdn33+uOH8X2/MMFMofJzaF7XC/mOCWtzSucTq1Obbng8ySAvp5ESIYfBtGeG3jgN+Pda7SzyDv8kQbyHFulyK+wzhYq/3lqOLnXP4P4nShHn6td7MunMJ6yPUX/swx5Pa35vpH/6IGxWHFMjJtzH/y8sliyy9SE3o4z6R3kVK+AWG7GvHRCdS9BNIJJyrjuEUuBcRlFpWPjEMw4n4ZSy81181113 > > +JJ0wlDpk8mlHflOLcuB72RAXyuHgsqi/8gIc5eM8Jf2968Av5WOeW0V4UHop0/wXOyXZf0bQisNzU0iBcRbszemlHPS3wCNmgkDIUgq2fkRZaLQO2hyurDzsEARQUE90XIKg7JWzkPXOMXZ6XR57INVL8kB3dPLkwzNOtC62YjbS8670SfWFjochnJSjlhAqFtVkuY8etyZ5mJRzrLl9b3MYlsmxHmdCCP5b1181113 > > JfC/Nq7ugKLq5RO/ACaz9Y934Q68K1lJYY2mHJHVnbeK9PewVfTjRG8A5NDhDBIduABoZi0DIAsz3lmEZGWG2V2GkIfGfha/fyvz9sasgO/Wrx4+Xpkt2aT+IU9f289ak9w1Bo/5DzdxmAZIHdonOUlb5TET5Eel114iW8mVO42snhKukWPMSayZVFPH9aCvdv3+5v7q5z8HPsnvSLpEiQ04LhVcGRHGV9+Qn1181113 > > EjcfFBTTIFjTFXb5d93+LPjoM6061kyqe60TqaTp1T0nxg4czhxTU1lpRkAFl88dyhJC4ZlJBsOp7tltzjyEa19Dw2nCySnyJ0dGVbZQ4vrIWkoQS4XxxbcWtP7gGLjmIL+bm3hEKSpotuiflAZlfbmqYILqHFzjkloYxSYjjJCoYzGaKCNCBT/ChGpDZamChdOVd2usCcSOqWZGa2dlQKp9kDkXQAMpefa8c1181113 > > YW0DRsgdqHLgXujEtHYbTrKxGhQGb4/yamDbZ+V4XvErxfAinlCUotrYUm7B71MSGgndXBTcq+/+oETKbaRdkx2qgbEqOZXUtv6uMrXIEvL3l+msb3VG1gz7d8cqWXs9VY/6HHCxNDKeldGMrGqTQPYznMzRiiCXyp+u+nOTf+gOzTfSl2BUm94jv/Hcjpj4zXnL1TV3KSjwBNRJFCIhh1sdqhvt0Xi9QJn2R1181113 > > QaLJ9y2TrBDGYEAmMgmXDVTa5CXBau0BwdiNiwXFvJkt7Vsvn7dRXTFyJ1YTO4c0H3n5B69QiWYaD3V7iRVmRzuQg8nSXTGh35Ol5Qh5lNI8byroC1gDMbxCjdJwR0fpUbn5fq4QFBgHDr9agFFf1kR2T4if+m4WFEGCP8bTwek5FL/K8CQLJ1JgCccWZy+2JZh2WQbVUvL7Bq5zcADUlM3ikHg4CrRZR8A3p1181113 > > 4PM95HN/fu9+f2ZRj6ZF5TYDxk0weLJBVHbnph8mOLB0li5iPdUytNuKTCFJc1G1QUjB7fuV9g6l5mTG46wg0xU1mYJ9LB1oFix1lysnXBBaSYmnEzfiJbYqKqcvXJ+ycMPPAfmNDEyrzo0LTQtvyat7M2UGiAQ5cn0ahXEcZzSqmog/VE8xGJkBi1coP7Unp0lscuLgRxD7ev5Yru9iv6at0WSuvpZskVajq1181113 > > +B+R2XVEBMPFTjVynSr53fI46lhqr/YxzCEjVSuz0JDs5IBaahzT1U/wVKHbY6it2ZFNAwsvAG46CoYtJF06rYsRNu7Y+WbJC97xJciY2rpEq+Y2ZMo7Z20/hTSnBCg0IqUNxO+G/p9GxGRqRPA440Mcs5wz9ye7sYz0Z2uyZ5UUgLiNqL5lyw+dd9IxcPq5qaxJYRscsiL2T7IlYAJLrBn/9iz2gHVFEvZQF1181113 > 1QRDJWQ3ApqmaQrT2ZyZINY4pDOkNshRnBOjyEFosp3DEvdQ==', null, _binary'SSL > Keystore for the management servers', _binary'Hidden', 0, null, null) > 1181114 2017-05-29 19:56:21,557 TRACE [c.c.u.d.T.Connection] (main:null) > Closing DB connection: dbconn2026396576 > 1181115 2017-05-29 19:56:21,558 TRACE [c.c.u.d.T.Connection] (main:null) > Creating a DB connection with no txn: for 0: dbconn1496793545. Stack: > -TransactionLegacy.prepareStatement:467-TransactionLegacy.prepareAutoCloseStatement:460-Gene1181115 > > ricDaoBase.findById:1022-GenericDaoBase.findByIdIncludingRemoved:987-GenericDaoBase.persist:1435-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:57-DelegatingMethodAccessorImpl.invoke:43-Method.invoke:606-AopU1181115 > > tils.invokeJoinpointUsingReflection:317-ReflectiveMethodInvocation.invokeJoinpoint:183-ReflectiveMethodInvocation.proceed:150 > 1181116 2017-05-29 19:56:21,559 TRACE [c.c.u.d.T.Statement] (main:null) > Preparing: SELECT configuration.instance, configuration.component, > configuration.name, configuration.value, configuration.default_value, > configuration.description, c1181116 onfiguration.category, > configuration.is_dynamic, configuration.scope, configuration.updated FROM > configuration WHERE configuration.name = ? > 1181117 2017-05-29 19:56:21,562 TRACE [c.c.u.d.T.Statement] (main:null) > Closing: com.mysql.jdbc.JDBC4PreparedStatement@3ac020e1: SELECT > configuration.instance, configuration.component, configuration.name, > configuration.value, configurati1181117 on.default_value, > configuration.description, configuration.category, configuration.is_dynamic, > configuration.scope, configuration.updated FROM configuration WHERE > configuration.name = _binary'ssl.keystore' > 1181118 2017-05-29 19:56:21,562 TRACE [c.c.u.d.T.Connection] (main:null) > Closing DB connection: dbconn1496793545 > 1181119 2017-05-29 19:56:21,562 TRACE [c.c.u.d.T.Transaction] (main:null) > Transaction is done > 1181120 2017-05-29 19:56:21,562 INFO [c.c.s.ConfigurationServerImpl] > (main:null) Stored SSL keystore to database. > > On 29/5/17, 8:12 pm, "Jason Kinsella" <ja...@cloudpeople.com.au> wrote: > > Hi Rohit, > Here’s the telnet responses for all scenarios are the same: The > connection is closed immediately. > > We found these in the trace logs > > 1187417 2017-05-29 19:56:41,471 TRACE [c.c.u.n.NioConnection] > (pool-3-thread-1:null) Keys Processing: 1 > 1187419 2017-05-29 19:56:41,476 TRACE [c.c.u.n.NioConnection] > (pool-3-thread-1:null) Connection accepted for > Socket[addr=/192.168.12.63,port=35673,localport=8250] > 1187426 2017-05-29 19:56:41,498 TRACE [c.c.u.n.NioConnection] > (pool-3-thread-1:null) Connection closed due to failure: Keystore was > tampered with, or password was incorrect > 1187427 2017-05-29 19:56:41,499 TRACE [c.c.u.n.NioConnection] > (pool-3-thread-1:null) Keys Done Processing. > 1187428 2017-05-29 19:56:41,499 TRACE [c.c.u.n.NioConnection] > (pool-3-thread-1:null) Keys Processing: 0 > 1187429 2017-05-29 19:56:41,499 TRACE [c.c.u.n.NioConnection] > (pool-3-thread-1:null) Keys Done Processing. > > Is this referring to the cloudmanagementserver.keystore file and > mysql.cloud.configuration ssl.keystore entry? These have been recreated many > times and permissions appear correct. > > Hopefully this helps. > > Regards, Jason > > > > > On 29/5/17, 5:35 pm, "Rohit Yadav" <rohit.ya...@shapeblue.com> wrote: > > Hi Jason, > > > - Ensure that the 'host' global setting is the IP of the mgmt server > or the VIP/LB in front of the mgmt server(s) > > - If you need to change the host global setting, restart mgmt server > and destroy old SSVMs > > - If you try telnet on mgmt server (host) IP, on port 8250, does > telnet immediately stop: > > telnet localhost 8250 > Trying 127.0.0.1... > Connected to localhost. > Escape character is '^]'. > > Try telnetting both from the mgmt server host (i.e. use localhost) > and from the ssvm. > > - If telnet exists immediately, that means mgmt server host is unable > to accept client connections on port 8250 because of which SSL handshake > fails leading to the exceptions in the logs. The error message "Failed to > send server's CLOSE message due to socket channel's failure." suggests that > mgmt server was not able to accept the connection. The possible reasons for > such a behaviour can be: (a) there is an issue with keystore setup in the > mgmt server (the server component NioServer handles accept(), during which > SSL handshake is performed and it fails), i.e. the SSLContext is not properly > initialized so any client ssl handshake fails; (b) iptables or > network/firewall rules blocking connections to port 8250 on the mgmt server > host, either flush the 'filter' table or add ALLOW rules for 0.0.0.0/0 on > port 8250. > > - In case the issue is with keystore setup, delete any *.keystore > file from the mgmt server's classpath, i.e. search and remove any .keystore > file from /etc/cloudstack/management and from > /usr/share/cloudstack-management/, and restart the management server. > > Regards. > > > ________________________________ > From: Jason Kinsella <ja...@cloudpeople.com.au> > Sent: 29 May 2017 05:23:09 > To: users@cloudstack.apache.org > Subject: Re: SSVM NIO SSL Handshake error > > UPDATE: I manually propagated a new systemvm.iso to secstorage and I > can now ssh to the ssvm. > > Unfortunately same handshake errors in logs. > > On 28/5/17, 9:44 pm, "Jason Kinsella" <ja...@cloudpeople.com.au> > wrote: > > Hi Rohit, > I completed the test. The results are as follows: > > The ssh.publickey and ssh.privatekey deletions in DB triggered > the keys to be recreated, but not in the /var/lib/cloudstack/management/.ssh > folder. They were recreated in /var/cloudstack/management/.ssh folder. > > Here’s the log entry: > > 2017-05-28 20:53:28,508 DEBUG [c.c.u.s.Script] > (localhost-startStop-1:null) (logid:) Executing: /bin/bash -c if [ -f > /var/cloudstack/management/.ssh/id_rsa ]; then rm -f > /var/cloudstack/management/.ssh/id_rsa; fi; ssh-keygen -t rsa -N '' -f > /var/cloudstack/management/.ssh/id_rsa –q > > I then destroyed the ssvm and after recreation could not ssh to > it from management server due to key publickey error. I have always > previously been able to ssh to the ssvm using the > var/cloudstack/management/.ssh keys. > > I checked the management server logs and found entry about id_rsa > files differ – injecting into systemvm.iso > > Hopefully this is helpful. > > Regards, > Jason > > > On 28/5/17, 5:29 pm, "Rohit Yadav" <rohit.ya...@shapeblue.com> > wrote: > > Hi Jason, > > > In your test environment that uses the same db, can you try > to do a workaround-experiment from [1]: > > > 0) chmod +r and chown cloud:cloud relevant file and locations. > > > 1) Stop Management Server > > 2) Delete SSH Keys in mysql Database: delete from > configuration where name = "ssh.publickey" ; delete from configuration where > name = "ssh.privatekey" ; > > 3) Delete the SSH Keys rm > /var/lib/cloudstack/management/.ssh/id_rsa.pub rm > /var/lib/cloudstack/management/.ssh/id_rsa > > 4) Start the Management Server - SSH Keys are generated and > mysql entries inserted > > > > [1] http://markmail.org/message/zfjyd7s22itg7t7q > > > Regards. > > ________________________________ > From: Jason Kinsella <ja...@cloudpeople.com.au> > Sent: 27 May 2017 05:33:43 > To: users@cloudstack.apache.org > Subject: Re: SSVM NIO SSL Handshake error > > Files are linked here. > > > https://dl.dropboxusercontent.com/u/10588206/acs492/managmenet-server-logs.tar.gz > > https://dl.dropboxusercontent.com/u/10588206/acs492/systemvm.tar.gz > > Today we did a couple of additional tests that proved > interesting. We’ve got a prod and a dev server. Both were upgraded last > month. The prod has the error, but the dev is working. Everything was the > same including CentOS 6.5. > > We restored the dev DB into the fresh CentOS7 box and it > displayed the same problem. This would suggest an OS issue. Therefore, the > converse should work. We restored the prod DB into the dev server and it > continues to exhibit the problem. > > This suggests that we may have missed something in the > migration between servers. Here’s steps: > > Stop cloudstack-man service on broken box > Dump DB > Copy to new and restore > Copy db.properties & key files and update IP entry in > db.properties > Update DB entry host to new IP > Delete DB ssl.keystore and keystore file > Destroy systemVMs in Vmware > Start cloudstack-man on new box > > The /var/cloudstack/management/.ssh/ files are referenced > when we ssh to ssvm from MS so they are correct. What about ssh.public and > ssh.private in db.cloud.configuration table? > > Regards, > Jason > > On 25/5/17, 7:51 pm, "Rohit Yadav" > <rohit.ya...@shapeblue.com> wrote: > > Hi Jason, > > > Thanks for sharing the details. Yes, with the new > setup please share with us the mgmt server logs and ssvm logs with TRACE > enabled in the log4j configuration. > > > Regards. > > ________________________________ > From: Jason Kinsella <ja...@cloudpeople.com.au> > Sent: 25 May 2017 12:49:50 > To: users@cloudstack.apache.org > Subject: Re: SSVM NIO SSL Handshake error > > Hi Rohit, > API login – fixed. > > Latest systemvmtemplate (shapeblue new) in place – no > improvement > > No loadbalancer or known service on MS port 8250 > > I am doing my testing now on a fresh install of > CentOS7 using shapeblue noredist with DB restored. > > Hypervisor = vmware vsphere 6.5 with ESX 6.5 > > Systemvm.iso is dated today > > All systemvms are exhibiting same behaviour. > > Would any other logs help? > > Regards, > Jason > > On 25/5/17, 4:55 pm, "Rohit Yadav" > <rohit.ya...@shapeblue.com> wrote: > > Hi Jason, > > > The API login issue can be fixed by following > this, which I believe you have already fixed: > http://docs.cloudstack.apache.org/projects/cloudstack-administration/en/4.9/accounts.html#using-dynamic-roles > > > If not already in-use, can you try using the > latest systemvmtemplate (for 4.6-4.9) from > http://packages.shapeblue.com/systemvmtemplate/4.6/new. > > > Do you have a load-balancer on port 8250 on the > management server(s), or any script/service that may be trying to perform a > tcp-connect on mgmt server's port 8250? > > > When you upgrade can you make sure that both > cloudstack-common and cloudstack-management packages are upgraded to 4.9.2.0? > Also, what hypervisor(s) are you using? > > > The following error may hint that the jars on > systemvms may not be updated, as one of the exception classes are missing: > > > 2017-05-23 11:58:22,468 INFO > [utils.exception.CSExceptionErrorCode] (main:null) Could not find exception: > com.cloud.utils.exception.NioConnectionException in error code list for > exceptions > > > Can you check that systemvm.iso are synced across > hosts: (1) make sure cloudstack-common package is upgraded/updated to the > same version as cloudstack-management (4.9.2.0), (2) if you're using vmware, > delete this from the secondary storage, (3) for xenserver force reconnect on > the host (from ui/api) or manually copy the scripts to xenserver host(s), (4) > for kvm upgrade the cloudstack-common package. > > > Destroy all other systemvms and see if you can > reproduce the issue? > > > Regards. > > ________________________________ > From: Jason Kinsella <ja...@cloudpeople.com.au> > Sent: 25 May 2017 09:32:25 > To: users@cloudstack.apache.org > Subject: Re: SSVM NIO SSL Handshake error > > Also, just wanted to mention that the symptoms we > have with systemvms not connecting is described in the mail-list > > CS 4.9 NIO Selector wait time PR-1601 - > https://www.mail-archive.com/dev@cloudstack.apache.org/msg69154.html > > The only difference is that this thread refers to > KVM hosts not connecting. > > I’ve tried most suggestions in this thread. > > On 25/5/17, 1:51 pm, "Jason Kinsella" > <ja...@cloudpeople.com.au> wrote: > > Java versions are as follows: > > MS: 1.7.0_141 > SSVM: 1.7.0_85 > > Deleted keystore files (again) and restarted > MS, then recreated the SSVM. > > Errors from SSVM:/var/log/cloud.log > > 2017-05-25 03:01:28,757 INFO > [utils.nio.NioClient] (main:null) Connecting to 192.168.12.5:8250 > 2017-05-25 03:01:29,293 WARN > [utils.nio.Link] (main:null) This SSL engine was forced to close inbound due > to end of stream. > 2017-05-25 03:01:29,293 ERROR > [utils.nio.Link] (main:null) Failed to send server's CLOSE message due to > socket channel's failure. > 2017-05-25 03:01:29,294 ERROR > [utils.nio.NioClient] (main:null) SSL Handshake failed while connecting to > host: 192.168.12.5 port: 8250 > 2017-05-25 03:01:29,294 ERROR > [utils.nio.NioConnection] (main:null) Unable to initialize the threads. > java.io.IOException: SSL Handshake failed > while connecting to host: 192.168.12.5 port: 8250 > at > com.cloud.utils.nio.NioClient.init(NioClient.java:67) > at > com.cloud.utils.nio.NioConnection.start(NioConnection.java:88) > at > com.cloud.agent.Agent.start(Agent.java:228) > at > com.cloud.agent.AgentShell.launchAgent(AgentShell.java:399) > at > com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:367) > at > com.cloud.agent.AgentShell.launchAgent(AgentShell.java:351) > at > com.cloud.agent.AgentShell.start(AgentShell.java:456) > at > com.cloud.agent.AgentShell.main(AgentShell.java:491) > > Same SSL engine forced to close inbound due > to end of stream > > > > > > > > On 25/5/17, 1:51 am, "Rajani Karuturi" > <raj...@apache.org> wrote: > > Can you check java version? Set the > default java to 1.7 and delete keystore > files and restart MS > > ~Rajani > > Sent from phone. > > On 24 May 2017 9:15 p.m., "Jason > Kinsella" <ja...@cloudpeople.com.au> wrote: > > > I have now moved management server to a > fresh CentOS7 server. But > > unfortunately I’m getting the exact > same SSL handshake error. Back to > > square one. > > > > On 24/5/17, 11:40 pm, "Jason Kinsella" > <ja...@cloudpeople.com.au> wrote: > > > > Hi All, > > Based on the feedback it seems like > the issue is related to CentOS > > version, so I’ve built a new CentOS7 > Management server using Blueshape > > noredist. I’ve restored the 4.9.2.0 DB > into this server and > > management-server.logs look clean on > boot. The only problem is that I can’t > > log into the webUI. > > > > The logs show a successful login > (user = kinsja), but the the API > > command either is not allowed or > doesn’t exist for the user. This means the > > UI doesn’t load. > > > > Anyone seen this with a restored DB? > > > > 2017-05-24 09:26:08,239 DEBUG > [c.c.u.AccountManagerImpl] > > (catalina-exec-17:ctx-ee2c5e26) > (logid:a8ca5ee5) User: kinsja in domain 1 > > has successfully logged in > > 2017-05-24 09:26:08,246 INFO > [c.c.a.ApiServer] (catalina-exec-17:ctx-ee2c5e26) > > (logid:a8ca5ee5) Current user logged in > under timezone > > 2017-05-24 09:26:08,246 INFO > [c.c.a.ApiServer] (catalina-exec-17:ctx-ee2c5e26) > > (logid:a8ca5ee5) Timezone offset from > UTC is: 0.0 > > 2017-05-24 09:26:08,251 DEBUG > [c.c.a.ApiServlet] (catalina-exec-17:ctx-ee2c5e26) > > (logid:a8ca5ee5) ===END=== > 192.168.10.38 -- POST > > 2017-05-24 09:26:08,320 DEBUG > [c.c.a.ApiServlet] (catalina-exec-13:ctx-a1d38347) > > (logid:3404c663) ===START=== > 192.168.10.38 -- GET > > > command=listCapabilities&response=json&_=1495632368256 > > 2017-05-24 09:26:08,325 DEBUG > [c.c.a.ApiServer] > > (catalina-exec-13:ctx-a1d38347 > ctx-960796a5) (logid:3404c663) The user with > > id:31 is not allowed to request the API > command or the API command does not > > exist: listCapabilities > > > > Thanks > > Jason > > > > From: Jason Kinsella > <ja...@cloudpeople.com.au> > > Date: Tuesday, 23 May 2017 at 10:11 > pm > > To: "users@cloudstack.apache.org" > <users@cloudstack.apache.org> > > Subject: SSVM NIO SSL Handshake > error > > > > Hi, > > We recently upgraded from 4.5.0 to > 4.9.2.0 and encountered a problem > > with the SSVM and Console Proxy. They > cannot connect to the management > > server. The SSVM cloud.log repeats this > error every couple of seconds. > > > > 2017-05-23 11:58:22,461 INFO > [utils.nio.NioClient] (main:null) > > Connecting to 192.168.12.1:8250 > > 2017-05-23 11:58:22,465 WARN > [utils.nio.Link] (main:null) This SSL > > engine was forced to close inbound due > to end of stream. > > 2017-05-23 11:58:22,465 ERROR > [utils.nio.Link] (main:null) Failed to > > send server's CLOSE message due to > socket channel's failure. > > 2017-05-23 11:58:22,466 ERROR > [utils.nio.NioClient] (main:null) SSL > > Handshake failed while connecting to > host: 192.168.12.1 port: 8250 > > 2017-05-23 11:58:22,466 ERROR > [utils.nio.NioConnection] (main:null) > > Unable to initialize the threads. > > java.io.IOException: SSL Handshake > failed while connecting to host: > > 192.168.12.1 port: 8250 > > at > com.cloud.utils.nio.NioClient. > > init(NioClient.java:67) > > at > com.cloud.utils.nio.NioConnection.start( > > NioConnection.java:88) > > at > com.cloud.agent.Agent.start(Agent.java:237) > > at > com.cloud.agent.AgentShell. > > launchAgent(AgentShell.java:399) > > at > com.cloud.agent.AgentShell. > > > launchAgentFromClassInfo(AgentShell.java:367) > > at > com.cloud.agent.AgentShell. > > launchAgent(AgentShell.java:351) > > at > com.cloud.agent.AgentShell. > > start(AgentShell.java:456) > > at > com.cloud.agent.AgentShell. > > main(AgentShell.java:491) > > 2017-05-23 11:58:22,468 INFO > [utils.exception.CSExceptionErrorCode] > > (main:null) Could not find exception: > com.cloud.utils.exception.NioConnectionException > > in error code list for exceptions > > 2017-05-23 11:58:22,468 WARN > [cloud.agent.Agent] (main:null) NIO > > Connection Exception > com.cloud.utils.exception.NioConnectionException: > > SSL Handshake failed while connecting > to host: 192.168.12.1 port: 8250 > > > > The setup is very simple. Single > management server and ports are open. > > > > Things checked / tried: > > > > · Destroyed SSVM multiple > times – still same problem. > > > > · SSH to SSVM from MS using > ssh -i /var/cloudstack/management/.ssh/id_rsa > > -p 3922 root@IPADDRESS – PASS > > > > · SSVM telnet on 8250 to MS > – PASS > > > > I’ve also tested a restore of the > DB into our working development > > 4.9.2.0 server. It also exhibits the > handshake errors, so most likely DB > > related. > > > > I’ve used up all my skills. Please > help > > > > Regards, > > Jason > > > > > > > > > > > > > > rohit.ya...@shapeblue.com > www.shapeblue.com<http://www.shapeblue.com> > 53 Chandos Place, Covent Garden, London WC2N > 4HSUK > @shapeblue > > > > > > > rohit.ya...@shapeblue.com > www.shapeblue.com<http://www.shapeblue.com> > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > @shapeblue > > > > > > > > > rohit.ya...@shapeblue.com > www.shapeblue.com<http://www.shapeblue.com> > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > @shapeblue > > > > > > > > > rohit.ya...@shapeblue.com > www.shapeblue.com > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > @shapeblue > > > > > > >
