Rohit, Systemvms are 4.11, i can see that in console before login :) cat /proc/cmdline returnes some info. Please see attached pics
Management sevrer is working just fine. I dont have however autostart on in upgrade time. Im using old centos7 management server. Once allready made upgrade from ACS 4.9 to 4.10 Now from 4.10 to 4.11.. Is there maybe point to use clean 4.11 management server to resolve system vm issue ? And do i need to have HVM flag cheked with 4.11 system management template install ? There is nothing about it in instructions. However i have tried both ways without any success. Lugupidamisega / Regards Kristian Liivak Tegevjuht / Executive director WaveCom As Endla 16, 10142 Tallinn Estonia Tel: +3726850001 Gsm: +37256850001 E-mail: k...@wavecom.ee Skype: kristian.liivak http://www.wavecom.ee http://www.facebook.com/wavecom.ee ----- Original Message ----- From: "rohit yadav" <rohit.ya...@shapeblue.com> To: "users" <users@cloudstack.apache.org> Sent: Sunday, February 25, 2018 4:04:54 PM Subject: Re: System VM Issues 4.11? Hi Kristian, Thanks for sharing the logs. Looks like systemvm template failed to grab the cmdline options from XenServer to patch itself, reconfigure the nics. The ifconfig does not return at least the control nic. Can you check what `cat /etc/cloudstack-release` returns on the systemvm (this is to test that the systemvm template is indeed the 4.11 one). On the systemvm, see if cat /proc/cmdline returns anything? It also looks to me that the mgmt server is failing to run check for cloudstack-management start failures, see /etc/default/cloudstack-management and the cloudstack-management.service to find why the process fails, perhaps the classpath (or JAVA_HOME/java path) needs to be fixed? Lastly, can you also comment if you're using a new host/server to run mgmt server, or you did an in-place upgrade? - Rohit <https://cloudstack.apache.org> ________________________________ From: Kristian Liivak <k...@wavecom.ee> Sent: Saturday, February 24, 2018 8:31:35 AM To: users Subject: Re: System VM Issues 4.11? Hi Rohit, I managed fix absent of hosts. It was host tags related afterall. db host_tags contained tags before xenserver update in past. After dropping them update succeeded until new systemvm deployment. Like others said in thread, systemvm are successfully deployd, but not network will be started and they will not come up. I gathered logs , maybe you have time to check. https://wavecom.ee/files/cloudstack.rar Lugupidamisega / Regards Kristian Liivak Tegevjuht WaveCom As Endla 16, 10142 Tallinn Estonia Tel: +3726850001 Gsm: +37256850001 E-mail: k...@wavecom.ee Skype: kristian.liivak http://www.wavecom.ee http://www.facebook.com/wavecom.ee rohit.ya...@shapeblue.com www.shapeblue.com 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue ----- Original Message ----- From: "rohit yadav" <rohit.ya...@shapeblue.com> To: "users" <users@cloudstack.apache.org> Sent: Wednesday, February 21, 2018 6:25:11 PM Subject: Re: System VM Issues 4.11? Hi Kristian, Do you get any results if you execute the listRouters/listHosts apis using say a CLI such as cloudmonkey as the root admin? 🐵 > list routers listall=true 🐵 > list hosts Also, do you have any modifications to the UI, or webserver/middleware? I looked at the logs, and I could not find any exceptions or anything suspicious. About host tags, clearing the tags and doing reconnect causes cloudstack to scp/copy scripts and files to xenserver host which usually happens when you upgrade cloudstack. It's not an issue for you as in your logs I see such activities after cloudstack upgrades. (otherwise, it's done using, something like: # for host in $(xe host-list | grep ^uuid | awk '{print $NF}') ; do xe host-param-clear uuid=$host param-name=tags; done; ) - Rohit <https://cloudstack.apache.org> ________________________________ From: Kristian Liivak <k...@wavecom.ee> Sent: Tuesday, February 20, 2018 10:25:42 PM To: users Subject: Re: System VM Issues 4.11? Hi Rohit, Made another shot. Recreated all systemvm before update and get rid most of errors had before. Like before no host nor systemvm was present in gui However i saw correct numbers under infrastructure. I attached some pics form gui host and systemvm selection. I did destroyd old systemvms last time New ones started without network. New templates were present and new systemvm are created from 4.11 templates, checked from xen console. My management server resides in different vmware enviroment, however it have same management network like all my xenservers. I started with 4.9, where dynamic roles were allready introduced, but i doublechecked. My old enviroment was configured with different port using custom ssl, so therefore no cache. Enviroment is only xenserver7.0 hosts Uploaded new management logs and also /var/log/messages where update logs are present https://wavecom.ee/files/logs.rar Maybe you can find time to check.. And i can allways rollback management where i was to have additional info PS what you mean about : reset host flags and reconnect on all such hosts. i tried add hosts agan in gui bot that ended with failure. Lugupidamisega / Regards Kristian Liivak Tegevjuht / Executive director WaveCom As Endla 16, 10142 Tallinn Estonia Tel: +3726850001 Gsm: +37256850001 E-mail: k...@wavecom.ee Skype: kristian.liivak http://www.wavecom.ee http://www.facebook.com/wavecom.ee rohit.ya...@shapeblue.com www.shapeblue.com<http://www.shapeblue.com> 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue ----- Original Message ----- From: "rohit yadav" <rohit.ya...@shapeblue.com> To: "users" <users@cloudstack.apache.org> Sent: Sunday, February 18, 2018 3:52:00 PM Subject: Re: System VM Issues 4.11? Thanks Kristian, Please confirm: - Did you see the new 4.11 systemvmtemplate and tried to destroy old systemvms and/or virtual routers? - Are all the xenserver hosts version 7.0 in all your clusters? - Was the upgrade in-place (on the same host), or on a different server? - Is dynamic roles enabled or you're still using the old/deprecated static roles i.e. commands.properties file? - Did you try accessing the UI in incognito mode or after clearing the browser cache? Looking at the logs, some errors: > 2018-01-01 00:01:22,481 ERROR [c.c.u.s.SshHelper] > (DirectAgent-29:ctx-62676ad7) (logid:48c55dee) SSH execution of command > /opt/cloud/bin/router_proxy.sh netusage.sh 169.254.3.20 -g has an error > status code in return. Result output: /root/func.sh: line 61: > /tmp/1514746614947437999-28844-biglock.lock: Read-only file system > 2018-01-01 00:01:22,482 WARN > [c.c.h.x.r.w.x.XenServer56NetworkUsageCommandWrapper] > (DirectAgent-29:ctx-62676ad7) (logid:48c55dee) Failed to get network usage This looks like a case where an old (pre 4.11) or new (4.11) based virtual router ended up having read-only file system. If there are old virtual routers they need to be removed, restart network with cleanup=true. > 2018-01-01 00:01:24,503 DEBUG [c.c.h.x.r.XenServerConnectionPool] > (ClusteredAgentManager Timer:ctx-198e0371) (logid:509434bb) Unable to create > master connection to host(172.16.10.13) , due to > org.apache.xmlrpc.XmlRpcException: Failed to read server's response: connect > timed out > 2018-01-01 00:01:24,503 DEBUG > [c.c.h.Status] (ClusteredAgentManager Timer:ctx-198e0371) (logid:509434bb) > Transition:[Resource state = Enabled, Agent event = AgentDisconnected, Host > id = 864, name = xen-13-10-b1-khi02] I'm not sure why your mgmt server was unable to make a connection with existing xenserver host. After a major version upgrade, on reconnection scripts will be copied to your xenserver host. In case that failed (because I don't have your upgrade logs), you'll need to reset host flags and reconnect on all such hosts. > 2018-01-01 00:01:26,367 ERROR [c.c.u.s.SshHelper] > (AgentTaskPool-12:ctx-cfea21e4) (logid:c2b516fe) SSH execution of command xe > sm-list | grep "resigning of duplicates" has an error status code in return. > Result output: Not sure about these ones. > 2018-01-01 00:40:09,411 WARN [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-271:ctx-c9e202e1) (logid:1f6aafe2) Unable to start > i-1744-15758-VM due to- com.cloud.utils.exception.CloudRuntimeException: createIsoSRbyURI failed! mountpoint: 172.16.2.37/nfs/KHI03/template/tmpl/1326/465 due to SR_BACKEND_FAILURE_222Could not mount the directory specified in Device Configuration >---at >com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.createIsoSRbyURI(CitrixResourceBase.java:988) Can you check if your storage pools are accessible to your hosts? Lastly, if you attempt upgrade again (or perhaps on a test environment), do keep the logs for future reference for example sharing with us in case it fails for you again. - Rohit <https://cloudstack.apache.org> ________________________________ From: Kristian Liivak <k...@wavecom.ee> Sent: Saturday, February 17, 2018 8:49:26 PM To: users Subject: Re: System VM Issues 4.11? Hi Rohit Here it is https://wavecom.ee/files/management-server.rar Log beginning little bit earlier then upgrade. I did,nt fallow errors during gui access this time. If it helps i can allways revert management server where i was in that moment and grab more info. Lugupidamisega / Regards Kristian Liivak WaveCom As Endla 16, 10142 Tallinn Estonia Tel: +3726850001 Gsm: +37256850001 E-mail: k...@wavecom.ee Skype: kristian.liivak http://www.wavecom.ee http://www.facebook.com/wavecom.ee rohit.ya...@shapeblue.com www.shapeblue.com<http://www.shapeblue.com> 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue ----- Original Message ----- From: "rohit yadav" <rohit.ya...@shapeblue.com> To: "users" <users@cloudstack.apache.org> Sent: Saturday, February 17, 2018 9:15:56 PM Subject: Re: System VM Issues 4.11? Hi Kristian, Yes, can you share the logs and if you saw any sql/db or other errors when you tried to access the UI? - Rohit <https://cloudstack.apache.org> ________________________________ From: Kristian Liivak <k...@wavecom.ee> Sent: Saturday, February 17, 2018 3:07:04 PM To: users Subject: Re: System VM Issues 4.11? Hi All I also upgraded from 4.10->4.11 and had similar issues ,even more. After upgrade, Gui under infrastructure i didnt show any host nor systemvm anymore , except VR Destroyd consoleproxy, new was created but no network running. Enviroment is Xenserver7 and i also runned before upgrade sql update mentioned below . Before reverting back i grabbed all log files. If anyone is intrested i can make them downloadable. Lugupidamisega / Regards Kristian Liivak WaveCom As Endla 16, 10142 Tallinn Estonia Tel: +3726850001 Gsm: +37256850001 E-mail: k...@wavecom.ee Skype: kristian.liivak http://www.wavecom.ee http://www.facebook.com/wavecom.ee rohit.ya...@shapeblue.com www.shapeblue.com<http://www.shapeblue.com> 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue ----- Original Message ----- From: "Paul Angus" <paul.an...@shapeblue.com> To: "users" <users@cloudstack.apache.org> Sent: Friday, February 16, 2018 10:45:26 AM Subject: RE: System VM Issues 4.11? Sami, Have you run the update step which is mentioned here? http://markmail.org/message/f42kqr3mx4r4hgih I don't KNOW if it is causing your problem, but if not, it won't help if it is missing. Kind regards, Paul Angus paul.an...@shapeblue.com www.shapeblue.com<http://www.shapeblue.com> 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue -----Original Message----- From: sami [mailto:sami.raj...@fapps.com] Sent: 16 February 2018 07:44 To: users@cloudstack.apache.org Subject: Fwd: System VM Issues 4.11? Hi, Actually I have a related issue, The System VMs stay in starting state. when I look situation on XenCenter, I see CPVM /SSVM running, but does not deploy configuration (no linklocal/private/publi IP on those VM¹s) I did not try VPC router yet, The Network routers starting ok. ACS upgraded 4.10 to 4.11 on Centos 6 xenserver 7.0 Advance network Regards -Sami > 15.2.2018 20.14, "Rohit Yadav" <rohit.ya...@shapeblue.com>: > >> Hi Benjamin, >> >> Starting 4.11, systemvms don't reboot after the patching operation so >> the shared observation matches expectations. Do you see vpc router >> vms stuck in starting state, can you share the logs and details of >> your hypervisor and other environment details? >> >> >> - Rohit >> >> <https://cloudstack.apache.org> >> >> >> >> ________________________________ >> From: Benjamin Naber <benjamin.na...@coders-area.de> >> Sent: Thursday, February 15, 2018 6:01:51 PM >> To: users >> Subject: System VM Issues 4.11? >> >> Hi @all, >> >> I've installed 4.11 from shape blue CentOS 7 Repository. >> System VM and Router VM Upgrades working well. But have some issues >> if I deploy a new Vpc. Router VMs stalled in state starting. >> Inspecting the hypervisor shows that the router VM doesn't reboot and >> doesn't deploy the configuration. Anyone other have the same issues? >> >> Kind Regards >> >> Benjamin >> >> Von meinem Huawei-Mobiltelefon gesendet >> >> rohit.ya...@shapeblue.com >> www.shapeblue.com<http://www.shapeblue.com> >> 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue >> >>
