Thanks Kristian,
Please confirm: - Did you see the new 4.11 systemvmtemplate and tried to destroy old systemvms and/or virtual routers? - Are all the xenserver hosts version 7.0 in all your clusters? - Was the upgrade in-place (on the same host), or on a different server? - Is dynamic roles enabled or you're still using the old/deprecated static roles i.e. commands.properties file? - Did you try accessing the UI in incognito mode or after clearing the browser cache? Looking at the logs, some errors: > 2018-01-01 00:01:22,481 ERROR [c.c.u.s.SshHelper] > (DirectAgent-29:ctx-62676ad7) (logid:48c55dee) SSH execution of command > /opt/cloud/bin/router_proxy.sh netusage.sh 169.254.3.20 -g has an error > status code in return. Result output: /root/func.sh: line 61: > /tmp/1514746614947437999-28844-biglock.lock: Read-only file system > 2018-01-01 00:01:22,482 WARN > [c.c.h.x.r.w.x.XenServer56NetworkUsageCommandWrapper] > (DirectAgent-29:ctx-62676ad7) (logid:48c55dee) Failed to get network usage This looks like a case where an old (pre 4.11) or new (4.11) based virtual router ended up having read-only file system. If there are old virtual routers they need to be removed, restart network with cleanup=true. > 2018-01-01 00:01:24,503 DEBUG [c.c.h.x.r.XenServerConnectionPool] > (ClusteredAgentManager Timer:ctx-198e0371) (logid:509434bb) Unable to create > master connection to host(172.16.10.13) , due to > org.apache.xmlrpc.XmlRpcException: Failed to read server's response: connect > timed out > 2018-01-01 00:01:24,503 DEBUG > [c.c.h.Status] (ClusteredAgentManager Timer:ctx-198e0371) (logid:509434bb) > Transition:[Resource state = Enabled, Agent event = AgentDisconnected, Host > id = 864, name = xen-13-10-b1-khi02] I'm not sure why your mgmt server was unable to make a connection with existing xenserver host. After a major version upgrade, on reconnection scripts will be copied to your xenserver host. In case that failed (because I don't have your upgrade logs), you'll need to reset host flags and reconnect on all such hosts. > 2018-01-01 00:01:26,367 ERROR [c.c.u.s.SshHelper] > (AgentTaskPool-12:ctx-cfea21e4) (logid:c2b516fe) SSH execution of command xe > sm-list | grep "resigning of duplicates" has an error status code in return. > Result output: Not sure about these ones. > 2018-01-01 00:40:09,411 WARN [c.c.h.x.r.CitrixResourceBase] > (DirectAgent-271:ctx-c9e202e1) (logid:1f6aafe2) Unable to start > i-1744-15758-VM due to- com.cloud.utils.exception.CloudRuntimeException: createIsoSRbyURI failed! mountpoint: 172.16.2.37/nfs/KHI03/template/tmpl/1326/465 due to SR_BACKEND_FAILURE_222Could not mount the directory specified in Device Configuration >---at >com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.createIsoSRbyURI(CitrixResourceBase.java:988) Can you check if your storage pools are accessible to your hosts? Lastly, if you attempt upgrade again (or perhaps on a test environment), do keep the logs for future reference for example sharing with us in case it fails for you again. - Rohit <https://cloudstack.apache.org> ________________________________ From: Kristian Liivak <k...@wavecom.ee> Sent: Saturday, February 17, 2018 8:49:26 PM To: users Subject: Re: System VM Issues 4.11? Hi Rohit Here it is https://wavecom.ee/files/management-server.rar Log beginning little bit earlier then upgrade. I did,nt fallow errors during gui access this time. If it helps i can allways revert management server where i was in that moment and grab more info. Lugupidamisega / Regards Kristian Liivak WaveCom As Endla 16, 10142 Tallinn Estonia Tel: +3726850001 Gsm: +37256850001 E-mail: k...@wavecom.ee Skype: kristian.liivak http://www.wavecom.ee http://www.facebook.com/wavecom.ee rohit.ya...@shapeblue.com www.shapeblue.com 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue ----- Original Message ----- From: "rohit yadav" <rohit.ya...@shapeblue.com> To: "users" <users@cloudstack.apache.org> Sent: Saturday, February 17, 2018 9:15:56 PM Subject: Re: System VM Issues 4.11? Hi Kristian, Yes, can you share the logs and if you saw any sql/db or other errors when you tried to access the UI? - Rohit <https://cloudstack.apache.org> ________________________________ From: Kristian Liivak <k...@wavecom.ee> Sent: Saturday, February 17, 2018 3:07:04 PM To: users Subject: Re: System VM Issues 4.11? Hi All I also upgraded from 4.10->4.11 and had similar issues ,even more. After upgrade, Gui under infrastructure i didnt show any host nor systemvm anymore , except VR Destroyd consoleproxy, new was created but no network running. Enviroment is Xenserver7 and i also runned before upgrade sql update mentioned below . Before reverting back i grabbed all log files. If anyone is intrested i can make them downloadable. Lugupidamisega / Regards Kristian Liivak WaveCom As Endla 16, 10142 Tallinn Estonia Tel: +3726850001 Gsm: +37256850001 E-mail: k...@wavecom.ee Skype: kristian.liivak http://www.wavecom.ee http://www.facebook.com/wavecom.ee rohit.ya...@shapeblue.com www.shapeblue.com<http://www.shapeblue.com> 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue ----- Original Message ----- From: "Paul Angus" <paul.an...@shapeblue.com> To: "users" <users@cloudstack.apache.org> Sent: Friday, February 16, 2018 10:45:26 AM Subject: RE: System VM Issues 4.11? Sami, Have you run the update step which is mentioned here? http://markmail.org/message/f42kqr3mx4r4hgih I don't KNOW if it is causing your problem, but if not, it won't help if it is missing. Kind regards, Paul Angus paul.an...@shapeblue.com www.shapeblue.com<http://www.shapeblue.com> 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue -----Original Message----- From: sami [mailto:sami.raj...@fapps.com] Sent: 16 February 2018 07:44 To: users@cloudstack.apache.org Subject: Fwd: System VM Issues 4.11? Hi, Actually I have a related issue, The System VMs stay in starting state. when I look situation on XenCenter, I see CPVM /SSVM running, but does not deploy configuration (no linklocal/private/publi IP on those VM¹s) I did not try VPC router yet, The Network routers starting ok. ACS upgraded 4.10 to 4.11 on Centos 6 xenserver 7.0 Advance network Regards -Sami > 15.2.2018 20.14, "Rohit Yadav" <rohit.ya...@shapeblue.com>: > >> Hi Benjamin, >> >> Starting 4.11, systemvms don't reboot after the patching operation so >> the shared observation matches expectations. Do you see vpc router >> vms stuck in starting state, can you share the logs and details of >> your hypervisor and other environment details? >> >> >> - Rohit >> >> <https://cloudstack.apache.org> >> >> >> >> ________________________________ >> From: Benjamin Naber <benjamin.na...@coders-area.de> >> Sent: Thursday, February 15, 2018 6:01:51 PM >> To: users >> Subject: System VM Issues 4.11? >> >> Hi @all, >> >> I've installed 4.11 from shape blue CentOS 7 Repository. >> System VM and Router VM Upgrades working well. But have some issues >> if I deploy a new Vpc. Router VMs stalled in state starting. >> Inspecting the hypervisor shows that the router VM doesn't reboot and >> doesn't deploy the configuration. Anyone other have the same issues? >> >> Kind Regards >> >> Benjamin >> >> Von meinem Huawei-Mobiltelefon gesendet >> >> rohit.ya...@shapeblue.com >> www.shapeblue.com<http://www.shapeblue.com> >> 53 Chandos Place, Covent Garden, London WC2N 4HSUK @shapeblue >> >>
