Hi Rohit,

I managed fix absent of hosts. It was host tags related afterall. db host_tags 
contained tags before xenserver update in past. After dropping them update 
succeeded until new systemvm deployment.
Like  others said in thread, systemvm are successfully deployd, but not network 
will be started and they will not come up.
I gathered logs , maybe you have time to check.
https://wavecom.ee/files/cloudstack.rar


Lugupidamisega / Regards
 
Kristian Liivak

Tegevjuht

WaveCom As
Endla 16, 10142 Tallinn
Estonia
Tel: +3726850001
Gsm: +37256850001
E-mail: k...@wavecom.ee
Skype: kristian.liivak
http://www.wavecom.ee
http://www.facebook.com/wavecom.ee

----- Original Message -----
From: "rohit yadav" <rohit.ya...@shapeblue.com>
To: "users" <users@cloudstack.apache.org>
Sent: Wednesday, February 21, 2018 6:25:11 PM
Subject: Re: System VM Issues 4.11?

Hi Kristian,


Do you get any results if you execute the listRouters/listHosts apis using say 
a CLI such as cloudmonkey as the root admin?


🐵 > list routers listall=true


🐵 > list hosts

Also, do you have any modifications to the UI, or webserver/middleware? I 
looked at the logs, and I could not find any exceptions or anything suspicious.

About host tags, clearing the tags and doing reconnect causes cloudstack to 
scp/copy scripts and files to xenserver host which usually happens when you 
upgrade cloudstack. It's not an issue for you as in your logs I see such 
activities after cloudstack upgrades. (otherwise, it's done using, something 
like:

# for host in $(xe host-list | grep ^uuid | awk '{print $NF}') ; do xe 
host-param-clear uuid=$host param-name=tags; done; )

- Rohit

<https://cloudstack.apache.org>



________________________________
From: Kristian Liivak <k...@wavecom.ee>
Sent: Tuesday, February 20, 2018 10:25:42 PM
To: users
Subject: Re: System VM Issues 4.11?

Hi Rohit,

Made another shot.

Recreated all systemvm before update and get rid most of errors had before.

Like before no host nor systemvm was present in gui
However i saw correct numbers under infrastructure.
I attached some pics form gui  host and systemvm  selection.

I did destroyd old systemvms last time
New ones started without network.
New templates were present and new systemvm are created from 4.11 templates, 
checked from xen console.

My management server resides in different vmware enviroment, however it have 
same management network like all my xenservers.

I started with 4.9, where dynamic roles  were allready introduced, but i 
doublechecked.

My old enviroment was configured with different port using custom ssl, so 
therefore no cache.

Enviroment is only xenserver7.0 hosts

Uploaded new management logs and also /var/log/messages where update logs are 
present https://wavecom.ee/files/logs.rar

Maybe you can find time to check..

And i can allways rollback management where i was to have additional info

PS what you mean about : reset host flags and reconnect on all such hosts. i 
tried add hosts agan in gui bot that ended with failure.

Lugupidamisega / Regards

Kristian Liivak

Tegevjuht / Executive director

WaveCom As
Endla 16, 10142 Tallinn
Estonia
Tel: +3726850001
Gsm: +37256850001
E-mail: k...@wavecom.ee
Skype: kristian.liivak
http://www.wavecom.ee
http://www.facebook.com/wavecom.ee


rohit.ya...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

----- Original Message -----
From: "rohit yadav" <rohit.ya...@shapeblue.com>
To: "users" <users@cloudstack.apache.org>
Sent: Sunday, February 18, 2018 3:52:00 PM
Subject: Re: System VM Issues 4.11?

Thanks Kristian,


Please confirm:

- Did you see the new 4.11 systemvmtemplate and tried to destroy old systemvms 
and/or virtual routers?

- Are all the xenserver hosts version 7.0 in all your clusters?

- Was the upgrade in-place (on the same host), or on a different server?

- Is dynamic roles enabled or you're still using the old/deprecated static 
roles i.e. commands.properties file?

- Did you try accessing the UI in incognito mode or after clearing the browser 
cache?


Looking at the logs, some errors:


>  2018-01-01 00:01:22,481 ERROR [c.c.u.s.SshHelper] 
> (DirectAgent-29:ctx-62676ad7) (logid:48c55dee) SSH execution of command 
> /opt/cloud/bin/router_proxy.sh netusage.sh 169.254.3.20 -g has an error 
> status code in return. Result output: /root/func.sh: line 61: 
> /tmp/1514746614947437999-28844-biglock.lock: Read-only file system

> 2018-01-01 00:01:22,482 WARN  
> [c.c.h.x.r.w.x.XenServer56NetworkUsageCommandWrapper] 
> (DirectAgent-29:ctx-62676ad7) (logid:48c55dee) Failed to get network usage


This looks like a case where an old (pre 4.11) or new (4.11) based virtual 
router ended up having read-only file system. If there are old virtual routers 
they need to be removed, restart network with cleanup=true.


> 2018-01-01 00:01:24,503 DEBUG [c.c.h.x.r.XenServerConnectionPool] 
> (ClusteredAgentManager Timer:ctx-198e0371) (logid:509434bb) Unable to create 
> master connection to host(172.16.10.13) , due to 
> org.apache.xmlrpc.XmlRpcException: Failed to read server's response: connect 
> timed out                          > 2018-01-01 00:01:24,503 DEBUG 
> [c.c.h.Status] (ClusteredAgentManager Timer:ctx-198e0371) (logid:509434bb) 
> Transition:[Resource state = Enabled, Agent event = AgentDisconnected, Host 
> id = 864, name = xen-13-10-b1-khi02]

I'm not sure why your mgmt server was unable to make a connection with existing 
xenserver host. After a major version upgrade, on reconnection scripts will be 
copied to your xenserver host. In case that failed (because I don't have your 
upgrade logs), you'll need to reset host flags and reconnect on all such hosts.

> 2018-01-01 00:01:26,367 ERROR [c.c.u.s.SshHelper] 
> (AgentTaskPool-12:ctx-cfea21e4) (logid:c2b516fe) SSH execution of command xe 
> sm-list | grep "resigning of duplicates" has an error status code in return. 
> Result output:


Not sure about these ones.


> 2018-01-01 00:40:09,411 WARN  [c.c.h.x.r.CitrixResourceBase] 
> (DirectAgent-271:ctx-c9e202e1) (logid:1f6aafe2) Unable to start 
> i-1744-15758-VM due to-

com.cloud.utils.exception.CloudRuntimeException: createIsoSRbyURI failed! 
mountpoint: 172.16.2.37/nfs/KHI03/template/tmpl/1326/465 due to 
SR_BACKEND_FAILURE_222Could not mount the directory specified in Device 
Configuration
>---at 
>com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.createIsoSRbyURI(CitrixResourceBase.java:988)

Can you check if your storage pools are accessible to your hosts?


Lastly, if you attempt upgrade again (or perhaps on a test environment), do 
keep the logs for future reference for example sharing with us in case it fails 
for you again.



- Rohit

<https://cloudstack.apache.org>



________________________________
From: Kristian Liivak <k...@wavecom.ee>
Sent: Saturday, February 17, 2018 8:49:26 PM
To: users
Subject: Re: System VM Issues 4.11?

Hi Rohit

Here it is https://wavecom.ee/files/management-server.rar
Log beginning little bit earlier then upgrade.
I did,nt fallow errors during gui access this time.
If it helps i can allways revert management server where i was in that moment 
and grab more info.

Lugupidamisega / Regards

Kristian Liivak

WaveCom As
Endla 16, 10142 Tallinn
Estonia
Tel: +3726850001
Gsm: +37256850001
E-mail: k...@wavecom.ee
Skype: kristian.liivak
http://www.wavecom.ee
http://www.facebook.com/wavecom.ee


rohit.ya...@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue



----- Original Message -----
From: "rohit yadav" <rohit.ya...@shapeblue.com>
To: "users" <users@cloudstack.apache.org>
Sent: Saturday, February 17, 2018 9:15:56 PM
Subject: Re: System VM Issues 4.11?

Hi Kristian,


Yes, can you share the logs and if you saw any sql/db or other errors when you 
tried to access the UI?


- Rohit

<https://cloudstack.apache.org>



________________________________
From: Kristian Liivak <k...@wavecom.ee>
Sent: Saturday, February 17, 2018 3:07:04 PM
To: users
Subject: Re: System VM Issues 4.11?

Hi All

I also upgraded from 4.10->4.11 and had similar issues ,even more.
After upgrade, Gui under infrastructure i didnt show any host nor systemvm 
anymore , except VR
Destroyd consoleproxy, new was created but no network running.

Enviroment is Xenserver7 and i also runned before upgrade sql update mentioned 
below .
Before reverting back i grabbed all log files. If anyone is intrested i can 
make them downloadable.


Lugupidamisega / Regards

Kristian Liivak

WaveCom As
Endla 16, 10142 Tallinn
Estonia
Tel: +3726850001
Gsm: +37256850001
E-mail: k...@wavecom.ee
Skype: kristian.liivak
http://www.wavecom.ee
http://www.facebook.com/wavecom.ee


rohit.ya...@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue



----- Original Message -----
From: "Paul Angus" <paul.an...@shapeblue.com>
To: "users" <users@cloudstack.apache.org>
Sent: Friday, February 16, 2018 10:45:26 AM
Subject: RE: System VM Issues 4.11?

Sami,

Have you run the update step which is mentioned here?
http://markmail.org/message/f42kqr3mx4r4hgih

I don't KNOW if it is causing your problem, but if not, it won't help if it is 
missing.


Kind regards,

Paul Angus

paul.an...@shapeblue.com
www.shapeblue.com<http://www.shapeblue.com>
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue




-----Original Message-----
From: sami [mailto:sami.raj...@fapps.com]
Sent: 16 February 2018 07:44
To: users@cloudstack.apache.org
Subject: Fwd: System VM Issues 4.11?


Hi,

Actually I have a related issue, The System VMs stay in starting state.
when I look situation on XenCenter, I see CPVM /SSVM running, but does not 
deploy configuration (no linklocal/private/publi IP on those VM¹s)

I did not try VPC router yet, The Network routers starting ok.


ACS upgraded 4.10 to 4.11 on Centos 6
xenserver 7.0
Advance network

Regards
-Sami


> 15.2.2018 20.14, "Rohit Yadav" <rohit.ya...@shapeblue.com>:
>
>> Hi Benjamin,
>>
>> Starting 4.11, systemvms don't reboot after the patching operation so
>> the shared observation matches expectations. Do you see vpc router
>> vms stuck in starting state, can you share the logs and details of
>> your hypervisor and other environment details?
>>
>>
>> - Rohit
>>
>> <https://cloudstack.apache.org>
>>
>>
>>
>> ________________________________
>> From: Benjamin Naber <benjamin.na...@coders-area.de>
>> Sent: Thursday, February 15, 2018 6:01:51 PM
>> To: users
>> Subject: System VM Issues 4.11?
>>
>> Hi @all,
>>
>> I've installed 4.11 from shape blue CentOS 7 Repository.
>> System VM and Router VM Upgrades working well. But have some issues
>> if I deploy a new Vpc. Router VMs stalled in state starting.
>> Inspecting the hypervisor shows that the router VM doesn't reboot and
>> doesn't deploy the configuration. Anyone other have the same issues?
>>
>> Kind Regards
>>
>> Benjamin
>>
>> Von meinem Huawei-Mobiltelefon gesendet
>>
>> rohit.ya...@shapeblue.com
>> www.shapeblue.com<http://www.shapeblue.com>
>> 53 Chandos Place, Covent Garden, London  WC2N 4HSUK @shapeblue
>>
>>

Reply via email to