Issue with storage while deploying VM

[victor]

Hello Guys,


I was using cloudstack for some tome. But today while creating VM', I 
got the following error from backend.


===

2018-02-25 19:56:50,105 DEBUG [c.c.s.StorageManagerImpl] 
(API-Job-Executor-8:ctx-6b5aebd2 job-403 ctx-0358836c) (logid:0ec966c2) 
Checking pool 11 for storage, totalSize: 160982630400, usedBytes: 
53763637248, usedPct: 0.333971665852467, disable threshold: 0.85
2018-02-25 19:56:50,105 DEBUG [c.c.s.StorageManagerImpl] 
(API-Job-Executor-8:ctx-6b5aebd2 job-403 ctx-0358836c) (logid:0ec966c2) 
Destination pool id: 11
2018-02-25 19:56:50,115 DEBUG [c.c.s.StorageManagerImpl] 
(API-Job-Executor-8:ctx-6b5aebd2 job-403 ctx-0358836c) (logid:0ec966c2) 
pool id for the volume with id: 83 is null
2018-02-25 19:56:50,115 DEBUG [c.c.s.StorageManagerImpl] 
(API-Job-Executor-8:ctx-6b5aebd2 job-403 ctx-0358836c) (logid:0ec966c2) 
Found storage pool Glus2 of type Gluster
2018-02-25 19:56:50,115 DEBUG [c.c.s.StorageManagerImpl] 
(API-Job-Executor-8:ctx-6b5aebd2 job-403 ctx-0358836c) (logid:0ec966c2) 
Total capacity of the pool Glus2 id: 11 is 160982630400
2018-02-25 19:56:50,116 DEBUG [c.c.s.StorageManagerImpl] 
(API-Job-Executor-8:ctx-6b5aebd2 job-403 ctx-0358836c) (logid:0ec966c2) 
Checking pool: 11 for volume allocation [Vol[83|vm=57|ROOT]], maxSize : 
160982630400, totalAllocatedSize : 549755813888, askingSize : 0, 
allocated disable threshold: 10.0
2018-02-25 19:56:50,116 DEBUG [c.c.s.StorageManagerImpl] 
(API-Job-Executor-8:ctx-6b5aebd2 job-403 ctx-0358836c) (logid:0ec966c2) 
Insufficient un-allocated capacity on: 11 for volume allocation: 
[Vol[83|vm=57|ROOT]], not enough storage, maxSize : 160982630400, 
totalAllocatedSize : 549755813888, askingSize : 0

==


I couldn't understand the meaning of this. I am glusterfs for storage 
and have enough space available, enough ram and ip. Can anybody shed 
some light regarding the above error. Especially with the part "maxSize 
: 160982630400, totalAllocatedSize : 549755813888, askingSize : 0"



Regards

Victor

Re: System VM Issues 4.11?

[Rohit Yadav]

Hi Kristian,


Thanks for sharing the logs. Looks like systemvm template failed to grab the 
cmdline options from XenServer to patch itself, reconfigure the nics. The 
ifconfig does not return at least the control nic. Can you check what `cat 
/etc/cloudstack-release` returns on the systemvm (this is to test that the 
systemvm template is indeed the 4.11 one).


On the systemvm, see if cat /proc/cmdline returns anything? It also looks to me 
that the mgmt server is failing to run check for cloudstack-management start 
failures, see /etc/default/cloudstack-management and the 
cloudstack-management.service to find why the process fails, perhaps the 
classpath (or JAVA_HOME/java path) needs to be fixed?


Lastly, can you also comment if you're using a new host/server to run mgmt 
server, or you did an in-place upgrade?


- Rohit






From: Kristian Liivak 
Sent: Saturday, February 24, 2018 8:31:35 AM
To: users
Subject: Re: System VM Issues 4.11?

Hi Rohit,

I managed fix absent of hosts. It was host tags related afterall. db host_tags 
contained tags before xenserver update in past. After dropping them update 
succeeded until new systemvm deployment.
Like  others said in thread, systemvm are successfully deployd, but not network 
will be started and they will not come up.
I gathered logs , maybe you have time to check.
https://wavecom.ee/files/cloudstack.rar


Lugupidamisega / Regards

Kristian Liivak

Tegevjuht

WaveCom As
Endla 16, 10142 Tallinn
Estonia
Tel: +3726850001
Gsm: +37256850001
E-mail: k...@wavecom.ee
Skype: kristian.liivak
http://www.wavecom.ee
http://www.facebook.com/wavecom.ee


rohit.ya...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

- Original Message -
From: "rohit yadav" 
To: "users" 
Sent: Wednesday, February 21, 2018 6:25:11 PM
Subject: Re: System VM Issues 4.11?

Hi Kristian,


Do you get any results if you execute the listRouters/listHosts apis using say 
a CLI such as cloudmonkey as the root admin?


 > list routers listall=true


 > list hosts

Also, do you have any modifications to the UI, or webserver/middleware? I 
looked at the logs, and I could not find any exceptions or anything suspicious.

About host tags, clearing the tags and doing reconnect causes cloudstack to 
scp/copy scripts and files to xenserver host which usually happens when you 
upgrade cloudstack. It's not an issue for you as in your logs I see such 
activities after cloudstack upgrades. (otherwise, it's done using, something 
like:

# for host in $(xe host-list | grep ^uuid | awk '{print $NF}') ; do xe 
host-param-clear uuid=$host param-name=tags; done; )

- Rohit






From: Kristian Liivak 
Sent: Tuesday, February 20, 2018 10:25:42 PM
To: users
Subject: Re: System VM Issues 4.11?

Hi Rohit,

Made another shot.

Recreated all systemvm before update and get rid most of errors had before.

Like before no host nor systemvm was present in gui
However i saw correct numbers under infrastructure.
I attached some pics form gui  host and systemvm  selection.

I did destroyd old systemvms last time
New ones started without network.
New templates were present and new systemvm are created from 4.11 templates, 
checked from xen console.

My management server resides in different vmware enviroment, however it have 
same management network like all my xenservers.

I started with 4.9, where dynamic roles  were allready introduced, but i 
doublechecked.

My old enviroment was configured with different port using custom ssl, so 
therefore no cache.

Enviroment is only xenserver7.0 hosts

Uploaded new management logs and also /var/log/messages where update logs are 
present https://wavecom.ee/files/logs.rar

Maybe you can find time to check..

And i can allways rollback management where i was to have additional info

PS what you mean about : reset host flags and reconnect on all such hosts. i 
tried add hosts agan in gui bot that ended with failure.

Lugupidamisega / Regards

Kristian Liivak

Tegevjuht / Executive director

WaveCom As
Endla 16, 10142 Tallinn
Estonia
Tel: +3726850001
Gsm: +37256850001
E-mail: k...@wavecom.ee
Skype: kristian.liivak
http://www.wavecom.ee
http://www.facebook.com/wavecom.ee


rohit.ya...@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue



- Original Message -
From: "rohit yadav" 
To: "users" 
Sent: Sunday, February 18, 2018 3:52:00 PM
Subject: Re: System VM Issues 4.11?

Thanks Kristian,


Please confirm:

- Did you see the new 4.11 systemvmtemplate and tried to destroy old systemvms 
and/or virtual routers?

- Are all the xenserver hosts version 7.0 in all your clusters?

- Was the 

Re: [4.11] Management to VR connection issues

[Rohit Yadav]

Hi Rene,


Paul is correct, for default VMware systemvm I had fixed it here:



https://github.com/apache/cloudstack/blob/4.11/engine/schema/resources/META-INF/db/schema-41000to41100.sql#L403


But the above would have worked only for new installations, for upgraded ones 
we'll need to fix the release notes to ask users/admins to select 'Other Linux 
64-bit'. Can you try that and share if that works for you?


I also checked, we're still using the 6.0 sdk jars. That needs to be fixed as 
well.


- Rohit







rohit.ya...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

From: Paul Angus
Sent: Sunday, February 25, 2018 8:57:55 AM
To: d...@cloudstack.apache.org; users@cloudstack.apache.org
Cc: Rohit Yadav
Subject: RE: [4.11] Management to VR connection issues

Hey Rene.

Can you check that OS type that has been applied to your system VM template.
I found that mine were coming up as 32bit Debian 5, making them go REALLY slow 
and if there are rules applied to the firewall it takes forever to provision.  
Switching the guest OS fixed it.

If you use Linux (other 64)  - which is guestos_id 99 they run properly

I suspect that the VMware 6.5 mappings are failing when they aren’t supported 
by the 6.0 SDK which we use, but I'll need to get that verified



I think that we should have a 'ACS SystemVM' guest OS, which we can map to the 
best performing guest OS for each hypervisor version.




VP Technology
paul.an...@shapeblue.com
www.shapeblue.com




-Original Message-
From: Rene Moser [mailto:m...@renemoser.net]
Sent: 22 February 2018 16:27
To: users@cloudstack.apache.org; d...@cloudstack.apache.org
Subject: Re: [4.11] Management to VR connection issues


On 02/20/2018 08:04 PM, Rohit Yadav wrote:
> Hi Rene,
>
>
> Thanks for sharing - I've not seen this in test/production environment yet. 
> Does it help to destroy the VR and check if the issue persists? Also, is this 
> behaviour system-wide for every VR, or VRs of specific networks or topologies 
> such as VPCs? Are these VRs redundant in nature?

We have non-redundant VRs, and we haven't looked at VPC routers yet.

The current analyses shows the following:

1. Started the process to upgrade an existing router.
2. Router gets destroyed and re-deployed with new template 4.11 as expected.
3. Router OS has started, ACS router state keeps "starting". When we login by 
console, we see some actions in the cloud.log. At this point, router will be 
left in this state and gets destroyed after job timeout.
4. We reboot manually on the OS level. VR gets rebooted.
5. After the OS has booted, ACS Router state switches to "Running"
6. We can login by ssh. however ACS router still shows "requires upgrade" (but 
the OS has already booted with template 4.11) 7. When we upgrade, the same 
process happens again points 1-3. Feels like a dead lock.


Logs:
https://transfer.sh/DdTtH/management-server.log.gz

We continue our investigations

Regards
René