Re: Can't cancel Maintenance Mode on host

[Makrand]

Hi Dag,

1) Thanks for the reply. I was talking about canceling MM from cloud stack.
No issues taking out it in and out of MM on xenserver/xencenter level. With
normaly scene, one first puts host in MM from Cstack>>Then from XEN
center>>DO your reboot>>Exit MM from Xencenter>>Exit from Cstack.

2) When you said rebuild, you mean eject the host out of the pool and
reinstall OS? Also, I am yet to try to delete the host from Cstack and add
it back. Should I try that? Do you think it will work?

3) I also found this:- https://issues.apache.org/jira/browse/CLOUDSTACK-8210.
I know this is for KVM, but we are using Cstack 4.4.

BTW, on a broader view, this zone has some funky stuff happening. Its
Cstack 4.4.x and XEN server 6.2
We have noticed that VRs go into reboot loops once we reboot the storage.
VMs are stuck on XenServer in start stages. Sometimes we can't shut down
VMs. Sometimes we can't migrate VMs between hosts. We have also found dead
beef on Xenservers (whatever that means..one of our engineers told
me).  Let me dig some logs for these things and I will try to share it.

I am seriously thinking of reinstalling everything here. But I just need to
justify this to senior management.









--
Makrand


On Thu, Feb 22, 2018 at 6:14 PM, Dag Sonstebo 
wrote:

> Hi Makrand,
>
> Yes this rings a bell – first of I would advise you to thread very
> carefully – this is most likely an issue with your underlying XAPI db on
> your poolmaster, so there is a risk of further problems.
>
> We have seen this in the past with a couple of clients – and I think we
> found XS servers still in MM in XenCentre (unbeknownst to CloudStack) – but
> we have then had some problems getting the hosts out of MM again from the
> Xen side. We have also seen situations where taking one host out of MM in
> XenCentre puts another host into MM, which is odd. I know in on one
> occasion we ended up removing / rebuilding / reading the stubborn MM host.
> Unfortunately we never found the actual root cause.
>
> Hopefully your issue is something simpler – have you checked that all SRs
> are plugged on all hosts?
>
> Regards,
> Dag Sonstebo
> Cloud Architect
> ShapeBlue
>
> On 22/02/2018, 10:32, "Makrand"  wrote:
>
> Hi All,
>
> Couple of days back we had some iSCSI issue and all the LUNs were
> disconnected from Xenserver hosts. After the issue was  fixed and when
> all
> LUNs were back online, for some BIOS checks, we put one of compute
> node in
> Maintenance Mode from cloudstack. It took more than usual time for it
> to go
> into MM (was stuck in PrepateforMaintenance), but it went anyhow. Now
> whenever we are trying to cancel its MM, it just fails:- Command
> failed due
> to Internal Server Error.
>
> The logs are indicating below
>
> 2018-02-16 09:44:24,291 INFO  [o.a.c.f.j.i.AsyncJobMonitor]
> (API-Job-Executor-27:ctx-1e865550 job-72477) Add job-72477 into job
> monitoring
> 2018-02-16 09:44:24,292 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-27:ctx-1e865550 job-72477) Executing AsyncJobVO
> {id:72477, userId: 2, accountId: 2,
>  instanceType: Host, instanceId: 26, cmd:
> org.apache.cloudstack.api.command.admin.host.CancelMaintenanceCmd,
> cmdInfo:
> {"id":"4bca233d-0e61-495c-a522-43800fe311fc","r
> esponse":"json","sessionkey":"ZxtGyco2RuYHil/VnglSOgguw5c\
> u003d","ctxDetails":"{\"com.cloud.host.Host\":\"4bca233d-
> 0e61-495c-a522-43800fe311fc\"}","cmdEventType":"MA
> INT.CANCEL","ctxUserId":"2","httpmethod":"GET","_":"
> 1518774059073","uuid":"4bca233d-0e61-495c-a522-
> 43800fe311fc","ctxAccountId":"2","ctxStartEventId":"51924"},
> cmdVe
> rsion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
> null, initMsid: 16143068278473, completeMsid: null, lastUpdated: null,
> lastPolled: null, crea
> ted: null}
> 2018-02-16 09:44:24,301 ERROR [c.c.a.ApiAsyncJobDispatcher]
> (API-Job-Executor-27:ctx-1e865550 job-72477) Unexpected exception
> while
> executing org.apache.cloudstack.a
> pi.command.admin.host.CancelMaintenanceCmd
> java.lang.NullPointerException
> at
> com.cloud.resource.ResourceManagerImpl.doCancelMaintenance(
> ResourceManagerImpl.java:2083)
> at
> com.cloud.resource.ResourceManagerImpl.cancelMaintenance(
> ResourceManagerImpl.java:2140)
> at
> com.cloud.resource.ResourceManagerImpl.cancelMaintenance(
> ResourceManagerImpl.java:1127)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(
> NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.springframework.aop.support.AopUtils.
> 

RE: Cloudstack 4.11 : NFS and system VMs NIC issue

[Khosrow Moossavi]

Hi Aican

I've seen similar behavior but in our case it was custom 4.10 (somehow
similar to 4.11 in this case) and XenServer 7.1 not 7.3.

Can you check in which mode VMs are started on XenServer? Since it's 7.3
I'm assuming they are brought up in HVM.

If this is the case there's an open PR (#2465) to fix this in 4.11.0.1.

Khosrow Moossavi
CloudOps

On Feb 22, 2018 09:08, "Paul Angus"  wrote:

Hi Aican,

I don’t think that XenServer 7.3 is supported.


Kind regards,

Paul Angus

paul.an...@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue




-Original Message-
From: Aican France [mailto:aican...@gmail.com]
Sent: 22 February 2018 08:31
To: users@cloudstack.apache.org
Subject: Cloudstack 4.11 : NFS and system VMs NIC issue

Hi All,



I’m trying to build a small cloudstack lab inside Vmware Workstation 14.0.



I’m using :

-Ubuntu 14.04.5 NFS Server for primary / secondary storage –
192.168.117.129

-Ubuntu 16.04.3 for Cloudstack management server – 192.168.117.135

-Xenserver 7.3 with latest package 2017.1206 – 192.168.117.136



All these servers hosted on the same LAN (192.168.117.x/24), provisioning
VMs IP range on the same LAN.



I’m facing some issues, but may be they are linked :

-Secondary storage on cloudstack dashboard shows 0.00KB/0.00KB

-Secondary Storage VM s-1-VM and console Proxy VM v-2-VM still
pending il Starting state.



VMs are started (Xencenter View) but when logging into VMs console, NIC are
not configured.



Even if I can see the secondary storage configured in my Infrastructure
dashboard, only one NFS mount point appears on xenserver side :

192.168.117.129:/nfs/primary/ on
/run/sr-mount/e666f671-a903-16b1-7676-ff0ad35f3db4 type nfs
(rw,relatime,vers=3,rsize=262144,wsize=262144,namlen=
255,acdirmin=0,acdirmax=0,soft,proto=tcp,timeo=100,
retrans=12,sec=sys,mountaddr=192.168.117.129,mountvers=3,
mountport=39208,mountproto=tcp,local_lock=none,addr=192.168.117.129)

/dev/mapper/XSLocalEXT--ccb0abd6--d830--bc2e--8372--
daa16a7f5d78-ccb0abd6--d830--bc2e--8372--daa16a7f5d78
on /run/sr-mount/ccb0abd6-d830-bc2e-8372-daa16a7f5d78 type ext3
(rw,relatime,data=ordered)



Management-serv.log shows :

2018-02-22 08:50:06,346 DEBUG [c.c.s.StatsCollector]
(StatsCollector-2:ctx-79cce329) (logid:3cf7734e) There is no secondary
storage VM for secondary storage host nfs://192.168.117.129/nfs/secondary



Some sample logs or the Vms :



2018-02-22 08:51:31,840 DEBUG [c.c.h.x.r.w.x.CitrixStartCommandWrapper]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) 1. The VM s-1-VM is in
Starting state.

2018-02-22 08:51:31,874 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) Created VM
8e0a421d-cc05-baa4-49f2-6e75fab04d5a for s-1-VM

2018-02-22 08:51:31,880 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) PV args are
%template=domP%type=secstorage%host=192.168.117.135%port=8250%name=s-1-VM%
zone=1%pod=1%guid=s-1-VM%workers=5%resource=org.apache.
cloudstack.storage.resource.NfsSecondaryStorageResource%
instance=SecStorage%sslcopy=false%role=templateProcessor%
mtu=1500%eth2ip=192.168.117.40%eth2mask=255.255.255.0%
gateway=192.168.117.2%eth0ip=169.254.0.32%eth0mask=255.255.
0.0%eth1ip=192.168.117.6%eth1mask=255.255.255.0%mgmtcidr=
192.168.117.0/24%localgw=192.168.117.2%private.network.
device=eth1%internaldns1=192.168.117.2%dns1=8.8.8.8%nfsVersion=null

2018-02-22 08:51:49,208 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) Creating VIF for s-1-VM on
nic [Nic:Guest-192.168.117.40-vlan://untagged]

2018-02-22 08:51:49,234 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) Creating VIF for s-1-VM on
nic [Nic:Control-169.254.0.32-null]

2018-02-22 08:51:49,664 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) Creating VIF for s-1-VM on
nic [Nic:Management-192.168.117.6-null]

2018-02-22 08:52:01,239 INFO  [c.c.h.x.r.w.x.CitrixStartCommandWrapper]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) Programmed default network
rules for s-1-VM

2018-02-22 08:52:01,239 DEBUG [c.c.h.x.r.w.x.CitrixStartCommandWrapper]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) 2. The VM s-1-VM is in
Running state.

2018-02-22 08:54:11,074 DEBUG [c.c.a.t.Request]
(DirectAgentCronJob-3:ctx-1258f3db) (logid:2948b9e7) Seq
1-4371306388316487683: Processing:  { Ans: , MgmtId: 52231347383, via:
1(xenserver_7.3-cs4.11), Ver: v1, Flags: 10, [{"com.cloud.agent.api.
ClusterVMMetaDataSyncAnswer":{"_clusterId":1,"_vmMetaDatum":
{"s-1-VM":"device-model:qemu-trad;apic:true;viridian:true;
timeoffset:0;pae:true;acpi:1;hpet:true;nx:true","v-2-VM":"
device-model:qemu-trad;apic:true;viridian:true;timeoffset:
0;pae:true;acpi:1;hpet:true;nx:true"},"_isExecuted":false,
"result":true,"wait":0}}]
}

2018-02-22 08:57:11,054 DEBUG [c.c.a.t.Request]

Re: System VM Issues 4.11?

[Kristian Liivak]

Hi Rohit


After upgrade 
list routers listall=true
will give correct info
but list hosts will give none

I didnt had tags in hosts, anyhow runned clear and tried add hosts. No success 
at all.

i destroyd console proxy in one zone and vr in other zone
They both didnt had networks present
I have logs if needed for sysvm destroy and creation.
Anyhow i guess issue is more related with missing hosts..

I used in 4.10 tomcat ssl port 10250 and comodo ssl specified in 
server7-nonssl.xml
Outside access for others is usually via proxy but i accessed from same network
Nothing more modified except few cloudmonkey scripts for cleaning unneeded 
offerings created by api and update resources.

Lugupidamisega / Regards
 
Kristian Liivak

Tegevjuht / Executive director

WaveCom As
Endla 16, 10142 Tallinn
Estonia
Tel: +3726850001
Gsm: +37256850001
E-mail: k...@wavecom.ee
Skype: kristian.liivak
http://www.wavecom.ee
http://www.facebook.com/wavecom.ee

- Original Message -
From: "rohit yadav" 
To: "users" 
Sent: Wednesday, February 21, 2018 6:25:11 PM
Subject: Re: System VM Issues 4.11?

Hi Kristian,


Do you get any results if you execute the listRouters/listHosts apis using say 
a CLI such as cloudmonkey as the root admin?


 > list routers listall=true


 > list hosts

Also, do you have any modifications to the UI, or webserver/middleware? I 
looked at the logs, and I could not find any exceptions or anything suspicious.

About host tags, clearing the tags and doing reconnect causes cloudstack to 
scp/copy scripts and files to xenserver host which usually happens when you 
upgrade cloudstack. It's not an issue for you as in your logs I see such 
activities after cloudstack upgrades. (otherwise, it's done using, something 
like:

# for host in $(xe host-list | grep ^uuid | awk '{print $NF}') ; do xe 
host-param-clear uuid=$host param-name=tags; done; )

- Rohit






From: Kristian Liivak 
Sent: Tuesday, February 20, 2018 10:25:42 PM
To: users
Subject: Re: System VM Issues 4.11?

Hi Rohit,

Made another shot.

Recreated all systemvm before update and get rid most of errors had before.

Like before no host nor systemvm was present in gui
However i saw correct numbers under infrastructure.
I attached some pics form gui  host and systemvm  selection.

I did destroyd old systemvms last time
New ones started without network.
New templates were present and new systemvm are created from 4.11 templates, 
checked from xen console.

My management server resides in different vmware enviroment, however it have 
same management network like all my xenservers.

I started with 4.9, where dynamic roles  were allready introduced, but i 
doublechecked.

My old enviroment was configured with different port using custom ssl, so 
therefore no cache.

Enviroment is only xenserver7.0 hosts

Uploaded new management logs and also /var/log/messages where update logs are 
present https://wavecom.ee/files/logs.rar

Maybe you can find time to check..

And i can allways rollback management where i was to have additional info

PS what you mean about : reset host flags and reconnect on all such hosts. i 
tried add hosts agan in gui bot that ended with failure.

Lugupidamisega / Regards

Kristian Liivak

Tegevjuht / Executive director

WaveCom As
Endla 16, 10142 Tallinn
Estonia
Tel: +3726850001
Gsm: +37256850001
E-mail: k...@wavecom.ee
Skype: kristian.liivak
http://www.wavecom.ee
http://www.facebook.com/wavecom.ee


rohit.ya...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

- Original Message -
From: "rohit yadav" 
To: "users" 
Sent: Sunday, February 18, 2018 3:52:00 PM
Subject: Re: System VM Issues 4.11?

Thanks Kristian,


Please confirm:

- Did you see the new 4.11 systemvmtemplate and tried to destroy old systemvms 
and/or virtual routers?

- Are all the xenserver hosts version 7.0 in all your clusters?

- Was the upgrade in-place (on the same host), or on a different server?

- Is dynamic roles enabled or you're still using the old/deprecated static 
roles i.e. commands.properties file?

- Did you try accessing the UI in incognito mode or after clearing the browser 
cache?


Looking at the logs, some errors:


>  2018-01-01 00:01:22,481 ERROR [c.c.u.s.SshHelper] 
> (DirectAgent-29:ctx-62676ad7) (logid:48c55dee) SSH execution of command 
> /opt/cloud/bin/router_proxy.sh netusage.sh 169.254.3.20 -g has an error 
> status code in return. Result output: /root/func.sh: line 61: 
> /tmp/1514746614947437999-28844-biglock.lock: Read-only file system

> 2018-01-01 00:01:22,482 WARN  
> [c.c.h.x.r.w.x.XenServer56NetworkUsageCommandWrapper] 
> (DirectAgent-29:ctx-62676ad7) (logid:48c55dee) Failed to get network usage


This looks like a case where an old (pre 4.11) or new (4.11) based virtual 

Re: [4.11] Management to VR connection issues

[Rene Moser]

On 02/20/2018 08:04 PM, Rohit Yadav wrote:
> Hi Rene,
> 
> 
> Thanks for sharing - I've not seen this in test/production environment yet. 
> Does it help to destroy the VR and check if the issue persists? Also, is this 
> behaviour system-wide for every VR, or VRs of specific networks or topologies 
> such as VPCs? Are these VRs redundant in nature?

We have non-redundant VRs, and we haven't looked at VPC routers yet.

The current analyses shows the following:

1. Started the process to upgrade an existing router.
2. Router gets destroyed and re-deployed with new template 4.11 as expected.
3. Router OS has started, ACS router state keeps "starting". When we
login by console, we see some actions in the cloud.log. At this point,
router will be left in this state and gets destroyed after job timeout.
4. We reboot manually on the OS level. VR gets rebooted.
5. After the OS has booted, ACS Router state switches to "Running"
6. We can login by ssh. however ACS router still shows
"requires upgrade" (but the OS has already booted with template 4.11)
7. When we upgrade, the same process happens again points 1-3. Feels
like a dead lock.


Logs:
https://transfer.sh/DdTtH/management-server.log.gz

We continue our investigations

Regards
René

[DISCUSS] CloudStack packages upstream initiative

[Rohit Yadav]

All,


I've been trying to engage with people from upstream distributions (CentOS, 
Fedora, Debian etc) to get our free/oss packages published focusing on LTS 
releases. The noredist/non-oss packages most likely are not going to be 
supported due to licensing issues by these distros so they will continue to be 
supported by 3rd parties.


The larger goals are to make it easier for users to install CloudStack and for 
the project to be able to reach a wider people-base and grow our community.


This year's FOSDEM, I met a couple of people along with Daan to discuss this 
and we've made some progress in this regard. The purpose of this thread is to 
engage with others who may be interested and may help us do this.


Some of us have been participating and have been accepted into CentOS's 
CloudSIG group, our first goal is to understand their processes, tooling, and 
come up with first set of free/oss rpm packages for CentOS and then understand 
how we can help get them published.


I'm trying to aim the same for Debian (Ubuntu?), Fedora and if you've any 
pointers/advice for me that would be great!


All interested people are free to join me in the CloudSIG meetings that will 
happen weekly on Thursday on #centos-devel channel in Freenode at 1500UTC. The 
first meeting was held today, here are some useful links:


https://public.etherpad-mozilla.org/p/centos-cloud-sig

https://wiki.centos.org/SpecialInterestGroup/Cloud

https://wiki.centos.org/SIGGuide


Meeting log: 
https://www.centos.org/minutes/2018/February/centos-devel.2018-02-22-15.02.log.html



- Rohit





rohit.ya...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

RE: Cloudstack 4.11 : NFS and system VMs NIC issue

[Paul Angus]

Hi Aican,

I don’t think that XenServer 7.3 is supported.


Kind regards,

Paul Angus

paul.an...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 


-Original Message-
From: Aican France [mailto:aican...@gmail.com] 
Sent: 22 February 2018 08:31
To: users@cloudstack.apache.org
Subject: Cloudstack 4.11 : NFS and system VMs NIC issue

Hi All,



I’m trying to build a small cloudstack lab inside Vmware Workstation 14.0.



I’m using :

-Ubuntu 14.04.5 NFS Server for primary / secondary storage –
192.168.117.129

-Ubuntu 16.04.3 for Cloudstack management server – 192.168.117.135

-Xenserver 7.3 with latest package 2017.1206 – 192.168.117.136



All these servers hosted on the same LAN (192.168.117.x/24), provisioning VMs 
IP range on the same LAN.



I’m facing some issues, but may be they are linked :

-Secondary storage on cloudstack dashboard shows 0.00KB/0.00KB

-Secondary Storage VM s-1-VM and console Proxy VM v-2-VM still
pending il Starting state.



VMs are started (Xencenter View) but when logging into VMs console, NIC are not 
configured.



Even if I can see the secondary storage configured in my Infrastructure 
dashboard, only one NFS mount point appears on xenserver side :

192.168.117.129:/nfs/primary/ on
/run/sr-mount/e666f671-a903-16b1-7676-ff0ad35f3db4 type nfs
(rw,relatime,vers=3,rsize=262144,wsize=262144,namlen=255,acdirmin=0,acdirmax=0,soft,proto=tcp,timeo=100,retrans=12,sec=sys,mountaddr=192.168.117.129,mountvers=3,mountport=39208,mountproto=tcp,local_lock=none,addr=192.168.117.129)

/dev/mapper/XSLocalEXT--ccb0abd6--d830--bc2e--8372--daa16a7f5d78-ccb0abd6--d830--bc2e--8372--daa16a7f5d78
on /run/sr-mount/ccb0abd6-d830-bc2e-8372-daa16a7f5d78 type ext3
(rw,relatime,data=ordered)



Management-serv.log shows :

2018-02-22 08:50:06,346 DEBUG [c.c.s.StatsCollector]
(StatsCollector-2:ctx-79cce329) (logid:3cf7734e) There is no secondary storage 
VM for secondary storage host nfs://192.168.117.129/nfs/secondary



Some sample logs or the Vms :



2018-02-22 08:51:31,840 DEBUG [c.c.h.x.r.w.x.CitrixStartCommandWrapper]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) 1. The VM s-1-VM is in Starting 
state.

2018-02-22 08:51:31,874 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) Created VM 
8e0a421d-cc05-baa4-49f2-6e75fab04d5a for s-1-VM

2018-02-22 08:51:31,880 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) PV args are 
%template=domP%type=secstorage%host=192.168.117.135%port=8250%name=s-1-VM%zone=1%pod=1%guid=s-1-VM%workers=5%resource=org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource%instance=SecStorage%sslcopy=false%role=templateProcessor%mtu=1500%eth2ip=192.168.117.40%eth2mask=255.255.255.0%gateway=192.168.117.2%eth0ip=169.254.0.32%eth0mask=255.255.0.0%eth1ip=192.168.117.6%eth1mask=255.255.255.0%mgmtcidr=
192.168.117.0/24%localgw=192.168.117.2%private.network.device=eth1%internaldns1=192.168.117.2%dns1=8.8.8.8%nfsVersion=null

2018-02-22 08:51:49,208 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) Creating VIF for s-1-VM on nic 
[Nic:Guest-192.168.117.40-vlan://untagged]

2018-02-22 08:51:49,234 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) Creating VIF for s-1-VM on nic 
[Nic:Control-169.254.0.32-null]

2018-02-22 08:51:49,664 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) Creating VIF for s-1-VM on nic 
[Nic:Management-192.168.117.6-null]

2018-02-22 08:52:01,239 INFO  [c.c.h.x.r.w.x.CitrixStartCommandWrapper]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) Programmed default network rules 
for s-1-VM

2018-02-22 08:52:01,239 DEBUG [c.c.h.x.r.w.x.CitrixStartCommandWrapper]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) 2. The VM s-1-VM is in Running 
state.

2018-02-22 08:54:11,074 DEBUG [c.c.a.t.Request]
(DirectAgentCronJob-3:ctx-1258f3db) (logid:2948b9e7) Seq
1-4371306388316487683: Processing:  { Ans: , MgmtId: 52231347383, via:
1(xenserver_7.3-cs4.11), Ver: v1, Flags: 10, 
[{"com.cloud.agent.api.ClusterVMMetaDataSyncAnswer":{"_clusterId":1,"_vmMetaDatum":{"s-1-VM":"device-model:qemu-trad;apic:true;viridian:true;timeoffset:0;pae:true;acpi:1;hpet:true;nx:true","v-2-VM":"device-model:qemu-trad;apic:true;viridian:true;timeoffset:0;pae:true;acpi:1;hpet:true;nx:true"},"_isExecuted":false,"result":true,"wait":0}}]
}

2018-02-22 08:57:11,054 DEBUG [c.c.a.t.Request]
(DirectAgentCronJob-9:ctx-6ed1ca09) (logid:2948b9e7) Seq
1-4371306388316487683: Processing:  { Ans: , MgmtId: 52231347383, via:
1(xenserver_7.3-cs4.11), Ver: v1, Flags: 10, 

Re: Can't cancel Maintenance Mode on host

[Dag Sonstebo]

Hi Makrand,

Yes this rings a bell – first of I would advise you to thread very carefully – 
this is most likely an issue with your underlying XAPI db on your poolmaster, 
so there is a risk of further problems. 

We have seen this in the past with a couple of clients – and I think we found 
XS servers still in MM in XenCentre (unbeknownst to CloudStack) – but we have 
then had some problems getting the hosts out of MM again from the Xen side. We 
have also seen situations where taking one host out of MM in XenCentre puts 
another host into MM, which is odd. I know in on one occasion we ended up 
removing / rebuilding / reading the stubborn MM host. Unfortunately we never 
found the actual root cause.

Hopefully your issue is something simpler – have you checked that all SRs are 
plugged on all hosts?

Regards,
Dag Sonstebo
Cloud Architect
ShapeBlue

On 22/02/2018, 10:32, "Makrand"  wrote:

Hi All,

Couple of days back we had some iSCSI issue and all the LUNs were
disconnected from Xenserver hosts. After the issue was  fixed and when all
LUNs were back online, for some BIOS checks, we put one of compute node in
Maintenance Mode from cloudstack. It took more than usual time for it to go
into MM (was stuck in PrepateforMaintenance), but it went anyhow. Now
whenever we are trying to cancel its MM, it just fails:- Command failed due
to Internal Server Error.

The logs are indicating below

2018-02-16 09:44:24,291 INFO  [o.a.c.f.j.i.AsyncJobMonitor]
(API-Job-Executor-27:ctx-1e865550 job-72477) Add job-72477 into job
monitoring
2018-02-16 09:44:24,292 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-27:ctx-1e865550 job-72477) Executing AsyncJobVO
{id:72477, userId: 2, accountId: 2,
 instanceType: Host, instanceId: 26, cmd:
org.apache.cloudstack.api.command.admin.host.CancelMaintenanceCmd, cmdInfo:
{"id":"4bca233d-0e61-495c-a522-43800fe311fc","r

esponse":"json","sessionkey":"ZxtGyco2RuYHil/VnglSOgguw5c\u003d","ctxDetails":"{\"com.cloud.host.Host\":\"4bca233d-0e61-495c-a522-43800fe311fc\"}","cmdEventType":"MA

INT.CANCEL","ctxUserId":"2","httpmethod":"GET","_":"1518774059073","uuid":"4bca233d-0e61-495c-a522-43800fe311fc","ctxAccountId":"2","ctxStartEventId":"51924"},
cmdVe
rsion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
null, initMsid: 16143068278473, completeMsid: null, lastUpdated: null,
lastPolled: null, crea
ted: null}
2018-02-16 09:44:24,301 ERROR [c.c.a.ApiAsyncJobDispatcher]
(API-Job-Executor-27:ctx-1e865550 job-72477) Unexpected exception while
executing org.apache.cloudstack.a
pi.command.admin.host.CancelMaintenanceCmd
java.lang.NullPointerException
at

com.cloud.resource.ResourceManagerImpl.doCancelMaintenance(ResourceManagerImpl.java:2083)
at

com.cloud.resource.ResourceManagerImpl.cancelMaintenance(ResourceManagerImpl.java:2140)
at

com.cloud.resource.ResourceManagerImpl.cancelMaintenance(ResourceManagerImpl.java:1127)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at

sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at

sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at

org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at

org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at

org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at

org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at

org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at

org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy147.cancelMaintenance(Unknown Source)
at

org.apache.cloudstack.api.command.admin.host.CancelMaintenanceCmd.execute(CancelMaintenanceCmd.java:102)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141)
at
com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
at

org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
at

org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at

org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at

Re: change primary storage scope

[Andrija Panic]

Hi Piotr,

as far as I know, it's not possible to change in in the normal/official
way, but you can actually change it via DB (test on DEV first and makes
sure to run some tests, deploy VMs, do some live migrations etc...)

UPDATE cloud.storage_pool SET pod_id=1, cluster_id=1, scope='CLUSTER'
WHERE  id=6;

In example above id=6 is the ID value from the storage_pool table for the
storage you are editing (changing scope), and since you are changing from
ZONE (no specific pod/cluster), to CLUSTER (specific CLUSTER inside
specific POD), then also need to define IDs of you POD and cluster (again
these are IDs from the cloud.cluster table and cloud.host_pod_ref table)

Let me know if this works, I needed to change it once in same way, but we
didn't do it eventually.

Cheers,

Andrija

On 22 February 2018 at 08:49, Piotr Pisz  wrote:

> Hello,
>
> Is there any possibility of change primary storage scope from cluster to
> zone?
>
> Regards,
> Piotr
>
>
>


-- 

Andrija Panić

Can't cancel Maintenance Mode on host

[Makrand]

Hi All,

Couple of days back we had some iSCSI issue and all the LUNs were
disconnected from Xenserver hosts. After the issue was  fixed and when all
LUNs were back online, for some BIOS checks, we put one of compute node in
Maintenance Mode from cloudstack. It took more than usual time for it to go
into MM (was stuck in PrepateforMaintenance), but it went anyhow. Now
whenever we are trying to cancel its MM, it just fails:- Command failed due
to Internal Server Error.

The logs are indicating below

2018-02-16 09:44:24,291 INFO  [o.a.c.f.j.i.AsyncJobMonitor]
(API-Job-Executor-27:ctx-1e865550 job-72477) Add job-72477 into job
monitoring
2018-02-16 09:44:24,292 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-27:ctx-1e865550 job-72477) Executing AsyncJobVO
{id:72477, userId: 2, accountId: 2,
 instanceType: Host, instanceId: 26, cmd:
org.apache.cloudstack.api.command.admin.host.CancelMaintenanceCmd, cmdInfo:
{"id":"4bca233d-0e61-495c-a522-43800fe311fc","r
esponse":"json","sessionkey":"ZxtGyco2RuYHil/VnglSOgguw5c\u003d","ctxDetails":"{\"com.cloud.host.Host\":\"4bca233d-0e61-495c-a522-43800fe311fc\"}","cmdEventType":"MA
INT.CANCEL","ctxUserId":"2","httpmethod":"GET","_":"1518774059073","uuid":"4bca233d-0e61-495c-a522-43800fe311fc","ctxAccountId":"2","ctxStartEventId":"51924"},
cmdVe
rsion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
null, initMsid: 16143068278473, completeMsid: null, lastUpdated: null,
lastPolled: null, crea
ted: null}
2018-02-16 09:44:24,301 ERROR [c.c.a.ApiAsyncJobDispatcher]
(API-Job-Executor-27:ctx-1e865550 job-72477) Unexpected exception while
executing org.apache.cloudstack.a
pi.command.admin.host.CancelMaintenanceCmd
java.lang.NullPointerException
at
com.cloud.resource.ResourceManagerImpl.doCancelMaintenance(ResourceManagerImpl.java:2083)
at
com.cloud.resource.ResourceManagerImpl.cancelMaintenance(ResourceManagerImpl.java:2140)
at
com.cloud.resource.ResourceManagerImpl.cancelMaintenance(ResourceManagerImpl.java:1127)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy147.cancelMaintenance(Unknown Source)
at
org.apache.cloudstack.api.command.admin.host.CancelMaintenanceCmd.execute(CancelMaintenanceCmd.java:102)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141)
at
com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2018-02-16 09:44:24,305 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-27:ctx-1e865550 job-72477) Complete async job-72477,
jobStatus: FAILED, resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530}
2018-02-16 09:44:24,320 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
(DirectAgent-303:ctx-d1ac93ce) Done with process of VM state report. host: 1
2018-02-16 09:44:24,322 DEBUG 

Re: Virtual machines do not detect keyboard short cuts

[Dag Sonstebo]

Hi Swastik,

You can’t to my knowledge – the VNC session to your VM is brokered over your 
HTTP session – so what you are asking for is a method for your keyboard 
shortcuts to be intercepted by your browser rather than your OS. 

Keep in mind the use case for the CPVM terminal window is just for 
configuration and troubleshooting – if you want to do more advanced stuff the 
expectation is you set up something like RDP access to Windows or a forwarded 
X-session for a linux VM.

Regards,
Dag Sonstebo
Cloud Architect
ShapeBlue

On 22/02/2018, 10:04, "Swastik Mittal"  wrote:

Hey,

In my virtual machine whenever I use keyboard short cuts for instance
ctrl+alt+t to open terminal, it executes on host and not on my VM. How do I
enable hot keys in cloudstack VM?

Regards
Swastik



dag.sonst...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

Virtual machines do not detect keyboard short cuts

[Swastik Mittal]

Hey,

In my virtual machine whenever I use keyboard short cuts for instance
ctrl+alt+t to open terminal, it executes on host and not on my VM. How do I
enable hot keys in cloudstack VM?

Regards
Swastik

Re: VHD import

[Dag Sonstebo]

Hi Gregoire,

Glad the blog post is of use. It’s a while since I wrote it so I had to re-read 
it myself.  I have not come across this problem – but as you can probably guess 
we luckily don’t have to do this recovery procedure very often.

I can only assume same as yourself that the vhd-util coalesce using the 
downloaded vhd-util binary must give a slightly different format header which 
the import in CloudStack doesn’t like but XS is quite happy about. Also keep in 
mind that the vhd-util from 
http://download.cloud.com.s3.amazonaws.com/tools/vhd-util is slightly different 
than the one you find on the XenServers, so this may also be a factor.

Please let us know how you get on with your investigation – if you find the 
root cause let me know and I’ll add it as a subnote to the original article 
(don’t worry you’ll get the credit ( ).

Regards,
Dag Sonstebo
Cloud Architect
ShapeBlue

On 22/02/2018, 00:10, "Grégoire Lamodière"  wrote:

Dear All,

I recently lost a Xen 7 pool managed by CS 4.11 (hardware issue).

I tried to apply Dag's excellent article 
(http://www.shapeblue.com/recovery-of-vms-to-new-cloudstack-instance/), and 
found a strange behavior :


-  If I directly export the VHD from old NFS primary storage (after 
coalesce vhd with vhd-util) and import to CS using import Template, they always 
fail, with 2 types of error (mainly Failed post download script: vhd remove 
/mnt/SecStorage/2da072e7-a6fe-3e39-b07e-77ec4f34fd49/template/tmpl/2/258/dnld5954086457185109807tmp_
 hidden failed).

-  If I use the same VHD, import-it to a xen server (xe 
vdi-import), and then export it (xe vdi-export), I can successfully import it 
to CS

I made tries with Windows instances, from 15 to 100 Gb, and always the same 
result.
And I also made the same to a CS 4.7 with same results.

A quick look inside createtmplt.sh did not show anything relevant.
I guess the header / footer might have some differences, I'll check this 
tomorrow.

Is anyone aware of this ?
Is there any solution to avoid the intermediate vdi-import / export.

I will dig a bit more to find a proper solution, as it sounds interesting 
to get a disaster migration script working fine.

Regards.

---
Grégoire Lamodière
T/ + 33 6 76 27 03 31
F/ + 33 1 75 43 89 71




dag.sonst...@shapeblue.com 
www.shapeblue.com
53 Chandos Place, Covent Garden, London  WC2N 4HSUK
@shapeblue
  
 

Cloudstack 4.11 : NFS and system VMs NIC issue

[Aican France]

Hi All,



I’m trying to build a small cloudstack lab inside Vmware Workstation 14.0.



I’m using :

-Ubuntu 14.04.5 NFS Server for primary / secondary storage –
192.168.117.129

-Ubuntu 16.04.3 for Cloudstack management server – 192.168.117.135

-Xenserver 7.3 with latest package 2017.1206 – 192.168.117.136



All these servers hosted on the same LAN (192.168.117.x/24), provisioning
VMs IP range on the same LAN.



I’m facing some issues, but may be they are linked :

-Secondary storage on cloudstack dashboard shows 0.00KB/0.00KB

-Secondary Storage VM s-1-VM and console Proxy VM v-2-VM still
pending il Starting state.



VMs are started (Xencenter View) but when logging into VMs console, NIC are
not configured.



Even if I can see the secondary storage configured in my Infrastructure
dashboard, only one NFS mount point appears on xenserver side :

192.168.117.129:/nfs/primary/ on
/run/sr-mount/e666f671-a903-16b1-7676-ff0ad35f3db4 type nfs
(rw,relatime,vers=3,rsize=262144,wsize=262144,namlen=255,acdirmin=0,acdirmax=0,soft,proto=tcp,timeo=100,retrans=12,sec=sys,mountaddr=192.168.117.129,mountvers=3,mountport=39208,mountproto=tcp,local_lock=none,addr=192.168.117.129)

/dev/mapper/XSLocalEXT--ccb0abd6--d830--bc2e--8372--daa16a7f5d78-ccb0abd6--d830--bc2e--8372--daa16a7f5d78
on /run/sr-mount/ccb0abd6-d830-bc2e-8372-daa16a7f5d78 type ext3
(rw,relatime,data=ordered)



Management-serv.log shows :

2018-02-22 08:50:06,346 DEBUG [c.c.s.StatsCollector]
(StatsCollector-2:ctx-79cce329) (logid:3cf7734e) There is no secondary
storage VM for secondary storage host nfs://192.168.117.129/nfs/secondary



Some sample logs or the Vms :



2018-02-22 08:51:31,840 DEBUG [c.c.h.x.r.w.x.CitrixStartCommandWrapper]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) 1. The VM s-1-VM is in
Starting state.

2018-02-22 08:51:31,874 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) Created VM
8e0a421d-cc05-baa4-49f2-6e75fab04d5a for s-1-VM

2018-02-22 08:51:31,880 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) PV args are
%template=domP%type=secstorage%host=192.168.117.135%port=8250%name=s-1-VM%zone=1%pod=1%guid=s-1-VM%workers=5%resource=org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource%instance=SecStorage%sslcopy=false%role=templateProcessor%mtu=1500%eth2ip=192.168.117.40%eth2mask=255.255.255.0%gateway=192.168.117.2%eth0ip=169.254.0.32%eth0mask=255.255.0.0%eth1ip=192.168.117.6%eth1mask=255.255.255.0%mgmtcidr=
192.168.117.0/24%localgw=192.168.117.2%private.network.device=eth1%internaldns1=192.168.117.2%dns1=8.8.8.8%nfsVersion=null

2018-02-22 08:51:49,208 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) Creating VIF for s-1-VM on
nic [Nic:Guest-192.168.117.40-vlan://untagged]

2018-02-22 08:51:49,234 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) Creating VIF for s-1-VM on
nic [Nic:Control-169.254.0.32-null]

2018-02-22 08:51:49,664 DEBUG [c.c.h.x.r.CitrixResourceBase]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) Creating VIF for s-1-VM on
nic [Nic:Management-192.168.117.6-null]

2018-02-22 08:52:01,239 INFO  [c.c.h.x.r.w.x.CitrixStartCommandWrapper]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) Programmed default network
rules for s-1-VM

2018-02-22 08:52:01,239 DEBUG [c.c.h.x.r.w.x.CitrixStartCommandWrapper]
(DirectAgent-10:ctx-67d15938) (logid:2b6e4061) 2. The VM s-1-VM is in
Running state.

2018-02-22 08:54:11,074 DEBUG [c.c.a.t.Request]
(DirectAgentCronJob-3:ctx-1258f3db) (logid:2948b9e7) Seq
1-4371306388316487683: Processing:  { Ans: , MgmtId: 52231347383, via:
1(xenserver_7.3-cs4.11), Ver: v1, Flags: 10,
[{"com.cloud.agent.api.ClusterVMMetaDataSyncAnswer":{"_clusterId":1,"_vmMetaDatum":{"s-1-VM":"device-model:qemu-trad;apic:true;viridian:true;timeoffset:0;pae:true;acpi:1;hpet:true;nx:true","v-2-VM":"device-model:qemu-trad;apic:true;viridian:true;timeoffset:0;pae:true;acpi:1;hpet:true;nx:true"},"_isExecuted":false,"result":true,"wait":0}}]
}

2018-02-22 08:57:11,054 DEBUG [c.c.a.t.Request]
(DirectAgentCronJob-9:ctx-6ed1ca09) (logid:2948b9e7) Seq
1-4371306388316487683: Processing:  { Ans: , MgmtId: 52231347383, via:
1(xenserver_7.3-cs4.11), Ver: v1, Flags: 10,
[{"com.cloud.agent.api.ClusterVMMetaDataSyncAnswer":{"_clusterId":1,"_vmMetaDatum":{"s-1-VM":"device-model:qemu-trad;apic:true;viridian:true;timeoffset:0;pae:true;acpi:1;hpet:true;nx:true","v-2-VM":"device-model:qemu-trad;apic:true;viridian:true;timeoffset:0;pae:true;acpi:1;hpet:true;nx:true"},"_isExecuted":false,"result":true,"wait":0}}]
}

2018-02-22 09:00:11,023 DEBUG [c.c.a.t.Request]
(DirectAgentCronJob-10:ctx-49d205af) (logid:2948b9e7) Seq
1-4371306388316487683: Processing:  { Ans: , MgmtId: 52231347383, via:
1(xenserver_7.3-cs4.11), Ver: v1, Flags: 10,