Re: Can't start systemVM in a new advanced zone deployment

2019-06-06 Thread Sergey Levitskiy
org Subject: Re: Can't start systemVM in a new advanced zone deployment Hi, Sergey: I found more logs in vpxa.log ( the esxi hosts are using UTC time zone, so I was looking at wrong time periods earlier). I have uploaded more logs into

Re: Can't start systemVM in a new advanced zone deployment

2019-06-06 Thread Yiping Zhang
e 2019 23:38 To: users@cloudstack.apache.org Subject: Re: Can't start systemVM in a new advanced zone deployment Hi, Sergey: I found more logs in vpxa.log ( the esxi hosts are using UTC time zone, so I was looking at wrong time periods earlier). I have upl

Re: Can't start systemVM in a new advanced zone deployment

2019-06-06 Thread Sergey Levitskiy
Street, London WC2E 9DPUK @shapeblue -Original Message- From: Yiping Zhang Sent: 05 June 2019 23:38 To: users@cloudstack.apache.org Subject: Re: Can't start systemVM in a new advanced zone deployment Hi, Sergey: I found m

RE: Can't start systemVM in a new advanced zone deployment

2019-06-06 Thread Paul Angus
peblue.com Amadeus House, Floral Street, London WC2E 9DPUK @shapeblue -Original Message- From: Yiping Zhang Sent: 05 June 2019 23:38 To: users@cloudstack.apache.org Subject: Re: Can't start systemVM in a new advanced zone deployment Hi, Sergey: I found more logs in vpxa.log ( the

Re: Can't start systemVM in a new advanced zone deployment

2019-06-05 Thread Yiping Zhang
Hi, Sergey: I found more logs in vpxa.log ( the esxi hosts are using UTC time zone, so I was looking at wrong time periods earlier). I have uploaded more logs into pastebin. From these log entries, it appears that when copying template to VM, it tried to open destination VMDK file and got

Re: Can't start systemVM in a new advanced zone deployment

2019-06-05 Thread Sergey Levitskiy
Some operations log get transferred to vCenter log vpxd.log. It is not straightforward to trace I but Vmware will be able to help should you open case with them. On 6/5/19, 11:39 AM, "Yiping Zhang" wrote: Hi, Sergey: During the time period when I had problem cloning template,

Re: Can't start systemVM in a new advanced zone deployment

2019-06-05 Thread Yiping Zhang
Hi, Sergey: During the time period when I had problem cloning template, there are only a few unique entries in vmkernel.log, and they were repeated hundreds/thousands of times by all the cpu cores: 2019-06-02T16:47:00.633Z cpu9:8491061)FSS: 6751: Failed to open file 'hpilo-d0ccb15';

Re: Can't start systemVM in a new advanced zone deployment

2019-06-05 Thread Sergey Levitskiy
This must be specific to that environment. For a full clone mode ACS simply calls cloneVMTask of vSphere API so basically until cloning of that template succeeds when attmepted in vSphere client it would keep failing in ACS. Can you post vmkernel.log from your ESX host esx-0001-a-001? On

Re: Can't start systemVM in a new advanced zone deployment

2019-06-05 Thread Andrija Panic
Yiping, if you are sure you can reproduce the issue, it would be good to raise a GitHub issue and provide as much detail as possible. Andrija On Wed, 5 Jun 2019 at 05:29, Yiping Zhang wrote: > Hi, Sergey: > > Thanks for the tip. After setting vmware.create.full.clone=false, I was > able to

Re: Can't start systemVM in a new advanced zone deployment

2019-06-04 Thread Yiping Zhang
Hi, Sergey: Thanks for the tip. After setting vmware.create.full.clone=false, I was able to create and start system VM instances.However, I feel that the underlying problem still exists, and I am just working around it instead of fixing it, because in my lab CloudStack instance with the

Re: Can't start systemVM in a new advanced zone deployment

2019-06-04 Thread Sergey Levitskiy
Everything looks good and consistent including all references in VMDK and its snapshot. I would try these 2 routes: 1. Figure out what vSphere error actually means from vmkernel log of ESX when ACS tries to clone the template. If the same error happens while doing it outside of ACS then a

Re: Can't start systemVM in a new advanced zone deployment

2019-06-04 Thread Yiping Zhang
Hi, Sergey: Thanks for the help. By now, I have dropped and recreated DB, re-deployed this zone multiple times, blow away primary and secondary storage (including all contents on them) , or just delete template itself from primary storage, multiple times. Every time I ended up with the same

Re: Can't start systemVM in a new advanced zone deployment

2019-06-04 Thread Sergey Levitskiy
Have you tried deleting template from PS and let ACS to recopy it again? If the issue is reproducible we can try to look what is wrong with VMDK. Please post content of 533b6fcf3fa6301aadcc2b168f3f999a.vmdk , 533b6fcf3fa6301aadcc2b168f3f999a-01.vmdk and 533b6fcf3fa6301aadcc2b168f3f999a.vmx

Re: Can't start systemVM in a new advanced zone deployment

2019-06-04 Thread Yiping Zhang
I have manually imported the OVA to vCenter and successfully cloned a VM instance with it, on the same NFS datastore. On 6/4/19, 8:25 AM, "Sergey Levitskiy" wrote: I would suspect the template is corrupted on the secondary storage. You can try disabling/enabling link clone feature and

Re: Can't start systemVM in a new advanced zone deployment

2019-06-04 Thread Sergey Levitskiy
I would suspect the template is corrupted on the secondary storage. You can try disabling/enabling link clone feature and see if it works the other way. vmware.create.full.clonefalse Also systemVM template might have been generated on a newer version of vSphere and not

Can't start systemVM in a new advanced zone deployment

2019-06-03 Thread Yiping Zhang
Hi, list: I am struggling with deploying a new advanced zone using ACS 4.11.2.0 + vSphere 6.5 + NetApp volumes for primary and secondary storage devices. The initial setup of CS management server, seeding of systemVM template, and advanced zone deployment all went smoothly. Once I enabled the