Re: Can't start systemVM in a new advanced zone deployment

[Sergey Levitskiy]

Have you tried deleting template from PS and let ACS to recopy it again? If the 
issue is reproducible we can try to look what is wrong with VMDK. Please post 
content of 533b6fcf3fa6301aadcc2b168f3f999a.vmdk , 
533b6fcf3fa6301aadcc2b168f3f999a-000001.vmdk and 
533b6fcf3fa6301aadcc2b168f3f999a.vmx (their equitant after ACS finishes copying 
template). Also from one of your ESX hosts output of this
ls -al /vmfs/volumes
ls -al /vmfs/volumes/*/533b6fcf3fa6301aadcc2b168f3f999a (their equitant after 
ACS finishes copying template)

 Can you also post management server log starting from the point you unregister 
and delete template from the vCenter.

On 6/4/19, 8:37 AM, "Yiping Zhang" <yipzh...@adobe.com.INVALID> wrote:

    I have manually imported the OVA to vCenter and successfully cloned a VM 
instance with it, on the same NFS datastore.
    
    
    On 6/4/19, 8:25 AM, "Sergey Levitskiy" <serg...@hotmail.com> wrote:
    
        I would suspect the template is corrupted on the secondary storage. You 
can try disabling/enabling link clone feature and see if it works the other 
way. 
        vmware.create.full.clone                        false
        
        Also systemVM template might have been generated on a newer version of 
vSphere and not compatible with ESXi 6.5. What you can do to validate this is 
to manually deploy OVA that is in Secondary storage and try to spin up VM from 
it directly in vCenter.
        
        
        
        On 6/3/19, 5:41 PM, "Yiping Zhang" <yipzh...@adobe.com.INVALID> wrote:
        
            Hi, list:
            
            I am struggling with deploying a new advanced zone using ACS 
4.11.2.0 + vSphere 6.5 + NetApp volumes for primary and secondary storage 
devices. The initial setup of CS management server, seeding of systemVM 
template, and advanced zone deployment all went smoothly.
            
            Once I enabled the zone in web UI and the systemVM template gets 
copied/staged on to primary storage device. But subsequent VM creations from 
this template would fail with errors:
            
            
            2019-06-03 18:38:15,764 INFO  [c.c.h.v.m.HostMO] 
(DirectAgent-7:ctx-d01169cb esx-0001-a-001.example.org, job-3/job-29, cmd: 
CopyCommand) VM 533b6fcf3fa6301aadcc2b168f3f999a not found in host cache
            
            2019-06-03 18:38:17,017 INFO  [c.c.h.v.r.VmwareResource] 
(DirectAgent-4:ctx-08b54fbd esx-0001-a-001.example.org, job-3/job-29, cmd: 
CopyCommand) VmwareStorageProcessor and VmwareStorageSubsystemCommandHandler 
successfully reconfigured
            
            2019-06-03 18:38:17,128 INFO  [c.c.s.r.VmwareStorageProcessor] 
(DirectAgent-4:ctx-08b54fbd esx-0001-a-001.example.org, job-3/job-29, cmd: 
CopyCommand) creating full clone from template
            
            2019-06-03 18:38:17,657 INFO  [c.c.h.v.u.VmwareHelper] 
(DirectAgent-4:ctx-08b54fbd esx-0001-a-001.example.org, job-3/job-29, cmd: 
CopyCommand) [ignored]failed toi get message for exception: Error caused by 
file 
/vmfs/volumes/afc5e946-03bfe3c2/533b6fcf3fa6301aadcc2b168f3f999a/533b6fcf3fa6301aadcc2b168f3f999a-000001.vmdk
            
            2019-06-03 18:38:17,658 ERROR [c.c.s.r.VmwareStorageProcessor] 
(DirectAgent-4:ctx-08b54fbd esx-0001-a-001.example.org, job-3/job-29, cmd: 
CopyCommand) clone volume from base image failed due to Exception: 
java.lang.RuntimeException
            
            Message: Error caused by file 
/vmfs/volumes/afc5e946-03bfe3c2/533b6fcf3fa6301aadcc2b168f3f999a/533b6fcf3fa6301aadcc2b168f3f999a-000001.vmdk
            
            
            
            If I try to create “new VM from template” 
(533b6fcf3fa6301aadcc2b168f3f999a) on vCenter UI manually,  I will receive 
exactly the same error message. The name of the VMDK file in the error message 
is a snapshot of the base disk image, but it is not part of the original 
template OVA on the secondary storage.  So, in the process of copying the 
template from secondary to primary storage, a snapshot got created and the disk 
became corrupted/unusable.
            
            Much later in the log file,  there is another error message “failed 
to fetch any free public IP address” (for ssvm, I think).  I don’t know if 
these two errors are related or if one is the root cause for the other error.
            
            The full management server log is uploaded as 
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpastebin.com%2Fc05wiQ3R&amp;data=02%7C01%7Cyipzhang%40adobe.com%7C1541cabf6c9b42873e0708d6e900da78%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C0%7C636952587209705577&amp;sdata=qDALhWV4NoLWyidmyz6f1MUSERFWb9XN%2B7GtaAnExaU%3D&amp;reserved=0
            
            Any help or insight on what went wrong here are much appreciated.
            
            Thanks
            
            Yiping