Thank you Bobby and Daan for the update. However I have not encountered such issue while doing dev test with Vmware 5.5 & 6.5.
Regards, Pavan Aravapalli. ________________________________ From: Daan Hoogland <daan.hoogl...@gmail.com> Sent: 19 May 2020 20:56 To: users <users@cloudstack.apache.org> Cc: d...@cloudstack.apache.org <d...@cloudstack.apache.org> Subject: Re: [VOTE] Apache CloudStack 4.14.0.0 RC3 Thanks Bobby, All, I've been closely working with Bobby and seen the same things. Does anybody see any issues releasing 4.14 based on this code? I can confirm that it is not Pavernalli's UEFI PR and we should not create a new PR to revert it. thanks for all of your patience, (this is me giving a binding +1) On Tue, May 19, 2020 at 5:04 PM Boris Stoyanov <boris.stoya...@shapeblue.com> wrote: > Hi guys, > > I've done more testing around this and I can now confirm it has nothing to > do with cloudstack code. > > I've tested it with rc3, reverted UEFI PR and 4.13.1 (which does not > happen to have the feature at all). Also I've used a matrix of VMware > version of 6.0u2, 6.5u2 and 6.7u3. > > The bug is reproducible with all the cloudstack versions, and only vmware > 6.7u3, I was not able to reproduce this with 6.5/6.0. All of my results > during testing show it must be related to that specific version of VMware. > > Therefore I'm reversing my '-1' and giving a +1 vote on the RC. I think it > needs to be included in release notes to refrain from that version for now > until further investigation is done. > > Thanks, > Bobby. > > On 19.05.20, 10:08, "Boris Stoyanov" <boris.stoya...@shapeblue.com> > wrote: > > Indeed it is severe, but please note it's a corner case which was > unearthed almost by accident. It falls down to using a new feature of > selecting a boot protocol and the template must be corrupted. So with > already existing templates I would not expect to encounter it. > > As for recovery, we've managed to recover vCenter and Cloudstack after > reboots of the vCenter machine and the Cloudstack management service. > There's no exact points to recover for now, but restart seems to work. > By graceful failure I mean, cloudstack erroring out the deployment and > VM finished in ERROR state, meanwhile connection and operability with > vCenter cluster remains the same. > > We're currently exploring options to fix this, one could be to disable > the feature for VMWare and work to introduce more sustainable fix in next > release. Other is to look for more guarding code when installing a > template, since VMware doesn’t actually allow you install that particular > template but cloudstack does. We'll keep you posted. > > Thanks, > Bobby. > > On 18.05.20, 23:01, "Marcus" <shadow...@gmail.com> wrote: > > The issue sounds severe enough that a release note probably won't > suffice - > unless there's a documented way to recover we'd never want to > leave a > system susceptible to being unrecoverable, even if it's rarely > triggered. > > What's involved in "failing gracefully"? Is this a small fix, or an > overhaul? Perhaps the new feature could be disabled for VMware, or > disabled altogether until a fix is made in a patch release. > > Does it only affect new templates, or is there a risk that an > existing > template out in vSphere could suddenly cause problems? > > On Mon, May 18, 2020 at 12:49 AM Boris Stoyanov < > boris.stoya...@shapeblue.com> wrote: > > > Hi guys, > > > > A little further info on this, it appears when we use a > corrupted template > > and UEFI/Legacy mode when deploy a VM, it breaks the connection > between > > cloudstack and vCenter. > > > > All hosts become unreachable and basically the cluster is not > functional, > > have not investigated a way to recover this but seems like a > huge mess.. > > Please note that user is not able to register such template in > vCenter > > directly, but cloudstack allows using it. > > > > Open to discuss if we'll fix this, since it's expected users to > use > > working templates, I think we should be failing gracefully and > such action > > should not be able to create downtime on such a large scale. > > > > I believe the boot type feature is new one and it's not > available in older > > releases, so this issue should be limited to 4.14/current master. > > > > Thanks, > > Bobby. > > > > On 15.05.20, 17:07, "Boris Stoyanov" < > boris.stoya...@shapeblue.com> > > wrote: > > > > I'll have to -1 RC3, we've discovered details about an issue > which is > > causing severe consequences with a particular hypervisor in the > afternoon. > > We'll need more time to investigate before disclosing. > > > > Bobby. > > > > On 15.05.20, 9:12, "Boris Stoyanov" < > boris.stoya...@shapeblue.com> > > wrote: > > > > +1 (binding) > > > > I've executed upgrade tests with the following > configurations: > > > > 4.13.1 with KVM on CentOS7 hosts > > 4.13 with VMware6.5 hosts > > 4.11.3 with KVM on CentOS7 hosts > > 4.11.2 with XenServer7 hosts > > 4.11.1 with VMware 6.7 > > 4.9.3 with XenServer 7 hosts > > 4.9.2 with KVM on CentOS 7 hosts > > > > Also I've run basic lifecycle operations on the following > > components: > > VMs > > Volumes > > Infra (zones, pod, clusters, hosts) > > Networks > > and more > > > > I did not come across any problems during this testing. > > > > Thanks, > > Bobby. > > > > > > On 11.05.20, 18:21, "Andrija Panic" < > andrija.pa...@gmail.com> > > wrote: > > > > Hi All, > > > > I've created a 4.14.0.0 release (RC3), with the > following > > artefacts up for > > testing and a vote: > > > > Git Branch and Commit SH: > > > > > https://gitbox.apache.org/repos/asf?p=cloudstack.git;a=shortlog;h=refs/heads/4.14.0.0-RC20200511T1503 > > Commit: 6f96b3b2b391a9b7d085f76bcafa3989d9832b4e > > > > Source release (checksums and signatures are > available at the > > same > > location): > > > https://dist.apache.org/repos/dist/dev/cloudstack/4.14.0.0/ > > > > PGP release keys (signed using 3DC01AE8): > > > https://dist.apache.org/repos/dist/release/cloudstack/KEYS > > > > The vote will be open until 14th May 2020, 17.00 CET > (72h). > > > > For sanity in tallying the vote, can PMC members > please be > > sure to indicate > > "(binding)" with their vote? > > > > [ ] +1 approve > > [ ] +0 no opinion > > [ ] -1 disapprove (and reason why) > > > > Additional information: > > > > For users' convenience, I've built packages from > > 6f96b3b2b391a9b7d085f76bcafa3989d9832b4e and > published RC3 > > repository here: > > http://packages.shapeblue.com/testing/41400rc3/ > (CentOS 7 and > > Debian/generic, both with noredist support) > > and here > > > > > https://download.cloudstack.org/testing/4.14.0.0-RC20200506T2028/ubuntu/bionic/ > > (Ubuntu 18.04 specific, no noredist support - > thanks to > > Gabriel): > > > > The release notes are still work-in-progress, but > for the > > upgrade > > instructions (including the new systemVM templates) > you may > > refer to the > > following URL: > > > > > https://acs-www.shapeblue.com/docs/WIP-PROOFING/pr112/upgrading/index.html > > > > 4.14.0.0 systemVM templates are available from here: > > http://download.cloudstack.org/systemvm/4.14/ > > > > NOTES on issues fixed in this RC3 release: > > > > (this one does *NOT* require a full retest if you > were testing > > RC1/RC2 > > already - just if you were affected this issue): > > - https://github.com/apache/cloudstack/pull/4064 - > affects > > hostnames when > > attaching a VM to additional networks > > > > Regards, > > > > > > Andrija Panić > > > > > > > > > > > > boris.stoya...@shapeblue.com > > www.shapeblue.com<http://www.shapeblue.com> > > 3 London Bridge Street, 3rd floor, News Building, London > SE1 9SGUK > > @shapeblue > > > > > > > > > > > > > > boris.stoya...@shapeblue.com > > www.shapeblue.com<http://www.shapeblue.com> > > 3 London Bridge Street, 3rd floor, News Building, London SE1 > 9SGUK > > @shapeblue > > > > > > > > > > > > boris.stoya...@shapeblue.com > www.shapeblue.com<http://www.shapeblue.com> > 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK > @shapeblue > > > > > > > boris.stoya...@shapeblue.com > www.shapeblue.com<http://www.shapeblue.com> > 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK > @shapeblue > > > > -- Daan
