Thanks a ton Dag.....this is what I love about cloudstack email lists....explain ur situation thoroughly and someone will improvise it.
BTW all primary storage is under iSCSI. Secondary under NFS. Will iSCSI add any other step to plan? This ain't a big zone (less than 100 VMS) and customer access to cloud is via another GUI portal.....which is completely blocked during process. I will give mm of primary storage a shot though. -- Makrand Sent from smartphone, using (over) smart autocorrect. Thus typos! On May 13, 2016 3:53 PM, "Dag Sonstebo" <[email protected]> wrote: > Hi Makrand, > > first of all - using the same storage appliance for all purposes > (management VMs, primary and secondary storage) is fairly common, and fully > viable as long as you run separate LUNs / NFS shares and tie down access > accordingly. > > With regards to your shutdown / startup steps you're pretty much spot on. > You haven't mentioned if you use NFS or iSCSI but the steps should be more > or less the same. > A couple of suggestions inline below: > > > A) Shutdown all VMs in following order:- First user VMs, then VRs & then > System VMs. > >>> You probably want to disable the zone first of all to stop any further > user access happening during any shutdown and startup procedures. > >>> With regards to shutting down VMs you can achieve this by putting each > primary storage pool into maintenance - this will shut down all running VMs > and system VMs. > > B) Put all XEN hosts in MM (Maintenance mode) from cloudstack. > >>> Once they are in maintenance you will still have to shut them down > from XenCenter or command line. > > 1) Verify that all hosts are down (Shutdown them one by one) from XEN > Center > > C) ssh into Management Server VM. Backup cloudstack DB save it on JumpBOX > (Use WinSCP) > >>> good practice, make sure you dump both the cloud and cloud_usage > databases if you have both of these present > > D) Shutdown Cloudstack Management Server VM and other VMs on management > cluster. This needs to be done by issuing 'shutdown' command from OS level. > > E) Shutdown Management cluster XEN hosts. > > F) Upgrade the storage. (Reboot is needed at end.......hence so many > shutdowns) > > G) Once upgrade is done start the Machines (Physical and Virtual) in > reveres order than that of shutdown. (Step E to A above) Something like > below > > 1) Management clsuter XEN hosts > 2) Cloudstack management server VM and other management cluster VMs > 3) XEN Hypervisor Nodes under cloud. > 4) Take XEN hosts out of MM from cloudstack admin GUI. > >>> Take primary storage out of maintenance mode if you followed the > suggestion above. Also re-enable the zone. > > 5) Start SSVM and console proxy VM (I guess this time cloudstack will > recreate CPVM and SSVM if we start first user VM.....not sure if this > step will be needed) > >>> Correct, as long as everything is healthy these will be started > automatically. > > 6) Customer VMs and corresponding VRs > H) Verify from and try to deploy test VM. > > -- > > Last time same upgrade happened for one of similar zone and there was issue > of mapping primary storage LUNs with XEN hosts. Restart of hosts did trick > that time and hence my manager wants to shutdown all physical hosts this > time (Since all VMs have their disks on this storage box...which will > reboot once or twice during upgrade) > >>> Yes this seems to be a logical precaution, it ensures storage is > correctly rescanned on startup. > > -- > > 1) Is it necessary to put XEN hosts in MM from cloudstack? (If I am > shutting down all VMs). > >>> yes, in general always make sure CloudStack is aware of what is going > on the underlying hosts, it keeps everything in synch. > > 2) After I bring UP Management server in step G-1, will it still have host > in MM? or What would it try to do after its up? (Assuming all hosts in > cluster are down at that time. > >>> Yes they will still be in maintenance. > > > > > Hope this helps, good luck with your upgrade. > > > Regards, > Dag Sonstebo > Cloud Architect > ShapeBlue > > > > > > > On 13/05/2016, 07:58, "Makrand" <[email protected]> wrote: > > >Guys, > > > >Need little help. So in my second week on new job, I am suppose to upgrade > >Nexeta (Primary storage and secondary storage for cloud) OS. > > > >Overview of setup:- > > > >1Zone>>1POD>>1 Cluster>>8 Hosts > > > >Hypervisor:- Citrix Xenserer 6.5 (Free edition) > > > >Storage Primary (Cluster Level):- On nexenta > > > >Secondary Storage:- On same nexenta BOX (which caters primary) > > > >Management Node is VM (Ubuntu 12004 LTS) whos root disk resides again on > >primary storage coming from same nexenta BOX. Seems LUN is different. This > >VM is running on Management cluster (2 XEN 6.5 hosts). > > > > > >Well not sure why and how they kept single storage BOX for all primary and > >secondary. Pretty wiered. It was done by a Vendor or someone long back. > >Thats whole different story. > > > >Any-who, coming to point of this email. I've chalked out plan like below. > >------------------------------------------------------ > >A) Shutdown all VMs in following order:- First user VMs, then VRs & then > >System VMs. > > > >B) Put all XEN hosts in MM (Maintenance mode) from cloudstack. > > > > 1) Verify that all hosts are down (Shutdown them one by one) from XEN > >Center > > > >C) ssh into Management Server VM. Backup cloudstack DB save it on JumpBOX > >(Use WinSCP) > > > >D) Shutdown Cloudstack Management Server VM and other VMs on management > >cluster. This needs to be done by issuing 'shutdown' command from OS > level. > > > >E) Shutdown Management cluster XEN hosts. > > > >F) Upgrade the storage. (Reboot is needed at end.......hence so many > >shutdowns) > > > >G) Once upgrade is done start the Machines (Physical and Virtual) in > >reveres order than that of shutdown. (Step E to A above) Something like > >below > > > > 1) Management clsuter XEN hosts > > 2) Cloudstack management server VM and other management cluster VMs > > 3) XEN Hypervisor Nodes under cloud. > > 4) Take XEN hosts out of MM from cloudstack admin GUI. > > 5) Start SSVM and console proxy VM (I guess this time cloudstack will > >recreate CPVM and SSVM if we start first user VM.....not sure if this > >step will be needed) > > 6) Customer VMs and corresponding VRs > > > >H) Verify from and try to deploy test VM. > >------------------------------------------------------ > > > >All this downtime is ok. Do you think I am missing anything. Any comments > >on improvising? Should I expect some more glitches? What was your previous > >experience with primary storage upgrade. > > > >Note:- > >Last time same upgrade happened for one of similar zone and there was > issue > >of mapping primary storage LUNs with XEN hosts. Restart of hosts did trick > >that time and hence my manager wants to shutdown all physical hosts this > >time (Since all VMs have their disks on this storage box...which will > >reboot once or twice during upgrade) > > > >BTW > > > >1) Is it necessary to put XEN hosts in MM from cloudstack? (If I am > >shutting down all VMs). > > > >2) After I bring UP Management server in step G-1, will it still have host > >in MM? or What would it try to do after its up? (Assuming all hosts in > >cluster are down at that time. > > > >Thanks for reading. > > > >-- > >Best, > >Makrand > > Regards, > > Dag Sonstebo > > [email protected] > www.shapeblue.com > 53 Chandos Place, Covent Garden, London WC2N 4HSUK > @shapeblue >
