Our biggest problem in this situation would be VM. if we restore from a backup on Monday in a Dark Network and switch over a weekend, is there a way to grab newer Voicemails to put on the "new" (upgraded) UC at the time of the switch?
Scott On Sat, Oct 29, 2016 at 2:03 PM, Peter Slow <peter.s...@gmail.com> wrote: > Hey guys. At a Co. I was working at with a megacluster, we took backups of > the production cluster, and restored to a newly installed cluster on a > "dark" net, and I NAT'd all the VMs so they could run and be upgraded > separately and rebooted and tested with a few phones on a private network. > Then I used some VMware Power CLI scripts to cut over each datacenter in a > few seconds... ECATS was involved as well - there is a little bit of a > precedent, though I recognize that we tailored that specifically for the > customer's environment, and it might not work as well in some others. > It should also be noted that the NEW (and subsequently upgraded, and > switched into production) VMs were built using the actual TAC supported > install process, as opposed to being cloned. The fact that the megacluster > had 21 servers in basically equated to "not enough time to roll back before > monday" for a number of reasons, an issue this upgrade & cutover method > alleviated. I think it was an 8.6 to 9 upgrade. Because of some block level > stuff, sine VMware wasn't supported (IIRC) in the starting version, after > the upgrade of the 2nd set (referred to as "new" earlier) we took another > back up and restored to *another* cluster, a set of newly installed 9.x VMs > in *another* "dark" net. > > BE CAREFUL IF YOU DO THIS > I did 5 production clusters this way - Having two to three sets of > identical servers, with identical IPs WILL get confusing, i don't care how > smart you are. > Use login banners saying which cluster this server is in. It's annoying > but apply them to each node so you are forced to see and can confirm you're > logging in to ccmadmin on to cluster 1, 2 or 3. Having the cluster built > before cutover lets you do things like fix an issue with a sub, or rebuild > a node as many times as you want when/if an install fails. It's easy to get > confused and blast a sub on the production cluster by accident at 3 pm on > when you've got enough servers and have to admin one cluster, and build > another, when you have multiple sets of identical things, and applying 100 > banners is less annoying than the consequences of you or your customer > screwing this up. > When you need to restart/zap a UC VM that there are multiples of - require > two people to look at it. Have your team lead or customer say "yes, that's > the right one" before you click that button. > Be very careful with naming things so you can safely script things in a > low-risk fashion (E.g., so you can't reboot the wrong set of VMs while > working on build #2) > > -Peter > > On Sat, Oct 29, 2016 at 1:28 PM, Ryan Huff <ryanh...@outlook.com> wrote: > >> As it always has, IMO, it comes down to the nature and context of your >> client. For a true 8by5 SMB; large after hours maintenance windows >> generally aren't a problem as long as some sort of treatment is applied to >> the main ingress. >> >> >> However, the PS engagements I generally work with are medical/healthcare, >> specialized government and 24/7 private industries. If I would tell any of >> my clients: >> >> >> "starting at 9PM what we're gonna do is reboot the phone system and watch >> paint dry. In the case of an RU, we're gonna do that twice or thrice. Your >> phones will do a registration dance anywhere from 30 - 240 minutes." -I >> would get escorted out of their building. >> >> >> So as was mentioned earlier; we've all developed our 'ways' of shrinking >> the actual 'cut over window' to a matter of minutes. Those developments are >> rooted in the needs of our individual clients and spurred by the >> capabilities (some say limitations) of the software's upgrade functionality. >> >> >> Ideally, the inactive partition would have an externally addressable >> method where a unique/independent software version could be installed and >> run in tandem and the active partition had a 'publish' feature where when >> activated, would publish the active Informix schema to the inactive >> Informix schema (using a babble-fish style translator to comp the >> differences in schema versions). Then, your cut over is a matter of >> 'publish' and swap the partitions. >> >> >> All of that is possible, but would require a serious re-tool at the >> RedHat level. >> >> >> >> = Ryan = >> >> >> >> >> ------------------------------ >> *From:* cisco-voip <cisco-voip-boun...@puck.nether.net> on behalf of >> Erick Bergquist <erick...@gmail.com> >> *Sent:* Saturday, October 29, 2016 3:31 PM >> *To:* Lelio Fulgenzi >> >> *Cc:* cisco voip >> *Subject:* Re: [cisco-voip] Not supported I'm sure..... but what do you >> think? >> >> Just chiming in on this. >> >> I've done plenty of SU updates using the inactive partition method >> without any issues for most part. I've also done RU updates which are >> fine because updates are planned and done during mtce windows. Most of >> the work I've done is inplace. >> >> Sometimes the mtce window has to be extended due to longer then >> expected time to install the update. This is what is nice about the SU >> update, you can install it ahead of time. If I have a short mtce >> window, I push these a day ahead sometimes just to make sure the SU is >> done installing on all the servers before the reboot / mtce window. >> However, a RU update that needs reboots during the upgrade is >> sometimes harder to get especially if it is a 24x7 shop with minimum >> downtime. >> >> I really haven't had any real issues with any method (SU/RU) except >> one time during switching the servers on different versions where the >> servers were on mismatched versions one subscriber call processing got >> hung up and calls to phones registered on that node were busy. My fix >> was to stop the call manager server and cti manager service >> immediately which solved the problem until that node was switched to >> new version. >> >> The other time I've had issues was with the 10.5.2 SU3 flat version >> and the switch version bug on Unity, but due to way unity works the >> other server was up and running so no real impact to calls. >> >> However, clients are wanting these patches done more and more with no >> impact to calls or very little downtime for 24x7 organizations with >> call centers that operate 24x7. In those upgrade scenarios you need >> this SU/RU update method and patch process to work flawlessly with >> less interruption as possible and hopefully no glitches. >> >> >> On Thu, Oct 27, 2016 at 3:16 PM, Lelio Fulgenzi <le...@uoguelph.ca> >> wrote: >> > >> > This is exactly the reason I did my upgrades off line and swapped out >> the >> > hardware during the maintenance window. >> > >> > >> > That being said, it still took some time to do all the swapping. Just >> > looking at my notes from the last upgrade and it was about 2-3 hours. >> Much >> > of that was due to the amount of time the servers take to shutdown and >> > restart. >> > >> > >> > --- >> > Lelio Fulgenzi, B.A. >> > Senior Analyst, Network Infrastructure >> > Computing and Communications Services (CCS) >> > University of Guelph >> > >> > 519-824-4120 Ext 56354 >> > le...@uoguelph.ca >> > www.uoguelph.ca/ccs >> > Room 037, Animal Science and Nutrition Building >> > Guelph, Ontario, N1G 2W1 >> > >> > >> > >> > ________________________________ >> > From: cisco-voip <cisco-voip-boun...@puck.nether.net> on behalf of >> Scott >> > Voll <svoll.v...@gmail.com> >> > Sent: Thursday, October 27, 2016 4:50 PM >> > To: Stephen Welsh; Ryan Ratliff >> > Cc: cisco voip >> > >> > Subject: Re: [cisco-voip] Not supported I'm sure..... but what do you >> think? >> > >> > I'm going to start with, we don't have a complex deployment. >> > >> > 2 CM >> > 1 UC >> > 1 UCCX >> > 1 CER >> > 1 call recording server >> > ~2000 phones over ~8 sites >> > >> > our last upgrade we tried PCD (joke) spent 4 hours on it before just >> doing >> > it manually. Will be very hard pressed to every use PCD again. >> > >> > Then it was an additional 12-16 hours to upgrade. This was just a 8 to >> 10 >> > upgrade. >> > >> > We don't have that kind of time. and personally, I like my personal >> time a >> > lot. so the more I can do during the week leading up to the switch and >> as >> > small as I can make the switch, is what I'm looking for. >> > >> > Scott >> > >> > >> > On Thu, Oct 27, 2016 at 1:35 PM, Stephen Welsh < >> stephen.we...@unifiedfx.com> >> > wrote: >> >> >> >> I’ve not done CUCM project work in quite a while, so may be completely >> >> off, but what about making this scenario supportable: >> >> >> >> Complex cluster say, 1 Pub, 6 Sub, 2 TFTP >> >> >> >> Install new software to inactive partition on all nodes, once complete >> >> reboot part of the cluster: >> >> >> >> 1 Pub - new version >> >> 3 Sub - new version (primary subs) >> >> 1 TFTP - new version (primary TFTP) >> >> 3 Sub - old version (secondary subs) >> >> 1 TFTP - old version (secondary TFTP) >> >> >> >> Phone registers to upgraded primary subs, once everything >> >> working/stable/tested, flip remaining (secondary nodes) >> >> >> >> Maybe too complex for this split version to be workable, or not really >> >> much different than flipping all nodes, but may allow the phones to >> stay >> >> online with minimal disruption as long as all external elements >> strictly >> >> follow the primary/secondary node configuration. >> >> >> >> Thanks >> >> >> >> Stephen Welsh >> >> CTO >> >> UnifiedFX >> >> >> >> >> >> On 27 Oct 2016, at 21:23, Ryan Huff <ryanh...@outlook.com> wrote: >> >> >> >> >> >> You are right Anthony, this is a complex solution to avoid the reboot >> (and >> >> rolling the dice that nothing breaks in the first boot of the new >> version) >> >> in a switch-version however; if that is your goal .... as you state. >> >> >> >> -R >> >> >> >> ________________________________ >> >> From: avhollo...@gmail.com <avhollo...@gmail.com> on behalf of Anthony >> >> Holloway <avholloway+cisco-v...@gmail.com> >> >> Sent: Thursday, October 27, 2016 12:02 PM >> >> To: Ryan Huff >> >> Cc: Matthew Loraditch; Tommy Schlotterer; Scott Voll; >> >> cisco-voip@puck.nether.net >> >> Subject: Re: [cisco-voip] Not supported I'm sure..... but what do you >> >> think? >> >> >> >> If only there was an upgrade process wherein you install the new >> version >> >> to an inactive partition, and then could switch to the new version when >> >> you're ready. /sarcasm >> >> >> >> But seriously though, everyone in this thread is essentially coming up >> >> with their own clever way of replicating the promise Cisco failed to >> deliver >> >> on, which is performing your upgrades during production on the inactive >> >> partition and then switching versions in a maintenance window. If they >> >> would have only held themselves to a higher standard, we wouldn't need >> this >> >> complex of an alternate solution. >> >> >> >> On Tue, Oct 25, 2016 at 2:45 PM, Ryan Huff <ryanh...@outlook.com> >> wrote: >> >>> >> >>> Matthew is correct, copying is listed as "Supported with Caveats" at: >> >>> http://docwiki.cisco.com/wiki/Unified_Communications_VMware_ >> Requirements; >> >>> The caveat being found at >> >>> http://docwiki.cisco.com/wiki/Unified_Communications_VMware_ >> Requirements#Copy_Virtual_Machine >> >>> >> >>> The VM needs to be powered down first and the resulting VM will have a >> >>> different MAC address (unless it was originally manually specified); >> so >> >>> you'll need to rehost the PLM if it is co-res to any VM that you copy. >> >>> >> >>> Where I have seen folks get into trouble with this is where a >> subscriber >> >>> is copied, and the user mistakenly thinks that by changing the IP and >> >>> hostname it becomes unique and can be added to the cluster as a new >> >>> subscriber. I have also seen users make a copy of a publisher and >> change the >> >>> network details of the copy, thinking it makes a unique cluster and >> then >> >>> wonders why things like ILS wont work between the two clusters (and >> it isn't >> >>> just because the cluster IDs are the same). >> >>> >> >>> Having said all of that, I would NEVER do this in production ... maybe >> >>> that is just me being cautious or old school, but that is just me. >> Even >> >>> without changing network details on the copy, I have seen this cause >> issues >> >>> with Affinity. At the very least, if you travel this path I would >> make sure >> >>> that the copy runs on the same host and even in the same datastore. >> >>> >> >>> === An alternative path === >> >>> >> >>> Admittedly, this path is longer and there is a little more work >> involve >> >>> but is the safer path, IMO and is what I would trust for a production >> >>> scenario. >> >>> >> >>> 1.) Create a private port group on the host. If the cluster is on >> >>> multiple hosts, span the port group through a connecting network to >> the >> >>> other hosts but DO NOT create an SVI anywhere in the the topology for >> that >> >>> DOT1Q tag (remembering to add a DOT1Q tag on any networking devices >> between >> >>> the two hosts and allowing on any trunks between the two hosts). >> >>> >> >>> 2.) Upload Cisco's CSR1000V to the host. If you're not familiar with >> the >> >>> product it is at the core and unlicensed, a virtual router with three >> >>> interfaces by default. Out of the box, it is more than enough to >> replicate >> >>> DNS/NTP on your private network which is all you'll need. Assign the >> private >> >>> port group to the network adapters and configure DNS and NTP (master >> 2) on >> >>> this virtual router. >> >>> >> >>> 3.) Build out a replica of your production UC cluster on the private >> >>> network. >> >>> >> >>> 4.) Take a DRS of the production UC apps and then put your SFTP >> server on >> >>> the private network and do a DRS restore to the private UC apps. >> >>> >> >>> 5.) Upgrade the private UC apps and switch your port group labels on >> the >> >>> production/private UC apps during a maintenance window. >> >>> >> >>> Thanks, >> >>> >> >>> Ryan >> >>> >> >>> >> >>> >> >>> ________________________________ >> >>> From: cisco-voip <cisco-voip-boun...@puck.nether.net> on behalf of >> >>> Matthew Loraditch <mloradi...@heliontechnologies.com> >> >>> Sent: Tuesday, October 25, 2016 3:01 PM >> >>> To: Tommy Schlotterer; Scott Voll; cisco-voip@puck.nether.net >> >>> >> >>> Subject: Re: [cisco-voip] Not supported I'm sure..... but what do you >> >>> think? >> >>> >> >>> I can’t see any reason it wouldn’t be supported honestly. Offline >> Cloning >> >>> is allowed for migration/backup purposes. I actually did the NAT >> thing to do >> >>> my BE5k to 6K conversions. Kept both systems online. >> >>> >> >>> >> >>> >> >>> The only thing I can think to be thought of is ITLs, does an upgrade >> do >> >>> anything that you’d have to reset phones to go back to the old >> servers if >> >>> there are issues? I don’t think so, but not certain. >> >>> >> >>> >> >>> >> >>> Matthew G. Loraditch – CCNP-Voice, CCNA-R&S, CCDA >> >>> Network Engineer >> >>> Direct Voice: 443.541.1518 >> >>> >> >>> Facebook | Twitter | LinkedIn | G+ >> >>> >> >>> >> >>> >> >>> From: cisco-voip [mailto:cisco-voip-boun...@puck.nether.net >> <cisco-voip-boun...@puck.nether.net>] On Behalf Of >> >>> Tommy Schlotterer >> >>> Sent: Tuesday, October 25, 2016 2:49 PM >> >>> To: Scott Voll <svoll.v...@gmail.com>; cisco-voip@puck.nether.net >> >>> Subject: Re: [cisco-voip] Not supported I'm sure..... but what do you >> >>> think? >> >>> >> >>> >> >>> >> >>> I do a similar, but supported process. I take DRS backups and then >> >>> restore on servers in a sandbox VLAN. Works well. Make sure you check >> your >> >>> phone firmware and upgrade to the current version before the cutover >> or all >> >>> your phones will have to upgrade on cutover. >> >>> >> >>> >> >>> >> >>> Also make sure you don’t change Hostname/Ip addresses in the sandbox >> as >> >>> that will cause your ITL to regenerate and cause issues with phone >> >>> configuration changes after cutover. >> >>> >> >>> >> >>> >> >>> Thanks >> >>> >> >>> Tommy >> >>> >> >>> >> >>> >> >>> Tommy Schlotterer | Systems Engineer >> >>> Presidio | www.presidio.com >> >>> 20 N. Saint Clair, 3rd Floor, Toledo, OH 43604 >> >>> D: 419.214.1415 | C: 419.706.0259 | tschlotte...@presidio.com >> >>> >> >>> >> >>> >> >>> From: cisco-voip [mailto:cisco-voip-boun...@puck.nether.net >> <cisco-voip-boun...@puck.nether.net>] On Behalf Of >> >>> Scott Voll >> >>> Sent: Tuesday, October 25, 2016 2:43 PM >> >>> To: cisco-voip@puck.nether.net >> >>> Subject: [cisco-voip] Not supported I'm sure..... but what do you >> think? >> >>> >> >>> >> >>> >> >>> So my co-worker and I are thinking about upgrades. we are currently >> on >> >>> 10.5 train and thinking about the 11.5 train. >> >>> >> >>> >> >>> >> >>> What would be your thoughts about taking a clone of every VM. CM, UC, >> >>> UCCx, CER, PLM, >> >>> >> >>> >> >>> >> >>> placing it on another vlan with the same IP's. NAT it as it goes onto >> >>> your network so it has access to NTP, DNS, AD, etc. >> >>> >> >>> >> >>> >> >>> do your upgrade on the clones. >> >>> >> >>> >> >>> >> >>> Then in VM ware shut down the originals,and change the Vlan (on the >> >>> clones) back to the production vlan for your voice cluster. >> >>> >> >>> >> >>> >> >>> it would be like a telco slash cut. 10 minute outage as you move from >> >>> one version to the other. >> >>> >> >>> >> >>> >> >>> Thoughts? >> >>> >> >>> >> >>> >> >>> Scott >> >>> >> >>> >> >>> >> >>> >> >>> >> >>> This message w/attachments (message) is intended solely for the use of >> >>> the intended recipient(s) and may contain information that is >> privileged, >> >>> confidential or proprietary. If you are not an intended recipient, >> please >> >>> notify the sender, and then please delete and destroy all copies and >> >>> attachments. Please be advised that any review or dissemination of, >> or the >> >>> taking of any action in reliance on, the information contained in or >> >>> attached to this message is prohibited. >> >>> >> >>> _______________________________________________ >> >>> cisco-voip mailing list >> >>> cisco-voip@puck.nether.net >> >>> https://puck.nether.net/mailman/listinfo/cisco-voip >> >>> >> >> >> >> _______________________________________________ >> >> cisco-voip mailing list >> >> cisco-voip@puck.nether.net >> >> https://puck.nether.net/mailman/listinfo/cisco-voip >> >> >> >> >> >> >> >> _______________________________________________ >> >> cisco-voip mailing list >> >> cisco-voip@puck.nether.net >> >> https://puck.nether.net/mailman/listinfo/cisco-voip >> >> >> > >> > >> > _______________________________________________ >> > cisco-voip mailing list >> > cisco-voip@puck.nether.net >> > https://puck.nether.net/mailman/listinfo/cisco-voip >> > >> _______________________________________________ >> cisco-voip mailing list >> cisco-voip@puck.nether.net >> https://puck.nether.net/mailman/listinfo/cisco-voip >> >> _______________________________________________ >> cisco-voip mailing list >> cisco-voip@puck.nether.net >> https://puck.nether.net/mailman/listinfo/cisco-voip >> >> > > _______________________________________________ > cisco-voip mailing list > cisco-voip@puck.nether.net > https://puck.nether.net/mailman/listinfo/cisco-voip > >
_______________________________________________ cisco-voip mailing list cisco-voip@puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-voip